The Ultimate Coding Setup for Data Science

Подписаться 168 тыс.

Просмотров 103 тыс.

50% 1

In this video I go though how I setup my operating system to code for datascience. We talk about operating system, terminal, IDEs and more.
Tmux settings repo: github.com/gpakosz/.tmux
My tmux and vim settings: github.com/RobMulla/vim_settings
Timeline:
00:00 Intro
00:22 Operating System
03:53 Terminal Stuff
09:19 Virtual Environments
12:40 IDEs
Follow me on twitch for live coding streams: / medallionstallion_
My other videos:
Speed Up Your Pandas Code: • Make Your Pandas Code ...
Speed up Pandas Code: • Make Your Pandas Code ...
Intro to Pandas video: • A Gentle Introduction ...
Exploratory Data Analysis Video: • Exploratory Data Analy...
Working with Audio data in Python: • Audio Data Processing ...
Efficient Pandas Dataframes: • Speed Up Your Pandas D...
* RU-vid: youtube.com/@robmulla?sub_con...
* Discord: / discord
* Twitch: / medallionstallion_
* Twitter: / rob_mulla
* Kaggle: www.kaggle.com/robikscube
#datascience #python #coding

Развлечения

Опубликовано:

31 май 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 195

@saxegothaea_conspicua 9 месяцев назад

Really cool video. Is always hard at the beggining to know which is the beast set up, and chossing from so many options. Thanks a lot and keep the good work.

@robmulla 9 месяцев назад

Glad it was helpful! Thanks for the comment.

@nevelis Год назад

Two tools that I discovered while interviewing for my last job is fzf (fuzzy find, super fast replacement for find + grep on file names) and ag (silver searcher, like grep -R but lightning fast).

@robmulla Год назад

Whoa! These seems like awesome tools and I'm going to try using them. Find in linux always seems more annoying than it needs to be. Often I use locate which can work well.

@luis9802 Год назад

Really cool! This is exactly what is missing. I am just used to work on jupyter notebooks and this helps transition into a more terminal-scripts workflow. Would be interesting to follow you on a project setup. Something like the tensorflow developer exam.

@robmulla Год назад

Glad it was helpful! Check out my live streams to see it in action.

@luis9802 Год назад

@@robmulla they are always at 3 am, i would love to interact live but I need to sleep haha

@juan.o.p. Год назад

Thanks for the recommendations!! I'll definitely give Tmux a try.

@robmulla Год назад

If there is one thing to take away from this video - is that tmux is great!

@matematicasyprogramacioncl1251 Год назад

I was wandering what os to use in my Mac for dual boot (data science projects and some develop). I think this video illuminate me.

@liamboyd4676 Год назад

This guy’s content is always on point!

@robmulla Год назад

Glad you think so.

@alfredoch3811 Год назад

GREAT! I'm in the process of learning Ubuntu, so this vid comes just in time...thanks!

@robmulla Год назад

Glad I could help!

@jstello Год назад

So much value here! Every time

@robmulla Год назад

Apprecaite that.

@jti107 Год назад

this was awesome! I actually use a chrome book and use google colab for everything. really interesting to see different data science setups

@robmulla Год назад

Whoa, that's great! I'm suprised you are able to do everyhing just using colab but that is cool that you have such a light weight setup.

@method341 11 месяцев назад

Do you find any limiting factors with the chrome book or does it do everything you want it to? Obviously not including GPU intensive tasks.

@ArunYadav-qq1cj 9 месяцев назад

Yes you can do almost everything on colab but it sometimes gets stuck for object detection tasks and some opencv libraries

@deepak39754 Год назад

Great Video Rob.Thanks for sharing the setup .Could you pls do a video on how to ssh , using vim ? And also few tips and tricks 😃

@robmulla Год назад

Thanks for the idea! ssh and vim are both really powerful - need to think about how I could make it into a video.

@deepak39754 Год назад

@@robmulla Thank you Rob

@5MassaM5 Год назад

As usual. Amazing content! Maybe you could also specify the hardware setup. Screen / Desk / CPU Components / Mouse etc.

@robmulla Год назад

Thanks for the idea! Maybe in a future video. My PC was built many years back but it has served me well.

@FilippoGronchi Год назад

Agree with your point...I must buy a new machine but i don't really know which... I am thinking about a MacBook AIr M2 with ethe possibility to install asahi linux... not sure though! Iw ill be a very nice follow up to this video Rob

@CarlosDelcristo Год назад

@@robmulla that would be a good video. Looking forward to it...

@mohamedtekouk8215 10 месяцев назад

Hamdulilh i have found your youtube channel as i switch from R to python and i found your channel really really amazing

@robmulla 10 месяцев назад

Glad to hear that! Thanks for watching.

@ashaykatrojwar880 Год назад

Was looking for this video!!

@robmulla Год назад

Glad you found it helpful.

@bazoo513 Год назад

Excellent overview. Thanks.

@robmulla Год назад

Glad you liked it.

@juliocardenas4485 Год назад

I would recommend adding Docker to the set up ; it makes it easier to deploy your work

@robmulla Год назад

You're right. For a lot of work I do is quick and dirty so the overhead of docker isn't worth it. But I do use docker from time to time.

@cedric1731 Год назад

You talk about helping us on a daily basis with a bass guitar in the background... I see what you did there :D

@robmulla Год назад

On a daily BASSis. 😉

@ibowman_UCLA_BRAIN 10 месяцев назад

Curious, and I can't figure this out based on comments, do you have a dual boot Mac machine (that boots between Mac OS and Linux) or do you instead have a dedicated Mac OS machine and dual boot between Windows and Linux in a different computer? Extremely helpful video! I'm not going quite as old school as top w/o colors but I'm about to do some C++ Qt coding for the first time in a decade, and was mostly interested in your thoughts on IDEs. I did not expect the VS Code recommend. Will go with that, thank you!

@imanshaf8429 10 месяцев назад

Hi Rob! What do you think of Mint? Do you think it is still better to use Ubuntu over Mint?

@OazadOMER 10 месяцев назад

Hello Rob can you suggest an alternative of PowerBI for Linux , Thank you

@lamhintai Год назад

Would like to see your advice on best practices when convert or bridge between the exploratory / experimental code in Jupiter notebook and production code as modules. 😊

@robmulla Год назад

Great suggestion! I still feel like I lean too hard on jupyter but there should definately be more of a balance. It really depends on your end goal.

@solom465 Год назад

@@robmulla Quarto in VS Code has replaced Jupyter completely for me.

@robmulla Год назад

@@solom465 This is the first I'm hearing about it. Reading about Quarto now and it looks like it might deserve some futher investigation! Thanks for the tip.

@cubic3d1547 Год назад

I really haven't ever liked jupyter notebooks. 100% agree on the comments made by Rob about running in VS Code. Biggest challenge for me was exploring in jupyter but then converting to run on a different machine that didn't have jupyter installed or wanted it to run directly from a terminal. I found interactive python in VS Code to be extremely useful. Can run cell by cell just as in a notebook but then can run directly from a terminal without any modification. Not sure it's the "right" thing to do but works for me and what I'm doing.

@bazoo513 Год назад

Re: vi - Yes, it is unintuitive for novices, but once you learn _all_ of it, it is the most efficient programming editor by far. What it does _not_ provide (without extensions and/or external utilities) are niceties like keyword and variable completion, function/method synopsys help etc. IDE it is not, but as a pure editor, it is unmatched.

@robmulla Год назад

vim is really powerful. But I actually find most of the benefit is when editing existing code and not writing completely new code.

@Jackson_rw 3 месяца назад

Hey man love your setup! I am pursuing a data science major this year and want to do Linux as my main OS. What colors are you using for your terminal? It's so clean

@laminkhant8875 Год назад

This is such a cool content for me, now I stop dreaming about to reinstall windows :D

@robmulla Год назад

Glad you liked it! Thanks for the comment.

@gauravmalik3911 4 месяца назад

What is the difference between conda env and Python env, which one to chose when? Is python env is used when we are deploying the model?

@danieltheshark 7 месяцев назад

Thanks for sharing.. I really like the idea to work with linux. Just clean and go. but... what about the issue with the mouse scroll speed? IMWheel... I have a mx master and the first scroll doesn't work. What is this? What about that difficult switch for audio source. I guess the only way is to do that in settings? Do you have problems with your gpu? NVIDI? Are you using vs code + juypter notebook? I can't help --- but this sucks. Kernel broke so many times Thanks

@ChrisHalden007 Год назад

Great video. Thanks

@robmulla Год назад

Glad you liked it!

@oliver_ai Год назад

I finally learned how to load scripts in Jupyter lab. Thanks

@robmulla Год назад

That’s a good habit to get into. Using sys to update the path of your scripts makes it easy.

@arda3420 Год назад

Casual people: python, jupyter notebook and modules are the most important things for data science. Rob: tmux and linux or go cry

@arda3420 Год назад

Omg and Vim too xD. You are gigachad of data science

@robmulla Год назад

😆 I like to start with the basics and then move my way up!

@robmulla Год назад

vim keybindings are so worth mastering!

@thanoslyras3906 9 месяцев назад

I watched the video and it is very helpful. I would like to ask what is the theme you use because it is very clean.

@mirandaenergypark5271 Год назад

Great video Rob - thank you. I hope you don't mind me mentioning, but you have a spelling error in your website title banner. I know, a minor thing, but being a programmer and focused on detail, you may appreciate!

@robmulla Год назад

Oh! Great catch. You were the first to notice so you win a prize! Just kidding, but I changed it :D

@serdartohma 11 месяцев назад

In vs code in import pandas as pd and import numpy as np , pd and np is faded. I wonder why? and code is not outputting. can you help me?

@robmulla 11 месяцев назад

I'm not sure, but vscode will grey out the lines of imports if they aren't being used in the script. So if you never use `pd` or `np` they will be grey. But that shouldn't impact your code.

@philwebb59 10 месяцев назад

It's so funny hearing you praise bash. It's like tripping back to the early-80s when I used to praise the original Bourne shell and argued how GUI would never be popular. Of course, that was before 'ol Bill stole the mouse from Xerox. Boy, was that a bad business decision for Xerox.

@davidmora2241 Год назад

Great video! Do you know if it's possible to use the vim keybindings inside kaggle notebooks?

@robmulla Год назад

Yes it is! The keybindings to switch cells use Vim-like shortcuts with j/k. I believe you can also add vim bindings within cells but I don’t use that.

@satyamARAS Год назад

I used pop and parrot is on vms for learning Linux command and they were good pop And parrot felt like windows lite , ubantu or whatever distribution you use highly depend on project/team/company/domain , i guess?.

@robmulla Год назад

Not familiar with either of those, but I started using Linux Mint and when switched over to ubuntu way back.

@peblopadro Год назад

Thanks very much for all the knowledge you are sharing! Cool bass btw 🤟 Quick Q: after installing tensorflow, when I'm importing it in a Jupyter notebook ( [ import tensorflow ] ) the kernel dies. Have tried all potential solutions in stackoverflow and nothing has worked. Any guess is highly appreciated 🙏

@robmulla Год назад

Hey Pedro, thanks for watching! Sorry to hear your notebook keeps getting shutdown. I would recommend trying to run the same code in a kaggle notebook just to make sure that's not the issue. If it persists maybe attempt to create a new conda environment from scratch?

@alpsgrin4501 Год назад

How to create a cluster and set multi GPU in one computer hpc claster which Linux do you recommend and what settings?

@robmulla Год назад

First you need a lot of money.... 😀 then you buy the hardware. After that is just setting up ubuntu and installing nvidia drivers. Good luck!

@pomiro Год назад

Thanks!

@robmulla Год назад

You bet!

@MeuRefrigerador Год назад

Not that I'm advocating for Windows, but shouldn't you be able to do terminal stuff from linux using PowerShell?

@robmulla Год назад

Yes, or even better WSL- which allows you to run linux on windows. I've never used it but I've heard others like it.

@jcbritobr Год назад

The problem i see in vscode for python is the notebook. Seems like the things not fit exactly in interface, but pylance is awesome

@robmulla Год назад

I agree. If the notebooks in vscode felt like jupyter lab I would only use vscode.

@Gruuvin1 Год назад

When you ran htop, I got jealous. 32 cores with 64 threads? What processor are you running?

@robmulla Год назад

Ryzen Threadripper! 🔥🔥

@kesleyraimundo3527 Год назад

Why use pip instead of conda to install packages?

@bazoo513 Год назад

Re: "why Linux" - I have to do most of my work on Windows because my clients do. But I make sure to install CygWin on every machine I work on (including clients' servers running SW I maintain) - command line is immeasurably more efficient than GUI in 95% of cases. On my Windows I now have LTS versions of Ubuntu (22.04 currently) under WSL2 and, again, do almost everything from bash.

@robmulla Год назад

I’ve hear WSL2 is good but haven’t used windows in a while so I haven’t needed to use it.

@arielspalter7425 Год назад

That’s awesome tutorial, learned quite a few things. I have a question though- how does Excel fit into here?

@robmulla Год назад

Good question! Ubuntu has LibreOffice which is a quick alternative to excel, and you can also use office 365 to access excel in the browser. But for some people I know not having excel locally is a deal breaker.

@arielspalter7425 Год назад

@@robmulla thanks! One more question- could you please expand a bit how you run the Linux environment? Is it a dual boot or VM, or a dedicated machine altogether? Thanks!

@robmulla Год назад

@@arielspalter7425 I run linux as my main OS.

@aoa1015 Год назад

You read minds don't you !!! :) I wanted to ask about your software and hardware setup and recommendations.

@robmulla Год назад

haha! That's awesome. I get asked a lot when streaming about my setup so I finally made a video about it.

@oskwon72 Год назад

Great! 99% overlapped tools for me except for Org-Mode in Emacs

@KabeloMoiloa Год назад

Consider using emacs-vterm or the newer emacs-eat instead of tmux.

@MrEo89 Год назад

I’ve heard of the “eat” stuff, but dk how it’s better? Like, eat-jupyter is one in particular I’d love to check out for my older rMBP

@demohub Год назад

👍 thanks for sharing

@robmulla Год назад

Thanks for watching!

@kongson14 Год назад

it helps!

@robmulla Год назад

Thanks! I'm glad it does.

@sirsquirrel0 Год назад

“Well that’s just me” - me too! 👌👍

@robmulla Год назад

haha. Glad I'm not alone!

@thatsunpossible312 Год назад

Anaconda is no longer free for commercial use, so be careful. You can instead use conda-forge, which is free. The “conda” command line tool is written in python and can be slow. Instill mamba to get the same functionality written in c++. All of this is more important in windows than on Linux. Like you, I just use pip on Linux.

@6872elpado Год назад

I'm amazed that you keep broken environments 😂

@robmulla Год назад

I'm amazed you don't! 😉

@6872elpado Год назад

I will try then :D

@bazoo513 Год назад

Thanks for Tmux tip. BTW, 64 cores, 64 GB RAM? I am green with envy 😀

@robmulla Год назад

I built my machine myself! It’s pretty old but powerful.

@prashlovessamosa Год назад

I want you make everything in detail not in 1 hour i want to learn everything because your tutorials are awesome

@robmulla Год назад

Thanks for the comment! Are you saying you wished the tutorial was longer? You can always watch my live stream or past live stream recordings where I go at a slower pace. Good luck!

@sawekb8102 Год назад

I worked for some time on Linux but one thing that make me switch again to Windows was that there was no way to install Excel on Ubuntu

@robmulla Год назад

That's true, but you can use office 365 in the browser I believe. I also try to use pandas instead of excel whenever possible.

@geodancer7281 Год назад

Have you considered LibreOffice?

@sawekb8102 Год назад

@@geodancer7281 Libre office is limited but it is an option

@geodancer7281 Год назад

@Ralph Reilly - That's harsh. LibreOffice is, well, libre. That's not bad for an Excel-workalike. I appreciate your black/white attitude, but when you are searching for a tool, something close might be better than nothing.

@johnarroyo6376 Год назад

I would beleave if u were talking about MS Word, that's a nightmare... but Excel? U could easily work with some other alike excel app, and use it as is u where on excel

@gustavojuantorena Год назад

I don't know why, but suddenly I want to try tmux

@robmulla Год назад

You need to! It’s great.

@DylanOkyere Год назад

You didn’t add the tmax repo in the description

@robmulla Год назад

Just added it! Thanks for the reminder.

@shreyaskulkarni5823 Год назад

Can you make a video on dual booting?As a student I feel dual booting is a powerful tool

@robmulla Год назад

It can be kind of tricky, and hard to show in a video tutorial because it requires shutting down your PC a few times. I'll try to think of a way that I could make it work.

@sougaaat 4 месяца назад

Threadripper? MY GOD!

@Mutual_Information Год назад

Very interesting. I use a Mac at work and Windows at home, and I love bash via macos.. PowerShell on windows is garbage. But I learned that Windows has, as of fairly recently, a very good recreation of Linux. It's called WSL2. It's not bad. But maybe I should just become an OG linux user like you lol

@robmulla Год назад

I've heard good things about WSL2 but still prefer avoiding windows unless I need to. You should try setting up dual boot and checking out ubuntu.

@arminhejazian5306 Год назад

i've been using WSL for couple of months and it's great.

@MaxMustermann-on2gd Год назад

WSL and while in windows Git Bash. + use windows Terminal

@nissum5179 Год назад

What laptop is goof for data science?

@robmulla Год назад

Good question. I think it really depends on how much you plan to do locally vs remotely. What really matters to me is that the laptop is comfortable and easy to use.

@incremental_failure Год назад

Ubuntu is ok but Arch based distros are a bit better. The package manager is definitely much better. I use paru as a pm and it's great. That said, my VPS still runs Ubuntu.

@robmulla Год назад

Interesting. Maybe some day I'll give it a try. What is VPS?

@incremental_failure Год назад

@@robmulla Virtual Private Server. Your cloud box.

@aggbits Год назад

windows + WSL for the win!

@robmulla Год назад

Windows is overrated. Change my mind. 😁

@aggbits Год назад

I don t want to change anyone's mind .. Just saying that using windows with WSL2 integration gives you the best of both worlds.

@robmulla Год назад

@@aggbits I've been hearing a lot of good things about WSL2 so I don't want to bash it (pun intendend).

@DD-uq8qg Год назад

Kind of the same things that I use too

@robmulla Год назад

Love it. What do you use that is different?

@siddharthyadav2120 Год назад

Hey Rob! Just curious... do you play chess?

@robmulla Год назад

haha. What made you guess?

@sayednab Год назад

I tried VSC but I didn't like it at all. it's so confusing and busy. whereas Jupyter notebook is intuitive and easy to understand.. also there is not many good tutorials on VSC contra Jupyter.

@michaelomosebi6374 Год назад

This is alot.... Impressive or overwhelming...do not know which to say. Do you have a video that describes your journey, from start? I would really love to see it

@robmulla Год назад

Thanks, hopefully it’s not too overwhelming. I don’t have a video that describes my journey but have plans maybe in the future. Appreciate you watching.

@hatwong3637 Год назад

Couldn't agree more that Linux is a better platform over Windows as a coding machine. However, I always got stuck when installing GPU drivers on my Linux machine...😟

@robmulla Год назад

Ah, I’ve been there before. But it is a lot easier than it was before.

@Pi_Maaster 6 месяцев назад

debian can be better than ubuntu uses exactly the same pachages and with an extension can look similar to ubuntu

@AbdulREHMAN-co2kd 8 месяцев назад

i work with jupyter notebook the layout is lame but simple e to use

@TheCodeGuy12 Год назад

OO-boon-too

@robmulla Год назад

ewwwwww

@soren-1184 Год назад

I would use miniconda instead of anaconda and mamba instead of conda.

@tainicon4639 Год назад

Ubuntu is not the most popular or most stable linux distro. Though It is very beginner friendly.

@robmulla Год назад

My bad. What's the most popular distro? Debian?

@tainicon4639 Год назад

@@robmulla I believe based on Reddit poles (which may be biased due to the type of people who would take those poles…) Arch based distros beat out Ubuntu based distros. But Ubuntu is a close second. My stability comment was about the issues I have had with apt randomly breaking my system. As a general philosophical statement against Ubuntu, it’s preloaded with Spyware that sends information back to the canonical. But it’s is easy to install and is often the first choice for people migrating over from windows or Mac. So it’s a solid recommendation. The arch based distros tend to attract a certain type of elitist (even though it’s not hard to use…) which is probably best to avoid when you are first starting out.

@tainicon4639 Год назад

@@robmulla great video BTW!

@adrianmuresan7764 Год назад

I am using anaconda in windows and it started running out of SSD space. After watching your video, I ordered a 2Tb SSD with the intention of installing Ubuntu on it and using your setup. Trouble is, this Ubuntu bastard has a habit of installing on only part of the drive. I made a partition for swap, another for tmp, another for root and that was it, with the expectation that the home partition will take the remaining space. That´s not what happened, I got a 30Gb home partition and 1700Gb unused hahaha. Back to windows for me, I installed windows on my spanking new drive, until I figure out how to install ubuntu and still have space for anaconda. When I think that i used to be only an ubuntu user until aroud 2013.

@robmulla Год назад

Oh man! That’s annoying. I’ve spent many weekends just trying to fix issues like this. But trust me once you get Ubuntu setup you won’t need to go back and you’ll be happy you switched. Good luck!

@adrianmuresan7764 Год назад

@@robmulla thanks for the reply, Rob. It helps knowing that also the masters have trouble with 'yak shaving'´. In any case, I have now the 250gb SSD where the windows used to reside, when I feel like it I will give it a shot and install Ubuntu there. I do believe you are right about the superiority of Ubuntu.

@PerteTotale Год назад

I don't see why you condone windows: I had geospatial pkgs installed with pip/whl, with much effort, but it let me to know and understand python much better. So I'm now on windows and with geospatial wizz and QGIS witches, and with conda, and with multi envs. I must have broken your unix egg shell

@robmulla Год назад

Windows is fine! I’m just saying for me I’m not a big fan. But I also haven’t used it in years so things may have changed a lot since then!

@mirandaenergypark5271 Год назад

Sorry, I meant your RU-vid banner...

@robmulla Год назад

Thanks for the heads up!

@CrazyFanaticMan Год назад

The best IDE is notepad

@robmulla Год назад

The best IDE is ms word!

@freiTzer 3 месяца назад

I use visual studio and pycharm

@CrazyFanaticMan 3 месяца назад

@@freiTzer Rookie mistake

@hansdietrich1496 Год назад

As you talked a lot about your terminal ... why bash? There are way better shells out there!

@KarmaKula 11 месяцев назад

ctrl+b and shift+% ??? I just can't....

@robmulla 11 месяцев назад

I map caps lock to be another control.

@KarmaKula 11 месяцев назад

@@robmulla that's interesting

@linuxmill Год назад

you don't like emacs?

@robmulla Год назад

I’m sure it’s great. I just started with vim.

@linuxmill Год назад

"old school users"... guilty😅

@kslanov Год назад

Moving files with bash commands 🤦‍♂️

@robmulla Год назад

What’s a better alternative?

@rishavpapaji5349 Год назад

you are soo wrong about ubuntu

@robmulla Год назад

Ok. Change my mind.

@rishavpapaji5349 Год назад

Sorry don't take it personally but it something is popular doesn't mean its most up to date or latest, most latest packages are in arch linux and it doesn't mean it most stable its a tradeoff between two, but you might have to say that ubuntu has largest community and support not the most updated or latest kind....

@rishavpapaji5349 Год назад

Ubuntu 22.04 use python 3.10 which you know is technically outdated

@robmulla Год назад

@@rishavpapaji5349 I'm not taking it personally. I apprecaite your feedback and want to know what you think is a better option and why. What OS do you think is best? I'm still using Ubuntu 20.04 because I haven' gotten around to upgrading yet 😅

@rishavpapaji5349 Год назад

@@robmulla I use Arch By the way

@adrienpacifico5979 Год назад

I felt ashamed when I saw your htop compared to mine! 64 cores! I only have 16!

@robmulla Год назад

What you lack in cores I'm sure you make up for in other areas 😂

@johnlucas7333 Год назад

How many people are doing "data science" projects? Welcome AI

@robmulla Год назад

Not sure I understand your point. Are you saying that AI has replaced the need for science?

@petermoore8811 Год назад

tip .. I got board and stopped watching. because I thought it was going to be more focused on data science instead you devoted a big portion of the start on working environment. People (video creators) need to assume we know how to operate in linux vs windows vs mac. My tip is provide a link to basics and get on with the title of the video. Having said that. I commend your effort and keep it up.

@robmulla Год назад

Thanks for the feedback Peter. Sorry you got bored for the video, hopefully my future videos- or some of my other existing ones will be more interesting! I’ll keep your suggestions in mind in the future.