25 Nooby Pandas Coding Mistakes You Should NEVER make. 

I need to implement the chaining methods and using functions into what I do, much easier to use and read. Great video as always.

Usually these videos address REALLY nooby mistakes that any general programmer already avoids. THIS video however ACTUALLY addresses library FUNCTIONALITY and discusses the tools that a programmer may be unaware of to increase readability and efficiency. Rob, my good sir, you just earned a sub.

00:18 #1. Writing into csv with unnecessary index 00:53 #2. Using column names which include spaces 01:25 #3. Filter dataset like a PRO with QUERY method 01:44 #4. query strings with(@ symbol) to easily reach variables 02:07 #5. "inplace" method could be removed in future versions, better explicitly overwrite modifications 02:35 #6. better Vectorization instead of iteration 03:01 #7. Vectorization method are preferable than Apply method 03:30 #8. df.copy() method 04:08 #9. chaining formulas is better than creating many intermediate dataframes 04:28 #10. properly set column dtypes 05:01 #11. using Boolean instead of Strings 05:25 #12. pandas plot method instead of matplotlib import 05:45 #13. pandas str.upper() instead apply and etc 06:10 #14. use data pipeline once instead of repeating many times 06:41 #15. learn proper way of renaming columns 06:59 #16. learn proper way of grouping values 07:31 #17. proper way of complex grouping values 08:01 #18. percent_change or difference now could be implemend with function 08:25 #19. save time and space with large datasets with pickle,parquet,feather formats 08:58 #20. conditional format in pandas(like in Microsoft Excel) 09:22 #21. use suffixes while merging TWO dataframes 09:48 #22. check merging is success with validation 10:13 #23. wrapping expression so they are readable 10:33 #24. categorical datatypes use less space 10:55 #25. duplicating columns after concatenating, code snippet

Please keep doing this. No additional jargon, crisp, straight to the point explanations are what are required. No body needs a 10 hour tutorial. Thank you for this.

I thought I was pretty good in Pandas, but you gave me so many new things to improve. HUGE thank you!

I have been working 2 years now with pandas and I can strongly affirm that I have made like 70% of those bad practices, appreciate a lot your video!

I feel personally attacked. Thanks so much for releasing this. I knew my code was bad, but not THIS bad.

The pandas query function does not outperform the loc method. In fact, it is sometimes much slower when your query/data is so big. We industry users will utilize the loc method for quick EDA. Query might be useful when you have a scheduled cron

I'm currently working on my first major pandas project and I reckon that I may have done around 15/25 of these 'mistakes'. Looks like I have some optimisation to do over the coming days!

Wow dude! You are single handedly responsible for my data science growth. PLEASE keep making more of these videos I really appreciate it.

One of the best videos I've seen on Pandas! So glad someone prominent enough is advocating for method chaining and pandas methods!

Rob, thank you for all the time and energy you have put in for us. Would appreciate an updated video on "Exploratory Data Analysis" may be expanding on your year old one. Thank you again!

I had no experience with Pandas before joining a team where I need to work with it a lot. Have been learning as I go and it feels like the perfect time to see this video. I have enough time under my belt to have made or inherited code with many of these mistakes. With that context, I absorbed so much from what you shared. Thank you for helping me improve. I’m excited to refactor and apply what I learned!

This can be, some of my first times commenting in youtube after years of usage. This video was INCREDIBLY USEFUL! There's a lot of my previous team members did on scripts and sometimes are complicated to maintain or create new ones following the same logic. This covers exactly what they used and what is the best option to rewrite it and make it more understandable. Thank you so much for this godly information.

This video rocked me. I've been using python for a few months and watching this video made me bust out my laptop so I could try all of these items out. Thank you for this.

Learned more about Pandas in this video than a whole many videos worth hours combined. Seriously, thank you.

I've had little to no formal training. These tips are amazing and concise. Thank you so much.

Dude I've worked with pandas for 7 years and learned some new tricks, thanks a lot!

Matt Harrison's "Effective Pandas: Patterns for Data Manipulation" is one of the best resources I've read on idiomatic pandas.

I used the Pandas lib more then 2 years, but today I learned something new! Thank you, man!

The space need to be avoid part is so true! But wait a second, every time I face the space but not underscore is from others data, so I think what we actually need is how to deal with the space condition.(Which is a pain of journey)

Really enjoyed how fast this content came. I felt like it was a great speed to keep me engaged. I usually find these types of videos boring.

Found lots of favorite annoyances and learned a few new tricks! I'll add a shout-out to the ".pipe()" method to allow for wrapping all your transforms in a single statement when a single .method can't cover the required transform. An added bonus of "pipe()" - since it's using user defined functions to do the transforms, you can add decorators to automatically print out metadata on the resulting transform steps to get a quick insight into potential bugs.

oh wow the quality and clarity is worth subscribing! thank you !

I can't believe how good this video is. I love your no-nonsense delivery; I don't have time at work to watch a 4-hour "intro" video. Keep it up!

Oh man, I am making so many of these mistakes. Honestly, this is a great checklist to improve my clean coding.

I really think this should be written up in a medium blog article. Would be awesome to refer to.

I was genuinely worried I was making noob mistakes in Pandas...

Dear Rob, I'm a total beginner in Python and Pandas. From what I understand, the warning at 3:30 is not about making a copy of sliced data, but rather about not using the .loc method and using "direct assignment" for columns (or whatever it's called). I could be wrong, but this is what I've gathered from reading the documentation and encountering a similar warning in my code. Thanks for your valuable content. It has been a great help

I'm an experienced developer looking to get familiar with Pandas. I found this video very valuable.

This is awesome, I’ve been wanting to know what are the better ways to write my code and why. Please continue to make these videos.

Oh god. I clicked on this video just to confirm that this is one more overly exaggerated self-confident dude trying to teach newbies of 2 weeks experience. After watching this, this is god damn life changing. As an engineer focusing on fluid dynamics and floater response, I use pandas daily basis. Out of 25, I didn’t know approximately 20. Every single person who has any plan to use pandas must watch this. Awesome!

I'm so guilty of number 8! Thank you for this!

Awesome stuff. I've been using pandas for over 4 years, but it never occurred me to start using the query method instead of loc (despite me finding it tiresome to keep repeating "df" all over the place when using loc). I also appreciate the quick format. You see RU-vidrs taking too long to say nothing at all, so congrats on actually going through 25 tips in 10 minutes. You got yourself a sub!

I do several of these and never imagined Pandas has styling. Time to rewrite and share with my peers.

This video made me realize i have still a long road ahead in Pandas. Thanks! Just subscribed ;D

I can't believe I watched this whole video and only 2 of them were things I didn't know about! Thank you for sharing!

Awesome video! I work with Pandas for +3 years and learned a lot here! Thanks

9:13 another pet peeve, though this one is more important than the last one. Do not use backslashes. ever. well, not _never_, use them when writing a `with` statement with more than two context managers. But otherwise, don't. I'll quote the `Black` (the formatter) documentation: Backslashes and multiline strings are one of the two places in the Python grammar that break significant indentation. You never need backslashes, they are used to force the grammar to accept breaks that would otherwise be parse errors. That makes them confusing to look at and brittle to modify. This is why Black always gets rid of them.

some great tips here. i usually chain with \ and i didn't know a query method exists!! guess you learn everything new all the time!

Hey, Rob! Super video this one. I myself am Sr. DS working each day intensively with pandas, I will implement many of the tips you show! Thanks a million :)

When you mention the slice warning sometimes you don't care about the original data frame so it doesn't matter if you modified it

found your channels few days ago and man you have some epic content . The noob mistakes here are the exact way most tutorials teach you..just wondering why the hell the non noob ways are not taught as they are easier and shorter and the syntax makes more sense... thank you for this video

Have not, and will not make any of these mistakes because I’ve seen your “A Gentle Introduction to Pandas Guide” !!

I started to watch your videos recently, and from now on I'm doing the chaining and putting each function in "one row" to make the data cleaner, and also, the query method, so powerful and simple, I was used to replicate the dataframe with the column and value searched to filter my df. You are boosting my studies! Thanks for that!

This video is too damn good, I would love to find more videos like this.

Oh man, that guide is pro! Thanks, gonna apply all of that when refactoring my project!

Excellent points! Learned new stuff that a lot of tutorials don't explicitly teach.

Regarding the 'inplace' comment at 02:07 there's a very valid and very useful reason to prefer that and it's memory usage. `df = df.reset_index()` or anything similar creates an entire copy of the dataset before replacing it with the original and for extremely big data that is a problem, it may get over the physical memory available and have the OS kill the script.

Extremely underrated channel Extremely helpful

This video is literally a gem

As an R user, I would've never thought of doing all these mistakes. R is naturally a vectorized language. Use R.

OMG! I had to rest after first 10. So huge dose of information. Thanks.

Oi! There were several of those I didn't know. I wouldn't have thought I was a noob, but I guess we all have a bit of that in us. Thanks for the video!

Dude, Amazing video apparently clear the concept.

Releasing a notebook showing all these tips would be a great benefit to the community. The `.style()` trick at @9:18 is amazing.

Very illuminating video! I learned a lot quickly.

Wow, very useful - a true "tour de force" for better Pandas code. THX for this !

I wish I had this video 6 years ago. Thank you.

Fun fact from the *query* method that wasn't mentioned here. In *query* you actually (!) Reference to pandas columns. So you can do something like this : `df.query('Name.isna()')` - to query *NaN* containeings `df.query('Name.str.contains("John")')` - to filters all rows when Name containing John And even something crazy like `df.query('Price.rolling(7,1).mean() > Price.mean()')` to take rows that rolling means more than average

Nice video! I have been using pandas for years and still run into these issues :)

Learned tons with this. Short and succinct. New subscriber.

Merge validator! Excellent thanks!

Rob, as always, fantastic video. I have to admit, i get caught on some of those mistakes so it is great to have you point out and make suggestions on how to correct them. Thanks for sharing. Much appreciated.

At 6:23 (#14) you're returning the dataframe, but you're also modifying it in place. Having a return there gives the impression that the original dataframe isn't modified, specially if you also assign it to itself later. It ties back to #5.

Thanks Rob! I just made my first Kaggle notebook and I think I made all 25 of these mistakes 😂

great video! wanted to add on #7, may be someone would find that helpful: in case you need to apply some function to a several values in a row, one of the fastest solution is numpy.vectorize smth like: def divide(num, denom): if denom == 0: return 0 else: return num / denom so instead of doing df["div"] = df.apply(lambda row: divide(row["value1"], row["value2"]), row=1) you go with df["div"] = np.vectorize(divide)(df["value1"], df["value2"])

As a beginner this video made me learn some basic concept about pandas. thanks

lots of good info! thank you!

Thanks for the video!! A small comment about number nine, creating multiple intermediate dataframes. I understand that this can be costly in terms of memory, but I also think it can be nice for debugging and understanding during the development phase. Moreover, using the same name 'df' once and another can be prune to errors if you have different operations in different cells and you are 'playing' skipping some of them to see the effect, because you don't know which 'df' are actually taking as input.

Thank you! The .diff method is a lifesaver when computing velocities. The advice on not using inplace is excellent i got into various troubles because of it but i thought that's what the "experienced guys" do.

1. Using Pandas. It has always been more trouble than it's worth - at least in 90% of the places I encounter it at my job.

This was great! Just what I needed :)

This is amazing! Thanks a lot.

the last 5 were cool! thank you

I loved this to s be to my students. You did a great job in a short video!

Rob, you're an amazing person

phenomenal video. real learning accelerator.

Great video for new users not knowing tips and tricks.. Wish you shared the code also to keep it handy for reference

Rob, amazing video and intuitive. Happy to subscribe!

Hey Rob! You got me on that one right off the bat! I write a file to csv and when I load it back in, I get an 'unnamed' column and I wonder why....then I have to drop the column. 🤐Unnecessary work! Thanks a heap!

Great video. Thank you for being so direct and giving us valuable tips ☺

Great video. Very helpful. Please keep making more like this

#9 : I dont wanna, i like having legible transformations, but i'm aware its saving memory

You have turned my Dictionary into a Pamphlet 😂😂😂 Thank you

I feel like such an idiot, I wasted so much time writing inefficient code for my papers, if only I had known these. Better late than never though, thanks for the vid!

Another awesome, useful video, Rob. Thank you.

Very useful! Thank you for sharing in such an easy and agile way.

Great Video! I think its important to add that the Pandas Vectorization doesn't always mean the code will run faster. In particular for the case of working with string data types it can sometimes be slower (even if it looks cleaner).

Great video! I also like the jazz bass behind you, I also play bass :)

apply method is also a verctorisation function

Great video. Lots of operations and procedures that are helpful for effective coding. Would be really helpful to have a cheat sheet linked for easy reference.

Thanks for this great videos. One minor comment in number 14, to "pipe" and chaining several data transformation, one can use `df.pipe(process_data).pipe(yet_another)`.

Thanks for this great tutorial!!

Love it. Thank you!

I have been using the vectorised notation purely due to it requiring less syntax lol. But good to know its faster.

dot syntax is a noob way to do things. the correct way is always to use brackets because more often than not, you're accessing data, not creating it. also, dot syntax makes things more confusing.

Wonderful, this video is super helpful

Great insights, thanks for these important tips

Most of these always offer basic useless stuff that 99% already knows. This was not the case here! Well done mate.