Pandas 2.0 : Everything You Need to Know 

Rob, you are making the best videos. I am always watching them and learn new stuff. I learned python and pandas myself, and your videos helped me improve them several times fold. thank you!

I'm not sure what'd i'd be without you and this YT channel Rob. Thanks for being and amazing teacher to a rookie like myself lots of respect and love

Thanks Rob! Couple months seeing your videos and your content is on the spot ! 🎉

Thank you Rob! Your videos are always useful with a such nice flow!

Epic video Rob! I use pandas everyday and glad to hear it’s getting faster!

Thanks for sharing the update! Very well articulated

I am working exactly on my programming assignment of the date manipulation and it is very confusing. With the PyArrow datatype support, it is a lot easier from now. Thank you for highlighting the significance to us. You are a saint.

Great video Rob! I think Pandas will continue to be very useful in the data science community.

Your channel is pure gold.

Thank you Rob! Great explained!

Awesome! Thanks for the update.

thank you so much for such a kind information.

I particularly liked the conversion between the Pandas with Arrow backend and polars.. maybe you should make a separate video on it

Thanks for this update!

Hi Rob, thank you for the content. I'd be interested in seeing a video on using the old np.where() and np.select() in the new pandas 2.0.

Thanks, Rob. Great video!

Dope video! Would be interesting to see how it compares to polars. Also, what software do you use to record your face in that circle?

great video, tks Rob

Hey, your DS videos are awesome! I was using ChatGPT to learn more about Apache Arrow, Polars, etc from all your videos. Can you do a coding livestream or video about bioinformatics data like biological sequence data or drug development data? Thank you!

Good job pronouncing Interoperability!

very informative, thanks

I was about to switch to Polars but now with Pandas Arrow dtypes, I need to do more research. Speed is one thing but lazy processing and memory usage are very important.

Hey Rob! That text editor looks awesome. Which one is it?

ah yess the much needed Pandas improvement! this is what I like in a competitive environment..it sharpens everyone involved. I agree with what has been said in the comment section: now we can use Pandas to handle large dataset properly. especially that 20x speed improvement when reading data using Pyarrow backend is so good!

This int converting into float has been a long time PITA for me. This upgrade will be much welcome.

My very first thought was: yeah, with pyarrow as a backend, everything single software that relies on numpy has to be rewritten. But at the ends it doesn't looks like a big problem or a problem at all.

I would like to see more about aggregate performance. How it scales in CPU cores if it is still single-core.All for varying sizes of dfs

I would be interested to know how much time you lose when converting from pandas to polars and back to pandas?

Rob, do you have any limitations of Arrow to share? Until recently, for example, if I'm not mistaken, Arrow didn't seem to handle Pandas' multi indexes.

Thanks very much Rob! Your content is off the charts good, a true outlier, and a great outlier! To get my grad degree in stats, I had to take a PhD-level course in statistical consulting. The prof for that course actually wrote a whole book solely on the topic of outliers, outliers can be extremely important. To me, data science is a subject of stats, similar to the way experimental design or survey sampling are subjects of statistics. I think it is unfortunate that data science is not widely seen as belonging to “statistics” since it makes it harder (at least for me) to use data science tools when teaching stats, as the expectation is to use an old school approach to descriptive and inferential statistics in the classroom. I think your content has the clarity and insight to convince teachers that data science methods MUST be brought into the standard curriculum, e.g. courses like Introductory Business Statistics. Thanks again for posting truly remarkable content!

When you imported Polars, I thought you were going to do a versus on speed between Polars and Pandas with the new backend. But still very informative video, thanks so much!

Awesome! Are there any drawbacks to using the pyarrow backend?

Can you make a video giving techniques to update pandas (or polars) columns using vectoring instead of .apply() for more complicated custom function calls? eg. I want to take a str column, split it on a delimiter, do work on each section and then combine them back and return it as the new value for the same column

Pandas is life. Even if Polars is cool.

When you swap between pandas and polars, is python creating a copy of the dataframe into memory or is it just referencing the already allocated block of memory with the pyarrow backend?

Hey Rob, Great stuff! .. I moved my data transformation app to PD2.0 and I have a problem with data conversion. When Im converting data from int[pyarrow] to string using astype(str) im loosing pyarrow and pandas convert it to object :/. I cant find the tool to do it properly. All best!

Just one thing I don't understand. If boolaens are stored on single bits (really a great idea), how comes they can have 3 states (True, False, None). Wouldn't that require 2 bits ?

do you think this can have any "consequences" to the develop of numpy? I use pandas for data management but also use numpy and scipy and is very handy the way you can work with all of them till now. don't know if arrow can make the job so easy. also, can this make things more difficult for plotting stuff with matplotlib and seaborn, i.e.? p.d. thanks for the video and also for the previous 1h live stream in-ter-oper-a-bi-li-ty :D

There is an Int64 (capital i) which stores int with nulls

Wow! I'm still struggling in pronouncing "Interoperability" and it came out so fluid out of your mouth. Bet you are to release a video comparing pandas 2.0 vs Polars. Is Numpy still in pandas 2.0 due to backward compatibility?

It's a very good Video and this Library a good improvement, however ^^ nobody stops you from using a single int and store 32 different boolean states into it with bit shifting operations, but using a Interpreted Script Language means from the beginning "I trade resources for convenience". The underlaying Engine of Python is C and in C there is no actual boolean datatype, the smallest unit is the exact smallest unit the Operating System can provide which are 1 byte / 8 bits which the Operating itself only can provide to you if it reserves a whole virtual memory page for you. I just think, to nitpick about 7 wasted bits as a Python Programmer is a little bit awkward. The rest was great, thanks for the Video

Now that the beta is out is there any estimation about release ? Btw thx for the video !

and what about loading big dataframes? it will improve?

Is the PyArrow backend not gonna be the default? it would be annoying to have to specify the dtype for every table I have

One day, I learned that Python was slow for large data, so I started to study Julia who's reputation is "the fastest". One programmer I talked with said to me "No Python is fast!" And maybe new Pandas helps for speed matter. How should I take it from your point of view?

I get unknown engine: pyarrow but I have pandas 2.0 and Pyarrow 10.0.1. Am I missing something ?

pandas 2.0 with pyarrow backend VS pyspark dataframes?

Wow. First time seeing these data types. When would you not use pyarrow and just use default python or something else?

When should I be using pandas 2.0 vs Polars?

Can you make a tutorial on how to use the new upgraded MediaPipe object detection in python with live stream footage?

Do you have any affiliations to Polars?

This sorts two big issues, non-nullable ints, and the object dtype both can be frustrating.

Wait, how can Arrow store booleans as one bit each AND allow None values?

Hmm, Arrows seems great, but I had no idea something as fundamental as Pandas and NumPy was so janky, with really really sketchy quirks. Now I'm wondering what the speedup would be if you tried these operations with a system that wasn't using at least some scripting language interpreter calls for raw data shuffling? Like just a normal struct.

what is purpose of use_nullable_dtype=True

Id be living in the dark if it wasnt for this video 🤣

I wonder how much of my code this will break. Upgrading sqlalchemy to v2 was like sticking a grenade in my computer.

Hey rob this video is not showing not even got a notification re-upload it, only way I got access to this video by going to your playlist.

First time watcher feedback: Great video! Clickbait-y title, could at least have a more positive spin :)

in·tr·aa·puh·ruh·bi·luh·tee. Remember that Rob, 😉

pd.options.mode.dtype_backend = 'pyarrow' This option is dead now in pandas 2.0.3. It must have only been a part of the release candidate

I think with all this talk about pandas and polars people these days often neglect to even consider grizzlies.

Hi Rob my second comment I'm glad)

do you think 90% of data analyst will be eliminated due to GPT4.0 and further integration of machine learning in MS 365?

lol. funny stuff.

Apache Arrow is so much better.

Background music is slightly annoying. Your normal voice is peasant, so no need to supplement it.