Wow, this is one of the things that we rarely encounter in courses and yet the impact matters so much for overall efficiency. Thank you for making these types of videos. Wish this channel the best!
Thanks for the positive feedback. Totally agree that some of these specific details might not be covered in school but are great at speeding up your code and pipelines!
@@robmulla it depends very much on the amount of data. Essentially you are making a trade off between faster and more efficient code and developing time. If you work with relatively small amounts of metadata and quickly want to get something done, this might be too much of a hassle. But if your code goes to production and has to go through massive amounts of data then it's certainly worth it.
@@dirk-jantoot1029 Good point. There is always a tradeoff between time spent on implementation and code speed, but it's still best practice to properly set column dtypes.
This note in the docs goes into detail about how categorical values only reduce memory use when the number of unique values are low: pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html#categorical-memory
Listen Rob, i came across your channel pretty randomly and your content is pure gold! straight to the point, and professionally presented! Thanks a lot! keep rocking
I deal with stock data for my job a lot- dealing with data frames that have daily data for 3000 companies across 20+ years means dealing with 16+ million rows. These tips are incredibly helpful for saving memory- which for my role is often the limiting factor of pandas and my computer. Too much memory load can slow down groupby calls, your computer as a whole and all code, and even worse crash your computer which has happened to me.
Great explanation. Just found out about that a while ago and wish I would have seen this video first instead of doing a bunch of googling. Was trying to get a 36M record dataset with categoricals and positional data to fit in memory on an average laptop for a mapping application. Recasting the datatypes made it all work out.
This guy is a legend. He started by using “size” in the random function which he didn’t define. Please did you have the value of “size” in memory before you started recording?
Sorry, would you tell me how the first time you made the table it must have been a 1000 rows? I saying cause it says size = size but I don’t get where is the 1000 size from. Awesome vids btw!
Great question. I think someone else pointed this out. I think I might have edited out that part of the video but I did end up editing that function to take in the size = 1000 at some point. Check the gist I posted here: gist.github.com/RobMulla/f04b144bb766b692f9314e3782d724d3
🎯 Key Takeaways for quick navigation: 00:00 📊 *Efficient Memory Use in Pandas Introduction* - Importance of efficient memory use in pandas for code speed, reduced memory consumption, and storage efficiency. 02:44 📉 *Initial Data Size and Considerations* - Creating a large dataset (1 million rows) and checking its memory usage. - Highlighting the impact of increasing data size on performance and memory requirements. 05:27 🧹 *Optimizing Categorical Columns* - Demonstrating memory reduction by casting categorical columns (position and team). - Significant reduction in dataset size by utilizing categorical data types. 06:44 🎲 *Downcasting Integer Columns* - Explaining downcasting of integer columns to smaller types for memory optimization. - Choosing appropriate integer types based on the data range to avoid information loss. 07:40 📉 *Downcasting Float Columns* - Downcasting float columns to reduce memory usage while maintaining precision. - Highlighting the impact of float type selection on data frame size. 09:02 🧹 *Optimizing Boolean Columns* - Efficiently casting boolean columns for minimal memory usage. - Using boolean type for binary data representation (win column). 10:23 🔄 *Performance Comparison* - Comparing computation times before and after applying dtype optimizations. - Demonstrating the overall improvement in code performance and memory efficiency. Made with HARPA AI
I think I did it that way because you need to define how to convert the strings to a bool. Astype won’t automatically know to convert those strings unless the we “true” or “false”
I know there's a Python module that takes a dataframe and calculates what type transformations one could do on the columns to reduce the size of the frame (it's pretty neat). Can't remember the name of the package now, though... I watched a YT video on it just about yesterday.
Awesome! I saw in a Video by Matt Harrison that, there is a library which sort of tells which dtypes needs to be converted to do memory saving. Unfortunately, I do not remember exact video. Are you aware of any such library?
Thanks for watching. I believe any int will always take up more memory than a bool. That is because a bool only uses one bit. int8, int16, etc use 8, 16 bits. A bool is essentially an int1
@@robmulla This is not completely true. In theory a bool needs only 1bit (true/false, 0/1). In practice CPUs can't address anything smaller than a byte, therefore a bool usually needs 1byte (8bit) of memory just like an int8. Nevertheless great video, thanks!
Hey! I recently really enjoy watching your videos. Could you maybe create a video in which you explain how I can run my python scripts automatically online, so that I don't always have to do this manually and with a switched on computer? I am getting a little bit more advancecd through your videos and I'd be super interested in this topic. Cheers! :)
Thanks for watching. That’s a great idea for a video. I think it would differ a lot how you would automate it depending on what you were running. Small program vs a really computational intense process.
Hi Rob, very useful thank you, but how do you deal with the following situation: you have a large MongoDB collection that you want to use locally to develop functions and play with the data. If you import it as a pandas data frame, it is just too large for the PC to handle. What's the best practice in this case? Worth a video tutorial? Thank you
Thanks. Can you aggregate the data in some way before exploring it locally? You could also just get a really large ec2 instance to run it on :D - another option would be something like dask. I have a video about pandas alternatives you should check out.
I'm not sure what you mean. You can set the dtype to 'category' using .astype('category') read more about it here: pandas.pydata.org/docs/user_guide/categorical.html
@@robmulla thanks for reply sir but lemmi clear my question again. without using astype or any inbuilt method how to convert the dtype to categorical of any column
@@robmulla it is possible sir, inside categorical class is being called and two lists are returned but mechanics were too complex to be understood by me, but im sure you will help me out ,plz send me your email id,so that i will send u the github link then you can go through it..nd that would be gr8 content for your channel too..cuz nowhere its available.
For example, I made this code which could help anyone. But I bet there are even better options out there: # 6) Change DTYPES #if column dtype == float & has no values after the decimal point = change dtype to int: for col in data1.columns: if data1[col].dtype == 'float64': if data1[col].astype(int).equals(data1[col]): data1[col] = data1[col].astype(int) #Else: try to reduce float to float 8,16,32,64: else: if data1[col].min() >= -128 and data1[col].max() = -32768 and data1[col].max() = -2147483648 and data1[col].max() = -128 and data1[col].max() = -32768 and data1[col].max() = -2147483648 and data1[col].max()
I haven't used it myself but you could also look into this: pypi.org/project/pandas-dtype-efficiency/ Understanding this concept is important beyond just pandas dataframes, but I agree it could be somewhat automated.
This function is good (I've used it on kaggle before) but you just need to be sure you are ok with casting the datatypes automatically, for instance if you expect to have new data added that could be larger or more percise, or if you would not like to automatically cast as categorical then you might not want to do it automatically.
@@robmulla Sory for my late response. Thank you for the answer!. Yes, that package automates a big part of the process. I would add some functionalities to make it even more customizable but it´s great for anyone who wants to check it. I also agree with you that it´s important to understand the concept. I needed an automation because after your video I use this frequently in large datasets.
@@robmulla Yes, you are right. I usually work with past data that will not be updated in the future so the function comes in handy. But of course, if new data will be added one should cast dtypes carefully.
