Python Pandas Tutorial (Part 8): Grouping and Aggregating - Analyzing and Exploring Your Data

Подписаться 1,3 млн

Просмотров 419 тыс.

50% 1

In this video, we will be learning how to group and aggregate our data.
This video is sponsored by Brilliant. Go to brilliant.org/cms to sign up for free. Be one of the first 200 people to sign up with this link and get 20% off your premium subscription.
In this Python Programming video, we will be learning how to group and aggregate our data. This will allow us to explore our data in ways we have not yet done in this series. We will be able to answer questions such as: "What is the most popular social media site for each country?" We will be using the groupby method, and also some aggregate functions such as mean, median, value_counts, etc. Let's get started...
Video Timestamps:
Aggregate Column - 2:00
Aggregate DataFrame - 3:55
Value Counts - 7:51
Grouping - 12:30
Multiple Aggregates on Group - 26:00
People Who Know Python By Country - 27:20
Practice Question - 34:20
Concat Series - 37:27
The code for this video can be found at:
bit.ly/Pandas-08
StackOverflow Survey Download Page - bit.ly/SO-Survey-Download
✅ Support My Channel Through Patreon:
/ coreyms
✅ Become a Channel Member:
/ @coreyms
✅ One-Time Contribution Through PayPal:
goo.gl/649HFY
✅ Cryptocurrency Donations:
Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot
✅ Corey's Public Amazon Wishlist
a.co/inIyro1
✅ Equipment I Use and Books I Recommend:
www.amazon.com/shop/coreyschafer
▶️ You Can Find Me On:
My Website - coreyms.com/
My Second Channel - / coreymschafer
Facebook - / coreymschafer
Twitter - / coreymschafer
Instagram - / coreymschafer
#Python #Pandas

Опубликовано:

13 фев 2020

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 708

@coreyms 4 года назад

I hope everyone had a great week! We've got a long video this week, but we go over a lot of important topics about how to analyze data in Pandas. We will learn how to answer very interesting questions such as "What is the most popular social media site by country?". I put timestamps together for this video so that you all can skip around if you need to go back and watch a specific section. Here are those timestamps: Aggregate Column - 2:00 Aggregate DataFrame - 3:55 Value Counts - 7:51 Grouping - 12:30 Multiple Aggregates on Group - 26:00 People Who Know Python By Country - 27:20 Practice Question - 34:20 Concat Series - 37:27 Have a great weekend everybody!

@calebmbugua745 4 года назад

Thanks so much bro,,,,much love from kenya

@anonymous-kl1un 4 года назад

Hey, is this series gonna continue?

@anonymous-kl1un 4 года назад

Can you explain all the types of joins

@anonymous-kl1un 4 года назад

And if possible please explain multi-level Indexing as well

@JoshuaDHarvey 4 года назад

Corey, is it safe to assume if your coming from a SQL background, that you can effectively use things like the 'pd.concat()' to replace the various joins (left, right, inner etc) workflows in SQL and just use SQLAlchemy or pyodbc libs to load the data and then do all the calculations with python that you would normally do in whatever SQL dialect?

@parthrawri3001 3 года назад

I love the fact that there are no ads interrupting in the middle. So thoughtful. ❤️

@coreyms 3 года назад

Yeah, I didn’t want the to ruin the flow of the videos. Glad you noticed :)

@parthrawri3001 3 года назад

Corey Schafer OMG! Your reply just made my day!

@livingwithlinlin3122 3 года назад

@@coreyms Thank you so much for doing this. You are such a considerable person with a big heart.

@JoshKonoff1 3 года назад

Corey, do you have a Patreon page? Thank you for your exceptional videos; a huge help for me and so many people!

@anubhavtomar1384 4 года назад

3:10 median function 5:00 describe function 7:20 count() 8:05 value_counts() 12:51 grouping the data 14:39 groupby() function 16:07 get_group(), grabbing a specific group by name 17:30 doing same by using the filters 18:40 using value_counts on filters 20:20 value_counts() for groups 21:49 using loc to find for one country 23:40 percentage by using normalize 25:00 median by country group 26:13 agg function for multiple functions 27:30 using filtering to get python users by country 30:20 error on using same approach for groups 31:40 apply method to run that on group 35:40 finding the percentage of people using python in each country(group) 37:40 using concat for combining series in a dataframe 45:30 adding percentage column

@afdqwfqwqwdfqwdawdas 4 года назад

thx, this is very useful. The videos already are very concise and to the point, but if I am just looking for how to do a proper groupby quickly on my own dataset....

@umutdemir2762 4 года назад

thanks a lot.

@ravishekharprakash4172 3 года назад

@@afdqwfqwqwdfqwdawdas sure

@80expertube 3 года назад

FYI, the percentage problem can be solved alternatively as follows: country_grp['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').sum()/x.count())

@sayarmandal1885 3 года назад

@@80expertube It throws a RuntimeWarning

@kylebeckhorn885 4 года назад

Yes please, do a video on the topic of MULTIPLE INDEXING!!

@j3553hh Месяц назад

I would pay to see Corey's tutorial on this. Every time I encounter a multi-index, I'm on stack overflow. It just doesn't seem to stick.

@pewolo 3 года назад

Let's all admit that this dude is a hard working man and his work is just a wow! I've been following him for quite some time now and I am always impressed by how thoughtful, tactical and clear his explanation is in every tutorial he makes. Hat off to you, dude!

@felipegomez3047 3 года назад

I'd like to share my solution to the practice question: country_grp['LanguageWorkedWith'].apply( lambda x: x.str.contains('Python').sum() / len(x) * 100 ) As you can see it's just as symple as adding " / len(x) * 100 " in the lambda function, where len(x) is the total number of users for each country.

@ironpolux 2 года назад

como se te ocurrio esto? ahem I mean, How did u come up with this? well played

@BCS_FahadAhmad 2 года назад

I guess x.count() in place len(x) makes more sense, since there can be people who did not answer language(I highly doubt XD)

@nicocilia5871 2 года назад

@@BCS_FahadAhmad I think x.count() will not count NaN so I think len is better if you want to include people that skipped that question. I am assuming that was an option.

@gurjotsingh8631 2 года назад

I was thinking the same, so i downloaded his repository and tried it and it works. Came here to comment and saw your comment. so , i just wasted 5-10 minutes of my day. whatever.hallelujah.

@kingler199 2 года назад

Damn well played

@prakhararora8981 7 месяцев назад

hey if ur df.median() doesn't work and ur getting typeerror and valueerror u can do df.median(numeric_only=True)

@DilpreetSingh02 2 месяца назад

Thanks man

@atienograce2520 2 месяца назад

Thanks a bunch!

@milrione8425 3 года назад

I love how you are just using the same data throughout the whole series. Thank you so much, Corey!

@jorgetiz99 4 года назад

This has to be one of the best videos on youtube about Pandas, thank you so much. Greetings from Perú.

@Davidkiania 4 года назад

Best video in the series loving them and normally can’t wait for the next.

@jongyoonsohn8559 4 года назад

I'd like to share my solution to the practice question. ctr_knows_python = country_grp['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python', na=False).value_counts(normalize=True)) ctr_knows_python.rename({False:'Don\'t know', True:'I know'}, inplace=True) ctr_knows_python Hope this helps too!

@coreyms 4 года назад

Nice!

@moushumitamanna 4 года назад

Hi, can you please explain what "na=False" means here and why do we have to put this in the code? Thanks in advance

@tplano3794 4 года назад

@@moushumitamanna not applicable

@moushumitamanna 4 года назад

@@tplano3794 thanks. But why should we put na=false in this code

@tplano3794 4 года назад

@@moushumitamanna in a column which is expected to have numbers, na does not make sense so we filter out these values. also if you run any functions (mean, median) then you may run into syntax errors

@walternyc 3 года назад

Working on a project evaluating an employee survey and this is just what the doctor ordered. Thanks! One of the best channels in RU-vid for data analysis hands down

@YeekyYeeky 4 года назад

one of the best thing that happened to me when I woke up (I am on the opposite side of the world to Corey Schafer) is finding that Corey just upload another Pandas tutorial video , thank you !

@antonyjohne 4 года назад

Hey Corey! Thanks a million for the Pandas Series. As always, very intuitive and easy to follow. Now that you've taught Matplotlib and Pandas, would love to see a new Numpy series in order to complete the Data Science trinity. Please consider adding a Numpy Series.

@jiangxu3895 4 года назад

I just discover that your way of teaching is to tell not only how to do it but why this is how to do it. thumb up!!

@brewtalxxx 2 года назад

Thank you so much for this video. I learnt way more from this than the many hours I spent sitting in class listening to a teacher who just wanted to end the lesson early or have long lunch breaks. This is really precious. And thanks for the reassurance that if I find this difficult, there's nothing wrong with me LOL.

@diegoalarcon6062 4 года назад

I don't care if some of your videos are long, in other channels they're just redundant but that's not your case! If you start doing short videos we may be losing all that valuable information that you provide to us. So far, this is the best Python channel I've seen. Greetings from Medellín, Colombia.

@zhenpan2048 6 месяцев назад

numeric_columns = df.select_dtypes(include="number") medians=numeric_columns.median() print(medians) # this is a way of getting the medians of numerical values as I use df.median(), it gave me value error that says could not convert string to float"I am not a student who is learning to code" thanks for great work. I learn more from you than from my professors. Thank you so much for great efforts!😎

@heretolearndshare 6 месяцев назад

You saved my learning session, thanks!

@giovannimantovani795 4 месяца назад

Thank you bro

@salehabdullahi9356 4 месяца назад

Thank you, you save me alot of time,

@mn4769 3 месяца назад

I found that you can shorten it by writing numeric_columns.median()

@sick-ol3jd Месяц назад

Thanks man

@elnazdehkharghani6121 4 года назад

You make all your subscribers happy with just uploading your videos !!! Thanks, Corey

@coreyms 4 года назад

Thank you all for watching!

@shikharsaxena9989 4 года назад

after this lecture i started loving the complex coding of pandas and matplotlib. really you are an amazing teacher

@LibardoLambrano 4 года назад

Thanks Corey for sharing these videos. Pretty clear explanations. You are a great teacher.

@bobchannell3553 4 года назад

Thanks for doing this video in a detailed way, like you always do. Just under an hour is a good length for a video like this. Thanks!

@joncochran9647 2 года назад

I've watched quite a variety of different data analysis tutorials and this one was easily one of the most engaging for me. Having interesting data really helps.

@Blueshockful 4 года назад

Im browsing thru some of the videos to brush up on Python, and this is the first python video that didnt get me bored. Concise and brillliant. Love your videos! keep up the good work :)

@ahmedhawater7522 4 года назад

Man you are one of the best teachers who ever learned me something, much love and support ♥️

@vagelisilias 3 года назад

I am a GIS student and I want to thank you because I'm doing my last assignment for university and I'm using Geopandas, matplotlib, pandas, cartopy and forth on and you helped so much with your videos, I have build a nice map and I have produced different tables with my data. Thanks god you are out there and sharing your knowledge free

@deniscampana8345 3 года назад

Thanks so much Corey ! It's clearly impossible not to understand what you explain on all your videos : It's fluid, straightforward, crystal clear ! And more over your english : Whaoooo ... Congratulations !! I wonder if I've learned more Pandas or english !! 200% great !!

@amir_forooghi 4 года назад

YESSSS !!! Corey`s video for groupby. I press like before I watch it. Groupby is just a superpower. Thank you for this awesome series Corey. You are the best.

@matthiashupfer2659 4 года назад

These tutorials are well thought out and really great in explainatory purposes. Greats skills here from Corey! Thank you.

@ashkanfarahani6532 3 года назад

Hi Corey. I think this might be a relevant simpler approach for getting percentage. I used value_counts(normalize=True) instead of sum. df.groupby(['Country'])['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').value_counts(normalize=True)) This of course return both percentage who know Python and Who don't know. So if we want to get for a specific country, for instance Japan, then: df.groupby(['Country'])['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').value_counts(normalize=True)).loc['Japan'][1]

@RegularDude95 Год назад

I have a similar approach to,i am happy to see that i am not the only one who always sees the easiest ways =)))

@JahidHasan-Aneek 8 месяцев назад

instead of using : value_counts(), use sum() in second line. Then you'll get appropriate answer. df.groupby(['Country'])['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').sum())

@abdulkadirguven1173 8 месяцев назад

Great approach. Thanks for sharing

@thotarohith2060 7 месяцев назад

Here is my approach : filt=df['LanguageWorkedWith'].str.contains('Python',na=False) python_count=df.loc[filt]['Country'].value_counts() python_count.rename('p_c',inplace=True) python_count -- total_count=country_grp['Country'].value_counts() total_count.rename('t_c',inplace=True) total_count -- result_horizontal = pd.concat([total_count, python_count], axis=1) import numpy as np result_horizontal.replace({'p_c':np.nan},0,inplace=True) result_horizontal['perc']=(result_horizontal['p_c']/result_horizontal['t_c'])*100 result_horizontal

@dennisp5302 2 года назад

I just went through Part 8 a second time. Thanks a bunch!! I learned a lot.

@merajajam425 4 года назад

The level of my programming in Python has been substantially improved since I have started watching your great videos. Many thanks, Corey. Would you please prepare some videos regarding the networkx module as well?

@codewithluq 4 года назад

Corey Again. Very fantastic tutor. I press the like button before I watch.

@rukhan8900 3 года назад

The reassurance at the end was so appreciated as a beginner. Thank you for your help !!!

@MatthewFoulk 4 года назад

Really appreciate the addition of practice problems. It helps me to grasp the material

@aborucu 3 года назад

Perfect explanation. Making a convoluted yet so important concept crystal clear through step by step explaning and also giving connections to pandas object types. Cheers!

@AugustoGeografo 4 года назад

Always looking forward for your videos, Corey.

@nicholaspolino2657 4 года назад

LOL @ "If I did this correctly, and it's definitely possible I made a mistake." Happy I found these videos, thanks.

@sayantanchakraborty75 4 года назад

Best videos on pandas on RU-vid by Corey Sir. Loving them and normally wait for the next videos. Lots of love for you from India.

@AmanChauhan-hy1sb 4 года назад

Thanks a lot Corey! Got to learn complex syntax in simple ways. You are amazing teacher.

@way_to_be_analyst6042 2 года назад

im just diving into pandas and would like to say - GREAT THANK YOU for such nice and detailed explanation. great job!

@AI-Health-posts 4 года назад

Thanks Corey. I have waited for this video whole week. Great explaination

@ironpolux 2 года назад

Really enjoying this series, thank u Corey!

@Yasharvl 3 года назад

Thanks Corey! This is pure gold!

@stressfreetrading1341 4 года назад

NAMASTE!! Corey Schafer.. Love From India

@rauberhozenplotz7009 4 года назад

Helps me to get into my PhD. Thanks a lot for uploading this!

@user-tb2jp7kg2c 3 года назад

Corey, thank you very much for your free videos!

@Terence818 3 года назад

Yes Corey, having a future video on multi-index will be very helpful!

@sayarmandal1885 3 года назад

Thanks, Corey. This is one of the most comprehensive pandas tutorials on RU-vid. Love from India. I also noticed a subtle issue. We are adding the number of respondents who filled their Country and not who filled LanguageWorkedWith. Someone can fill Country and not LanguageWorkedWith.

@parsahosseini4241 3 года назад

47 minutes of a pure pandas tutorial from a god in python, man you're a hero🔥🔥

@HearterSG 4 года назад

This is an incredible tutorial! Thanks Corey!

@manish_chandra Год назад

One of the best and most easily understandable vid on Pandas. Thank you for creating this !!

@buzz.b Год назад

Thank you for the last example (percent that knows python). It was great to see how the different methods learnt can come together in a practical example; this really helped consolidate the knowledge gained.

@paulorcordeiro3916 2 года назад

Corey, the content of your videos are amazing. This tutorial in special is sensational.

@belleriveblvd 3 года назад

Corey, I learn a lot from your videos. But this one has been especially helpful. Thanks.

@aayushipawar915 4 года назад

Hey Corey, Thanks a ton for this amazing videos. love them and they are so easy to learn

@bhargav1811 2 года назад

Every second of your python video are really worth it!!!

@finncollins5696 11 месяцев назад

thanks so much for this series. started from the first video two weeks ago, now in the 8th. this series so far made a lot progress in me,. thanks so much, .May God Bless You. Love from Sri Lanka...

@Youngcl77 3 года назад

Thank you very much, Corey, for the great video! I am getting involved with the data team at my company, and this has helped me tremendously!

@weiyancheng6360 3 года назад

Hi Schafer, thanks a lot for making these great vidoes and sharing the programming knowledge with us. I have watched your python, django tutorials and now this pandas topic and matplotlib is my next plan. I am 35 years old with no basic knowledge about programmnig, but your great work helped me a lot to learn new things.

@yuewang9623 4 года назад

Every time I saw a new post, I click the 'like' button before watching:D

@federicohan1458 3 года назад

I found amusing explaining what a percentage% is after going over apply & lambda methods, but that's exactly the thoroughness that makes your videos so loved :)

@denizcicek7333 2 года назад

You are just wonderful, it makes so much fun watching your tutorials. I finde directly the answer, those I need. God bless you brother.

@xuanyibutzin4775 5 месяцев назад

Super helpful video! Thank you Corey!

@AkashSingh-yp8ip 3 года назад

I truely appreciate your hardwork and knowledge and you effort to make things easier for learns.. cheers Corey

@antoniodefalco6179 2 года назад

you're a amazing teacher man, thank you for this free content

@Boat-xs8lm Месяц назад

Thank you,I am very lucky that I found your tutorials.

@arashfasih7323 2 года назад

Your technique for explaining things are truly great.

@bobchannell3553 4 года назад

This was a lot to learn in one video. That's why I went back and watched it again this week. At the end, I added something I think would be useful in what I do. I added a filter to select records where the number of respondents is >= 5. filt = python_df['NumRespondents'] >= 5 python_df.loc[filt]

@MauriceWilliams 2 года назад

Corey this Tutorial was awesome my Guy!

@mggarekar 4 года назад

nice video :) i liked the q/a approach at the end where you left it open.

@Schmidt3k 4 года назад

For your practise question, use .mean() instead of .sum() .mean() on a Series of bool will give you the fractions in a quick and easy way. Multiply by 100 for %. edit: As per discussion below, .mean() ignores NA values whereas Corey's approach treats NA as '0'. An alternative is thus: mygroups['LanguageWorkedWith'].apply(lambda x:x.str.contains('Python').fillna(0).mean()) Now, the results should be equal to Corey's.

@davidsp7949 4 года назад

It looks like a nice solution but numbers from Corey's video are slightly different than those with .mean() and I do not know why. For example: for Afghanistan PctKnowsPython 18.181818, with mean is 20.512821 for Albania PctKnowsPython 26.744186, with mean 27.710843 Does anyone know why?

@sunramaroc 4 года назад

@@davidsp7949 yeah i have the same doubt,,i guess that s due to the fact that mean() take in consideration only the respondents who effectively answered the question,,and sum() take all respondents even the ones with NaN for the question.so corey solution is Pct over all respondents,, and the mean() is over only the ones who answered this Q.

@jasleung2932 4 года назад

.mean() neglects those "NAN" responses while if u use x.str.contains('Python').sum()/x.size instead, it would count those "NaN" as "no pythoner" which is what Corey was doing

@fjramons 2 года назад

Well played. For me your solution is quite elegant. BTW, in case you wanted to treat NA as zeros (to get the same results from the video), you can simply use .mean() with its 'skipna' option disabled. This would make: mygroups['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').mean(skipna = False)

@mahanansari6152 Год назад

I just changed sum to value_count(normalize=True) and it worked

@alejandropereyra438 3 года назад

This video is so useful , the simplification that python does for the problems is so helping. is the best language in the programming of code. And the proffesor of this video is really a genius. !!! thanks.

@saiakhil4751 3 года назад

I signed up for brilliant org just for Corey Schafer. Thanks for sponsoring him.

@moushumitamanna 4 года назад

First of all, Thank you so much for all your tutorials. You are a great teacher. and yes, please make a video on Multiple Indexing.

@smitpatel1358 2 года назад

Great Series, Corey! Thank you very much!

@mohammadghouse25 3 года назад

Best Pandas playlist in youtube. One point solution for python learners

@sumranms 4 года назад

I really like the way you speak. Your language is clearly understandable and you have a great accent. :)

@harisjaved1379 3 года назад

Dude this is amazing! Nice work

@AAND8805 2 года назад

I am following your pandas series since the last 3 days and may complete in 1 or 2 days max, I will come back to the series to revise it, very well made Series and keep up the good work !😀

@samratsengupta8881 4 года назад

good job Corey, your content is well suited for preparation of data science

@noureddineettayyeby5210 4 года назад

Thank you for this awesome series and multiplexing would be great

@fvdvhome 9 месяцев назад

Mr. Schafer, I am so happy I found your teaching. I have been on a journey to become a data analyst, and after completing the Google Analytics Course , I realized that I needed to learn much more. I am currently finishing a Python Course through Coursera offered by IBM. Not every professional, no matter how good they are, have the natural ability to teach. Your method and technique are so amazing and helped me to overcome some of the confusions I had with coding in Python. I learned so much from just this video alone. I will definitely visit the site you referenced, and look forward to learning more from your videos. Thank you so much!

@gregoryogunna9527 9 месяцев назад

American?

@yashkumarsinghpatwa9267 3 года назад

Hey Corey Schafer Thanks a lot for this amazing series which helped me to upgrade my skills which I was unaware of that.

@Aaronisification 3 года назад

you know the world is upside down when the free content on RU-vid is head and shoulders above crooked Universities. THANK YOU, COREY!

@abdelrahmanali4550 3 года назад

Thank you Corey. I am learning a lot from you.

@freehappymeal 2 года назад

This was a very helpful video. Thank you, Corey!

@mohammadsalimkhan4974 4 года назад

Good work Corey. Loved all the explanations:-)

@turksonmichael1236 8 месяцев назад

Thank you for this. Had clearer understanding of pandas than before. Wish you the very best

@sherryshen2780 4 года назад

Thanks to your clear video and examples. Pretty useful to me!

@WrongSmth 3 года назад

Hey, Corey. I'm a network engineer and I'm learning pandas to be able to do some packet analysis and your videos really help me a bunch! This is my solution for the coding problem from the video. Hope it helps! know_python = country_grp['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').sum()) total_respondents = country_grp['LanguageWorkedWith'].apply(len) know_python / total_respondents