Тёмный

How do I use the MultiIndex in pandas? 

Подписаться
Просмотров 172 тыс.
% 4 146

One of the most powerful features in pandas is multi-level indexing (or "hierarchical indexing"), which allows you to add extra dimensions to your Series or DataFrame objects. But when should you use a MultiIndex, and how do you create, slice, and merge MultiIndexed objects?
In this video, I'll demonstrate:
- How to create a Series with a MultiIndex, and how to convert it to a DataFrame
- How to select from a Series with a MultiIndex
- How to create a DataFrame with a MultiIndex
- How to select from a DataFrame with a MultiIndex
- How to merge two DataFrames with MultiIndexes
WANT TO JOIN MY NEXT WEBCAST? Become a member ($5/month):
www.patreon.com/dataschool
=== RELATED RESOURCES ===
Download the lesson notebook: nbviewer.jupyter.org/github/justmarkham/pandas-videos/blob/master/pandas_multiindex.ipynb
Using the pandas index (Part 1): ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-OYZNk7Z9s6I.html
Using the pandas index (Part 2): ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-15q-is8P_H4.html
Analyzing groups with groupby: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-qy0fDqoMJx8.html
Selection and slicing with loc: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-xvpNA7bC8cs.html
My full pandas video series: ru-vid.com/group/PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y
DataCamp course on MultiIndex: www.datacamp.com/courses/manipulating-dataframes-with-pandas?tap_a=5644-dce66f&tap_s=280411-a25fc8
DataCamp course on merging: www.datacamp.com/courses/merging-dataframes-with-pandas?tap_a=5644-dce66f&tap_s=280411-a25fc8
Tidy data: r4ds.had.co.nz/tidy-data.html
== LET'S CONNECT! ==
Newsletter: www.dataschool.io/subscribe/
Twitter: justmarkham
Facebook: DataScienceSchool/
LinkedIn: www.linkedin.com/in/justmarkham/

Опубликовано:

 

15 ноя 2018

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 314   
@infiltr80r
@infiltr80r 4 года назад
It's rare to find exactly what you're looking for when it comes to Pandas but this was it.
@dataschool
@dataschool 4 года назад
Great to hear! 😄
@rezas2626
@rezas2626 5 лет назад
Slice(None) trick was what I looked everywhere to find and found it here. Thank you so much!
@dataschool
@dataschool 5 лет назад
You're very welcome! :)
@annieyang4420
@annieyang4420 4 года назад
Life saver video! i have been panic for 2 days , read many pieces and still confused. this helped me out! thank you!
@dataschool
@dataschool 5 месяцев назад
You're very welcome!
@joaoostrowski
@joaoostrowski 5 лет назад
Hello from Hungary Kevin! After a period away from the technicalities of python working mostly with Google Analytics and CRO techniques I'm glad to be back to your videos playing around Jupyter. I appreciate a lot the work you do on Data School, so here is a bit of my honest appreciation, thank you and keep it up! John Ostrowski :)
@dataschool
@dataschool 5 лет назад
Thanks so much John! :)
@adamyork8407
@adamyork8407 5 лет назад
This was so cool! I am already thinking of a dozen ways to use multi-indexing in my newbie projects and adding them to my little presentation next week. I owe a lot of my comfort with pandas to you. I hope your holidays went well, Kevin!
@dataschool
@dataschool 5 лет назад
Awesome, thanks so much for your kind comments, Adam! 👌
@stephenmaddock2863
@stephenmaddock2863 4 года назад
Brilliant tutorial. This is the 2nd of yours that I have watched and it covered what I need in my new job. Thank you very much.
@dataschool
@dataschool 3 года назад
You're very welcome!
@carolinefawer-wasser2570
@carolinefawer-wasser2570 5 лет назад
As always great explaination. You are a gifted teacher, so surely well worth the Patreon membership for you.
@dataschool
@dataschool 5 лет назад
Thank you, Caroline, both for your kind words and for your support through Data School Insiders! :)
@searchingsoul5910
@searchingsoul5910 4 года назад
Extremely informative, thanks for making it so simple - really good job
@dataschool
@dataschool 5 месяцев назад
Thank you! 🙏
@khardu
@khardu 2 года назад
Thank you for sharing your knowledge and experience beautifully. I am new to Python and Pandas, was struggling to understand how to select a subset of the data in a MultiIndex scenario. You made it a piece of cake by putting it all in one video, and by comparing and contrasting between Series and Data Frame.
@dataschool
@dataschool 2 года назад
That's awesome to hear! 🙌
@ssankarnarayan
@ssankarnarayan 3 года назад
slice(None) instead of : was a very important input. I struggled with it for quite a while before watching this video. Thank you.
@dataschool
@dataschool 3 года назад
You're very welcome!
@brockobama257
@brockobama257 2 года назад
I couldn't find the Corey Schafer video on MultiIndex so I'm here. And I'm not disappointed... subscribed!
@dataschool
@dataschool 2 года назад
Welcome to the channel! 👋
@billy818
@billy818 5 лет назад
ive been stuck trying to understand this reading tru the documentation for ages. 10 mins and i get it this vid is great thanks
@dataschool
@dataschool 5 лет назад
Awesome to hear!
@emuratspahic
@emuratspahic Год назад
Great explanation! Even though I used pandas a lot, I never really understood how multi-indexing worked. Thank you so much for this tutorial, this helped me out a lot!
@dataschool
@dataschool Год назад
Glad it was helpful!
@amitsardanayt
@amitsardanayt 2 года назад
You made this complicated taboo topic so simple to understand. Worth half an hour i spent
@dataschool
@dataschool 2 года назад
Thanks for your kind words!
@carlomision9546
@carlomision9546 5 лет назад
Clear explanation with examples. This is exactly what I needed to know. You have yourself a new subscriber.
@dataschool
@dataschool 5 лет назад
Awesome! Thanks :)
@HD_157
@HD_157 5 лет назад
The most helpful video I've ever seen about Python. Thanks a lot!
@dataschool
@dataschool 5 лет назад
Awesome! Glad it was helpful to you :)
@markcampanelli
@markcampanelli 5 лет назад
Thanks a lot. Including slice(None) was very helpful.
@dataschool
@dataschool 5 лет назад
Great to hear - you're very welcome!
@shadrinan90
@shadrinan90 7 месяцев назад
Your lesson is sooo simple and clear. I found everything I wanted to know. Thank you
@dataschool
@dataschool 6 месяцев назад
Great to hear!
@owenlie
@owenlie 3 года назад
24 hours struggling to get the answer, and the answer just single line *unstack()* method. Thank you so much, I'll subscribe immediately!
@dataschool
@dataschool 5 месяцев назад
Great to hear!
@mahyarazad
@mahyarazad 4 года назад
Thank you gazillion, you are the best! I finally understand the functionality of the Groupby method!
@dataschool
@dataschool 4 года назад
Great to hear!
@adarshrai22
@adarshrai22 Год назад
This is really a value adding video for data specialists. Thanks a lot for this brother.
@dataschool
@dataschool Год назад
Glad it was helpful!
@Brocollipy
@Brocollipy 3 года назад
Thanks so much for this video. Really great to see a range of options with such clear advice.
@dataschool
@dataschool 3 года назад
You're very welcome!
@shabdarana2848
@shabdarana2848 5 лет назад
Thanks. This topic had been bugging me but you make it seem really simple.
@dataschool
@dataschool 5 лет назад
It is complex, but glad you are feeling good about it now!
@Mo3azSolomon
@Mo3azSolomon 9 месяцев назад
Thanks a lot for this simplified Explanation 😍
@dataschool
@dataschool 8 месяцев назад
You're very welcome!
@wiresandwavessolutions4723
@wiresandwavessolutions4723 4 года назад
Thanks for making this multi-index topic so easy for all of us. I understood, pivot table perfectly after this video. I am having difficulty in the following : I want to select all Symbols and calculate the difference between the closing price of all SYMBOLs from 4th to 05th .
@minglee5164
@minglee5164 3 года назад
Thanks for the excellent tutorial, I have been confused by this issue for a long time.
@dataschool
@dataschool 2 года назад
Glad it helped!
@shrishailgajbhar3191
@shrishailgajbhar3191 Год назад
Thanks for helping us out with multiindexing in pandas...Cleared my daunting confusions about it.
@dataschool
@dataschool Год назад
Happy to help!
@mdfaiz4583
@mdfaiz4583 5 лет назад
u make things very simple....easy to understand ....thanks man
@dataschool
@dataschool 5 лет назад
You're welcome!
@raminsadeghnasab9310
@raminsadeghnasab9310 Год назад
Finallllly, I understand how the multi-index works. Thanks man.
@dataschool
@dataschool Год назад
You're welcome!
@mohammadkaifi
@mohammadkaifi 3 года назад
This is the solution I was looking for. Thanks!
@dataschool
@dataschool 5 месяцев назад
You're welcome!
@MagicmathmandarinOrg
@MagicmathmandarinOrg 5 лет назад
Thanks for the thorough explanation. Very helpful.
@dataschool
@dataschool 5 лет назад
Great to hear!
@mkoc2071
@mkoc2071 5 лет назад
Thanks for additional topics. Thanks for your time.
@dataschool
@dataschool 5 лет назад
You're welcome!
@kilocesar
@kilocesar 2 месяца назад
Great content man! Exactly what I needed!
@dataschool
@dataschool 2 месяца назад
Great to hear! 🙌
@pankajthakkar311
@pankajthakkar311 5 лет назад
This is awesome, thanks for making this Video Kevin.
@dataschool
@dataschool 5 лет назад
You are very welcome!
@ankitsajwan7520
@ankitsajwan7520 4 года назад
Exactly what I was looking for. Thanks :)
@dataschool
@dataschool 4 года назад
You're welcome!
@robertodilisio7535
@robertodilisio7535 5 лет назад
Hi Mark! Great video on a very Syntax confusing matter such as multiindex. Thank you very much!
@dataschool
@dataschool 5 лет назад
You're very welcome!
@jsr7599
@jsr7599 3 года назад
Perfect video for what I needed. Thanks!
@dataschool
@dataschool 3 года назад
You're welcome!
@iloveno3
@iloveno3 5 лет назад
Thank you. I have learned so much from you already...
@dataschool
@dataschool 5 лет назад
Great to hear!
@5a035
@5a035 Год назад
19:50 "slice(None)", I never would have figured that out..... thank you!
@dataschool
@dataschool Год назад
You're welcome!
@reibalachandran4775
@reibalachandran4775 4 года назад
Thanks for the video, I found it very useful and enjoyable to watch :)
@dataschool
@dataschool 4 года назад
Great to hear!
@spiderschwein8g
@spiderschwein8g 5 лет назад
Saved me hours. Thanks!
@dataschool
@dataschool 5 лет назад
Great to hear!
@akashpoudel571
@akashpoudel571 5 лет назад
u r the bestest ever teacher to me.......damn lucid ua guideliness......thanks a tonne sir
@dataschool
@dataschool 5 лет назад
Thank you! :)
@ramvpage
@ramvpage 4 года назад
Excellent Tutorial ! Thank You!
@dataschool
@dataschool 4 года назад
You're welcome!
@ahmadalrashedi5718
@ahmadalrashedi5718 3 года назад
man thank you soooooooooooooo much, I had been stuck on one thing for hours and finally figured it out!! !
@dataschool
@dataschool 2 года назад
You're so very welcome!
@amitgaj1
@amitgaj1 5 лет назад
Thanks Kevin. That's brillant explanation.
@dataschool
@dataschool 5 лет назад
Awesome! Thanks for your kind words :)
@donaldbebel7979
@donaldbebel7979 3 года назад
Very good explanation of a confusing topic. Thanks you.
@dataschool
@dataschool 3 года назад
Glad it was helpful to you!
@rayli2221
@rayli2221 2 года назад
Thank you for sharing this information. I would like to ask if I wanted to get this to do portfolio optimization so calculating expected returns and covariance on multi-index dataframe how would you set this up?
@wildlife_and_drones1320
@wildlife_and_drones1320 3 года назад
Very helpful. I will be back with questions about machine learning.
@dataschool
@dataschool 3 года назад
Great! I've got a free course: courses.dataschool.io/introduction-to-machine-learning-with-scikit-learn
@mxmmalinin
@mxmmalinin 2 года назад
Thank you for sharing your knowledge!
@dataschool
@dataschool 2 года назад
My pleasure!
@SFayed-lg5sp
@SFayed-lg5sp 2 года назад
it's so helpful!!! thank you so much!
@dataschool
@dataschool 2 года назад
You're so welcome!
@ocamlmail
@ocamlmail Год назад
Great explanation, very clear, thank you!
@dataschool
@dataschool Год назад
Thanks!
@rubencrespo1720
@rubencrespo1720 2 года назад
thank you so much, finally I could change from a Dataframe to a multiindex
@dataschool
@dataschool 2 года назад
You're very welcome!
@niva090
@niva090 3 года назад
Thank you, this helped me out so much! I have one question however, I hope you can answer for me here: I have a multi index with ID as the first and Day as the second. Now I want to select all ID's that has more than 6 days of data. How can I pass this logical statement to the selector?
@DeebzFromThe90s
@DeebzFromThe90s Год назад
Thank you sir. I will shoot you a donation if/when my quant journey presents some alpha. You are a lifesaver.
@dataschool
@dataschool Год назад
Thank you so much!
@jerryetc
@jerryetc 3 года назад
Extremely clear, thank you
@dataschool
@dataschool 3 года назад
You're welcome!
@thimirabandara679
@thimirabandara679 3 года назад
Great video! is there a way to must index the column so there is an outer column and inner column? And if so how do you sort that? Because I know sort_column is not a callable function on the datafram
@jhonatangarcia3826
@jhonatangarcia3826 4 года назад
Thanks man! You helped me a lot!
@dataschool
@dataschool 4 года назад
Great to hear!
@user-dp1fz8gr8c
@user-dp1fz8gr8c 4 года назад
Thank you ! Your video is very helpful!
@dataschool
@dataschool 5 месяцев назад
You're welcome!
@umeshdev6403
@umeshdev6403 2 года назад
Thank you. Nice explanation. how to get max values with the dates
@willd0g
@willd0g 3 года назад
Excellent - yes a few 💡 moments indeed. Very good
@dataschool
@dataschool 5 месяцев назад
Great to hear!
@coolzkabhijit
@coolzkabhijit 3 года назад
Thanks for the video! I have a question. How to access a particular index without naming the index, like say without naming 'AAPL'
@dmitku
@dmitku 3 года назад
Very useful as always. Many thanks!
@dataschool
@dataschool 3 года назад
My pleasure!
@szymonk.7237
@szymonk.7237 3 года назад
Thank you very much for this explanation ❤️
@dataschool
@dataschool 3 года назад
You're welcome 😊
@ItsRainingSteak
@ItsRainingSteak 4 года назад
Top shelf content right here
@dataschool
@dataschool 4 года назад
Thank you!
@krishj8011
@krishj8011 2 года назад
Awesome tutorial...thanks
@dataschool
@dataschool 2 года назад
You're welcome!
@chancemeeting9756
@chancemeeting9756 5 лет назад
Super helpful, thanks!!
@dataschool
@dataschool 5 лет назад
You're welcome!
@BookOfMorman
@BookOfMorman 3 года назад
This was great! Thank you!!!
@dataschool
@dataschool 3 года назад
You're so welcome!
@marvinjoyceburgstededijk9749
@marvinjoyceburgstededijk9749 3 года назад
Great video and many thanks. Greetings from Amsterdam
@dataschool
@dataschool 2 года назад
Thanks for watching!
@CNW21
@CNW21 Год назад
As clear as day now , thank you
@dataschool
@dataschool Год назад
You're welcome!
@olenamelnyk2427
@olenamelnyk2427 3 года назад
You saved my life with slice(None) (19:45). Thank you so much.
@dataschool
@dataschool 3 года назад
You're welcome!
@mounah2182
@mounah2182 3 года назад
Thanks for the video but i have a question how to merge columns and indexes in order to not have blank spaces in headers in my html view ?
@ramutillu5005
@ramutillu5005 4 года назад
Thanks alot. Very useful. Can u teach how to use multiple groupby in single expression.
@uncommon_common_man
@uncommon_common_man 4 года назад
great tutorial
@dataschool
@dataschool 4 года назад
Thank you!
@matuwuhou
@matuwuhou 3 года назад
Thanks for the video! Does anyone know if it's possible to select AAPL and 2 dates and MSFT with only one date?
@prathameshmahankal4180
@prathameshmahankal4180 5 лет назад
Thanks a lot for this video!
@dataschool
@dataschool 5 лет назад
You're very welcome!
@semaphoreLock
@semaphoreLock 5 лет назад
this is excellent. thanks
@dataschool
@dataschool 5 лет назад
You're welcome!
@zennicliffzennicliff
@zennicliffzennicliff 5 лет назад
Thank you so much. You are such a talented teacher. Where can I find out how I can structurally learn courses from you? Thank you and greetings!
@dataschool
@dataschool 5 лет назад
Thanks! This post might be helpful to you: www.dataschool.io/launch-your-data-science-career-with-python/
@florianrudowable
@florianrudowable 3 года назад
pandas is awesome and you are too! I cant belive that pandas is free and this video is free. Thank you.
@dataschool
@dataschool 2 года назад
Thank you so much! 😄
@KarlosEPM
@KarlosEPM 2 года назад
Great video! Thanks :)
@dataschool
@dataschool 2 года назад
You're welcome!
@Zielu13
@Zielu13 2 года назад
Thanks. Any advice hot to plot multiindex?
@muralikrishnapolipallivenk2572
Hi Thanks for the video, I learned a lot, I currently struggling with concat, join and merge functions in Pandas, do you have any videos to help me understand it better. Thanks
@dataschool
@dataschool 5 лет назад
I cover concat in this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-15q-is8P_H4.html But you should watch this video first: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-OYZNk7Z9s6I.html
@ebena.k.8032
@ebena.k.8032 2 года назад
Thank you so much. 'unstack()' has alleviated my headache.
@dataschool
@dataschool 2 года назад
You're welcome!
@liamkholmes
@liamkholmes 3 года назад
Amazing! Thank you
@dataschool
@dataschool 2 года назад
You're welcome!
@Zielu13
@Zielu13 2 года назад
Thanks. Do you know how to change the background color in dataframe.plot graphs ( the background on which there are: labels and graph name). In subplots you can use 'facecolor' param, but it doesn't work in plots generated from multiindex dataframes.
@pankushkukreja3101
@pankushkukreja3101 5 лет назад
Thanks Alot, really a good series for pandas and easy to learn. I would request if you can make a video on Iteration in pandas like iterrows, itertuples and iteritems. Many Thanks In advance.
@dataschool
@dataschool 5 лет назад
I think I cover it in this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-B-r9VuK80dk.html
@ckim0717
@ckim0717 4 года назад
Thank you for the videos! I have an additional question on the slice(None) object you made. Let's say you have 100 more symbols and I don't want all of the symbols but want a specific slice of symbols (for example the all symbols after "CSCO" ) how would I do that? stock.loc[("CSCO": , ['2016-10-03', '2016-10-04']), : ] does not work :(
@willd0g
@willd0g 3 года назад
Slice(some)
@mountainscott5274
@mountainscott5274 4 года назад
Would you normally go through all of this? It seems like one can just stick to groupby() without complicating the syntax for selection or changing the shape of the dataframe. In other words, have you encountered multi-indexing often in the field?
@dataschool
@dataschool 4 года назад
Great question! Personally, I try not to use the MultiIndex very much. That being said, it is still useful to be able to work with it when you have to.
@marklittlewood2418
@marklittlewood2418 5 лет назад
Nice presentation Kevin well done. How about accessing a range of dates for say apple, date:date ?
@dataschool
@dataschool 5 лет назад
I'm not sure if that can be done using the colon, I'd have to check...
@zhenghaizhou6085
@zhenghaizhou6085 5 лет назад
Great video!
@dataschool
@dataschool 5 лет назад
Thanks!
@lathaprabakarg
@lathaprabakarg 5 лет назад
very nice.. so useful
@dataschool
@dataschool 5 лет назад
Thanks!
@ilianleandrolopezsepulveda1829
thanks! can u make a plot with multiindex?
@srosalia
@srosalia 4 года назад
Thank you!
@nguyenduyta7136
@nguyenduyta7136 2 года назад
Hi , thanks for great videos, I have a question , If I have many csv file in one folder each file is data frame of symbol of stocks (APPL.csv, BA.csv, CSCO.csv, MSFT.csv,....) but I just wanna pick a few of them but not all, instead of I use pd.read_csv one by one it's so slow and manually so I would like to use for loop but I still not find out the way. Would you please help me. Thank you
@asdkop7818
@asdkop7818 5 лет назад
Nice video, anyway I would have liked a final section in which you could have explained the usage of logic over the selection on multi-indexes. Eg: what if I wanted to select data for each symbol and for each date except one? Is it possible to use the same logic as .loc with simple indexes? Thanks :)
@dataschool
@dataschool 5 лет назад
Thanks for your suggestion!
@Zielu13
@Zielu13 2 года назад
Thanks. How to filter multiindex?
@venkynew9509
@venkynew9509 Год назад
15:06 indeed… you are eye opener… Also I’m wondering can we able to update cell by condition in side multi index value?
@manikandanpalanivel4941
@manikandanpalanivel4941 2 года назад
I learn something from your video👌but I have some doubts. Kindly explain me for example we are having a issues in our computer in different dates . We need to filter out with O/P as computer type, computer model, windows 10,7,8 , dates in sequence it got fault
@gggganzo
@gggganzo 2 года назад
Well explined. Would be great a video about data visualization with multiindex data structures also
@ridthrahman6637
@ridthrahman6637 4 года назад
How to use range selection (:) while inner and outer row labels are inside tuples? Use slice which probably did not allow range selection. Thank you!
@dataschool
@dataschool 4 года назад
Sorry, it's hard for me to say off-hand, good luck!
@vinaybharath4017
@vinaybharath4017 4 года назад
How would I calculate percent change for each stock in this multi index dataframe?
@dataschool
@dataschool 4 года назад
Not sure off-hand, I'm sorry!