Тёмный

You might never need Pandas again... 

Isaac Harris-Holt
Подписаться 15 тыс.
Просмотров 7 тыс.
50% 1

Опубликовано:

 

25 окт 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 55   
@houstonbova3136
@houstonbova3136 6 месяцев назад
Stops using pandas ~2 years ago for polars and it’s only gotten better since.
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
I'm glad you're a fan! I absolutely love it
@GoldenBeholden
@GoldenBeholden 6 месяцев назад
I have been sleeping on this one because the data scientists at my job and in academia wouldn't feel comfortable switching for what are ostensibly "software engineering" reasons. Maybe this is the wake-up call I need to just go for it.
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Do it! Force them into it 😅
@BabakFiFoo
@BabakFiFoo 6 месяцев назад
Until there is a geopolars with proper functionalities, I have to stay on pandas realm.
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Totally fair!
@zyelidarmeggedon2111
@zyelidarmeggedon2111 6 месяцев назад
There is geopolars. Not "production" ready, but around technically
@KManAbout
@KManAbout 6 месяцев назад
Can't you just do most of the stuff with polars and then port them over to pandas?
@BabakFiFoo
@BabakFiFoo 6 месяцев назад
@@KManAbout then there is no reason to do the stuff in polars. Spatial data manipulation is different. Aggregations, summaries and manupilations are not part of pandas, but they are part of geopandas.
@BabakFiFoo
@BabakFiFoo 6 месяцев назад
@@zyelidarmeggedon2111 I know! But you know the how nasty spatial data is. Production ready tools still have a lot of trouble.
@Sairysss1
@Sairysss1 6 месяцев назад
Dude, I almost got an epilepsy from all those GIFs.
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Apologies! Not everyone will enjoy this style of video, and I appreciate the feedback!
@crossbow170
@crossbow170 6 месяцев назад
Great video-very informative and well-put-together! The only suggestion I have is about the gifs used. They seem a bit too dynamic and short, making them jarring and hard to process before they loop again. Maybe try using longer, smoother gifs to enhance the viewing experience without distracting from the content. Everything else, especially your writing style, is spot on. Keep it up!
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Thanks for the feedback! I'll take it into account :)
@user-et9by5uu9e
@user-et9by5uu9e 6 месяцев назад
im no data scientist but can somone tell me why the like 20ms saved matters that much if the time for such large sets of data is so low does it really matter?
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
These aren't particularly massive datasets, and 20ms doesn't really matter for exploratory stuff, but if you're doing loads of transformations regularly, cleaning up data etc., perhaps running in a production setting, it can make a big difference. Also, Polars allows you to analyse datasets that won't fit into memory, meaning you don't necessarily need to rent a high memory cloud machine, which are always expensive
@ElFlor95
@ElFlor95 6 месяцев назад
In my last project I had to wrangle 5GB worth of unstructured data into a tabular format. My initial pandas attempt broke down due to running out of memory one hour into it. Polars completed it all in 15 minutes, without breaking a sweat.
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Nice! I do think Polars is great
@robosergTV
@robosergTV 6 месяцев назад
what? You measure difference in %, not absolute time. Polars here was 2x to 5x faster. wtf.
@romankovalful
@romankovalful 6 месяцев назад
I'm not a data scientist but I occasionally use Pandas. What I've always wondered though is why don't people just load their data into a normal Postgres database and query it with plain SQL? You get a lot of features with minimal work.
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Some data isn't necessarily suited to that, or you may need to present it e.g. in a Jupyter Notebook. Also, many data scientists aren't familiar with setting up DBs etc.
@moolavar9452
@moolavar9452 6 месяцев назад
Sqlite sufficient. ​@@IsaacHarrisHolt Reason is we need that data object in hand to play with it easily in Jlabs 😂.
@robosergTV
@robosergTV 6 месяцев назад
what? No, thats not how it works for DS
@Bozebo
@Bozebo 5 месяцев назад
Lack of knowledge and a need to use buzzword tools as much as possible. Basically everything I do now is just postgres with some decoration to make it do what I'm needing xD For almost anything with data postgres already has the answer and is optimal.
@IsaacHarrisHolt
@IsaacHarrisHolt 5 месяцев назад
@@Bozebo Postgres is good, but there's the upfront setup cost, and it can be more difficult for people who aren't familiar with SQL. For a lot of things though, it's great. Especially when you add a columnar engine
@DreySF
@DreySF 6 месяцев назад
So basically it is R data.frame or data.table in dplyr and tidy verse but with rust in python 😊
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
I'm not familiar with any of those, but very possibly!
@ArnabAnimeshDas
@ArnabAnimeshDas 6 месяцев назад
dplyr but better in every way
@risebio
@risebio 6 месяцев назад
Where were all these useful tools a few years back when I was doing my diploma work. I had a 2 sets of chem structures. 1 set with 500k elements 21 GB and another one with 150k. I was spending from 4 to 7 hours to finish the script. And if I wrote smth wrong (i did it quite often and get stupid results) I needed to spend another 15 to 30 minutes to find mistakes in my code and wait another 4-7 hours for my laptop to finish calculations. I was so done with python that time that i was eager to do the same in go but the result was almost the same then I accepted my fate and started the script, went to bed, woke up after 7 hours and that was like I finished my work XD
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
I know the pain! One of my physicist friends had the same issue, so we ended up using multiprocessing to parallelise the work and run it on my 12 core desktop instead of her 4 core laptop. Took the runtime down from 17 hours to 40 minutes or something ridiculous
@Charles-m7j
@Charles-m7j 5 месяцев назад
Switched to Polars about 19 months ago and never looked back. Pandas is awful. Spark is sometimes a necessary evil, but I believe that the folks behind Polars is looking to release a distributed compute framework soon.
@IsaacHarrisHolt
@IsaacHarrisHolt 5 месяцев назад
Oh that's interesting! I didn't know that
@Charles-m7j
@Charles-m7j 5 месяцев назад
@@IsaacHarrisHolt yeah they raised $4m from Bain IIRC.
@IsaacHarrisHolt
@IsaacHarrisHolt 5 месяцев назад
Awesome!
@samarnagar9699
@samarnagar9699 6 месяцев назад
Ofc it's made in rust
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
All the best things are 😉
@_tsu_
@_tsu_ 6 месяцев назад
BLAZINGLY FAST
@pedrogorilla483
@pedrogorilla483 6 месяцев назад
What about RAPIDS cuDF? Does it support it? Would be interested to see how it compares if you use GPU for processing data.
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
I'm not 100% sure, but I don't think Pandas does either, right? Not out the box
@marco_gorelli
@marco_gorelli 6 месяцев назад
Nvidia and Polars are collaborating on GPU support, check the Polars blog, there was an announcement there
@EarlZMoade
@EarlZMoade 6 месяцев назад
Polars have announced they are integrating cudf support directly.
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Awesome, thanks!
@samarnagar9699
@samarnagar9699 6 месяцев назад
How much time do you think will take this new tech to be adopted in market at scale
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Great question! Lots of tooling is built around pandas, and it's a slightly different way of thinking, but it's already in production use in a few places (including where I work!) so it's definitely on track
@samarnagar9699
@samarnagar9699 6 месяцев назад
@@IsaacHarrisHolt this type of thinking is what I've came to ask myself looking at the web dev world new run time compiler bundler framework version whatever but will it effect your personal work experience it's good that it's in use where you work. The Python world is a very different world than webdev which is a good thing in this scenario. Another point to notice is that all the tutorials for this new tech will assume you know pandas and will be an migraine to new programmers
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Potentially, but Polars and pandas are quite different in my opinion, so there's always going to have to be some explanation of the basics. People should probably learn SQL before learning either anyway
@samarnagar9699
@samarnagar9699 6 месяцев назад
@@IsaacHarrisHolt really sql before both
@takudzwamakusha5941
@takudzwamakusha5941 6 месяцев назад
What are the specs of your computer?
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
The CPU is a Ryzen 9 3900X :)
@dhaval1489
@dhaval1489 6 месяцев назад
My first choice is Polars, only if I get stuck I switch to Pandas, because your are sure get your solution on net with Pandas
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
This is true. Community support is much better for pandas atm
@fabricehategekimana5350
@fabricehategekimana5350 6 месяцев назад
👏👏👏
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Thanks! Hope you found it helpful
@Sebastian-gu6wr
@Sebastian-gu6wr 6 месяцев назад
Yeah! I want Pandas 🐼 extinct
@IsaacHarrisHolt
@IsaacHarrisHolt 6 месяцев назад
Sounds like there's some trauma there 👀
Далее
Kaspa KRC721 - O que devemos esperar?
10:55
Comprehensive Course on Python's Matplotlib
57:35
Просмотров 12 тыс.
Three NEW MAPS in Update 0.31.0 Nightmare | Standoff 2
01:48
UFC 308: Пресс-конференция
35:18
Просмотров 489 тыс.
Compiled Python is FAST
12:57
Просмотров 114 тыс.
Create Robust Web Apps with Gleam and Lustre
12:32
Просмотров 37 тыс.
Google Drive hates developers now
23:56
Просмотров 87 тыс.
Why I Don't Like Singletons
29:05
Просмотров 86 тыс.
The Ultimate Guide to Gleam Concurrency
13:36
Просмотров 20 тыс.
25 Nooby Pandas Coding Mistakes You Should NEVER make.
11:30
this vulnerability shouldn’t even exist
14:33
Просмотров 209 тыс.
Using Rust to write scalable Python APIs
6:57
Просмотров 13 тыс.
The cloud is over-engineered and overpriced (no music)
14:39
Three NEW MAPS in Update 0.31.0 Nightmare | Standoff 2
01:48