Make Python code 1000x Faster with Numba

Подписаться 30 тыс.

Просмотров 439 тыс.

50% 1

In this video I introduce Numba which can make your python code 1000x faster. Numba is a just in time compiler for a subset of Python and Numpy. The first half of the video is dedicated to a basic intro and to highlighting a number of very common mistakes people make when using Numba. The remaining video show how to use Numba in a real world-ish simulation problem and shows the code running 1000x faster with Numba in both single and multithreaded cases, and concludes with a "reading list" for learning more about Numba.
Twitter: / safijari
Find the notebook here: gist.github.com/safijari/fa4e...
MY OTHER VIDOES:
○ A.I. Learns to play Snake • Neural Network Learns ...
○ 5 Common Python Mistakes: • 5 Things You're Doing ...
○ 5 Amazing Python Libraries: • Five Amazing Python Li...
○ Making Python fast: • Can VSCode be a reason...
○ Learning programming language Julia: • How to learn Julia, a ...
Deeper topics to discover here
numba.pydata.org/numba-doc/de...
Supported python and numpy features
numba.pydata.org/numba-doc/de...
numba.pydata.org/numba-doc/de...
Important differences from python numba.pydata.org/numba-doc/de...
Defining types to compile at definition time: numba.pydata.org/numba-doc/lat...
Function factories numba.pydata.org/numba-doc/de...
Experimental version of jitted classes numba.pydata.org/numba-doc/de...
Debugging numba.pydata.org/numba-doc/de...
Dealing with types numba.pydata.org/numba-doc/de...
Ahead of time compilation for deployment numba.pydata.org/numba-doc/de...
Using approximate fastmath numba.pydata.org/numba-doc/de...
Deeper control of threading via tbb and/or omp numba.pydata.org/numba-doc/de...
Easily put your computation on a GPU!
numba.pydata.org/numba-doc/de...
numba.pydata.org/numba-doc/de...
F.A.Qs numba.pydata.org/numba-doc/de...
#numba #programming #python

Опубликовано:

7 июн 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 684

@gdsvsrv 3 года назад

finally, my print("hello world") can run at the speed of light!

@felicytatomaszewska2934 2 года назад

That's not true

@MartinoxxHD 2 года назад

@@felicytatomaszewska2934 no way

@dv_interval42 2 года назад

This meme is much closer to the truth than people would think.

@hanchen2355 2 года назад

You can’t because it’s IO

@aryanmn1569 2 года назад

@@dv_interval42 how?

@MichaelKudlatheInterested 3 года назад

Amazing, amazing video. As a PhD student, I'm sure you've saved me much more than 8 hrs of compute.

@JackofSome 3 года назад

I expect a co-author ship on all your papers now 👀

@sorry4all 3 года назад

Lol

@senwang1982 3 года назад

@@JackofSome fair enough😂

@alexp6013 3 года назад

I've once had code run for 48h, so I fully believe that it could be true

@username42 3 года назад

@@JackofSome i think he will thank you in the acknowledgement part in his thesis for sure :)

@tronali5703 4 года назад

One thing to note is that Numba really optimizes nested loops. If you say have to call a function a 100 or million times, better to have an outerloop instead that iterates within the function. With parallel turned on, you can see a 1000x increase.

@JackofSome 4 года назад

This actually makes a lot of sense. Thanks for the comment!

@chriskeo392 2 года назад

This is crazy! Awesome

@AlexanderHyll Год назад

Its one of the only ways to run the notoriously slow nested loops in Python (whenever it can’t be vectorized). We had a simulation to run that went from 81 days pure python to 2h (~1000x).

@klioseth4336 3 года назад

Seriously this is brilliant. By applying this to the 3 main functions of a 1500 lines project I went from a 24 minutes runtime to about 20 seconds. that's 70X faster for a day's work (it'd be faster now that I learned the more complicated details). The C++ equivalent of the code does it in about 15 seconds.

@ger15100 3 года назад

Numba really is great. Here is my first experience with it: I have some code that I estimated would take up to half a year to run. With some minor changes to the code for compatibility with Numba, the run time for the same conditions was up to 1.5 hours. This was a game-changer for me as I needed to run this script many times. One of my colleagues also implemented it into his machine learning code and managed to obtain some significant speed bumps.

@irok1 3 года назад

Finally, this is what Python has needed

@raycao88124 3 года назад

Just use C🤣

@JackofSome 3 года назад

cPython is already "just use C" so not quite

@tsunekakou1275 3 года назад

python doesn't need to be faster, it designed to be simple not efficient. there are plenty other programming languages that designed to be efficient like C, C++ that has proved themself. sure they are hard to learn but trying to get python faster is just like reinvented the wheel, learning C++ from scratch in my opinion is much more beneficial in long term than trying to make python faster because essentially you are learning the same things but you won't get far with python.

@tando6266 3 года назад

@@tsunekakou1275 Couldn't disagree more, for a lot of science we are paid to do things, not write c++ libraries to do things. As much as I like C++ it is woefully inadequate for a research setting with deadlines.

@tsunekakou1275 3 года назад

@@tando6266 which points?

@peterbonnema8913 3 года назад

Both Numba and this video are amazing. The video was really really clear and concise. I'm definitely gonna remember Numba for future projects.

@AnthonyThomas_Ant 3 года назад

Liked and subscribed. The length and pace of the video were spot on, especially as you’ve shared the notebook.

@jafarjuneidi7585 3 года назад

I already had a background in Numba but I thoroughly enjoyed this video and learned some useful details as well. I'd be glad to see some more python related videos from you

@valente.victor 3 года назад

This is one of the best programming-related video tutorials I've ever seen. No-nonsense style, very direct and still computationally rigorous. Awesome stuff!

@drepi57 3 года назад

Jack!!! Congrats for this fast, clear and insightful video. This is a game changer for me (I'm a Fortran 77...20** programmer now starting with python). Live long and prosper \\//

@graemeford1333 3 года назад

Ah yay. Thanks for this. I must say Numba is very impressive. What I am most interested in though is applying it to a research setting which you helped a lot with as well! I am just happy that it was designed for scientific applications as well!

@mpilosov 2 года назад

This was wonderfully clear and helpful! Thank you! My PhD involved running a lot of these forward problems under uncertainty, Lost much time trying to parallelize physics simulations and fighting with the GIL. Sometimes it is just worth waiting.

@sirosala Год назад

Excellent video, in one day I implemented 80% of the code and strategies you explain. Very good ! Greetings from Rosario - Argentina.

@boyuanchen4997 2 года назад

Simple, effective, to-the-point! Encountering a topic modeling problem that requires a lot of steps of training, I was about to try cython which would have taken a huge amount of time. Numba turned out to be ENORMOUSLY SIMPLER and equally effective. Many thanks to numba developers and Jack!!

@kaandonbekci6414 3 года назад

Similar to other commenters, I have to say this is one of the best RU-vid tutorials on writing fast Python code. Great buildup to the mic drop moment, amazing progression. Jack, I hope you make it big in the CS-tutorials RU-vid space and have a great day.

@JackofSome 3 года назад

Thanks for the kind words. Videos have been recommended a lot lately so fingers crossed.

@TimmyTam 3 года назад

I really thought the title was clickbait. It wasn't. This is honestly amazing, thanks for showing us this :D

@dinoscheidt 2 года назад

Awesome video. Simply forwarded this to two of my junior devs and they completely got it now. THANK YOU!

@jackrdye 3 года назад

Why didn't I find this sooner. I am still so glad I found it will 100% use this in the future. Literally had a situation identical to ur example a month ago needed to simulate over 160,000 situations. Was taking 11+ hours to run. I assumed there were better ways but I haven't had a lot of experience. Thanks for this very clear explanation

@nitsanbh 2 года назад

This is one of the best intro/tutor videos I’ve ever watched on any subject

@lordtejas9659 3 года назад

You deserve my like. Nice work sir. Now I can show to my friends how a optimizated and we'll written code looks like. Numba.

@jmnunezd1231 4 года назад

One of the most awesome Python tutorials I have seen. The narratives, the examples and the way you explain it is just awesome, 5 starts to you man! keep on doing it!

@journalperen 4 года назад

True

@saurabhpareek3310 3 года назад

I am in tears to see my code run so fast, thank you so much for this video

@user-bf4kb5hm4n 3 года назад

Thanks for your introduction! This really helps me. I have been working with Numpy but it is too time-consuming. Working with Numba is really fantastic, my Python module runs more than 1000x faster than the original one.

@Xaminn 3 года назад

Recursive Fibonacci will truly get ya.

@joshuatyler4657 2 года назад

I implemented it in my code, and it took around an hour worth of computation and put it down to 30 seconds. Amazing stuff.

@JackofSome 2 года назад

Nice nice!

@joshuatyler4657 2 года назад

@@JackofSome One funny thing that I found is that you can’t create numpy.linespace or numpy.arrange arrays with anything besides integers. I had to create an integer array, then divide it by 2 to make it a float32, and then populate it with a for loop to get the actual array I wanted. Also, numpy.zero arrays have to be initialized outside of the function and passed as an argument instead of being initialized in the array. But the extra work is worth the speed up.

@raccoonteachesyou 3 года назад

Damn, I love your Python videos ! Very interresting, you talk about stuff other don't. Thx man !

@jakubkahoun8383 Год назад

This is the best Numba video, short , good examples etc.

@MisterFresh022 2 года назад

What an amazing video. I'm a student in Computer Science and we just did a week ago an introduction to Numba. And you sir not only explained to me the concepts in a positive astonishing way, but also gave me the desire and the excitement to apply it on all my old and future codes. Thanks you ! PS : Is there more videos where you're explaining the others topics ( convolution, neural networks .. )

@JackofSome 2 года назад

Great. I hope you share it with your class :) Unfortunately no other videos on Numba

@Blooddarkstar 3 года назад

Very well explained and saved me from some mistakes. You deserve way more subscribers!

@preyasurakshit3405 2 года назад

I cannot thank you enough for this video. I made a simulation of Gray Scott's Reaction diffusion in python, but it was painstakingly slow. After using numba, it now runs quite fast. Earlier it took nearly about 1 hour to render just 100 frames at a mere 400x400 resolution, now it renders 100 frames in 45 seconds, in 1280x720 resolution. Thank you very much for this video.

@JackofSome 2 года назад

That's amazing to hear! Glad it could be helpful. Would love to see the result.

@JackofSome 2 года назад

If you're just trying to do convolutions then you might want to look into numba's own way for doing them. I forget what it's called but it's optimized specifically for the convolution operator

@eddantes_ 3 года назад

Great video. The fact that you are sharing the jup notebook is amazing. Thanks a lot. Please consider making a follow up video with more complex applications of numba. For example, integrations!

@id104335409 3 года назад

I nod my head with approving noises as if I understand what is going on.

@andylib 3 года назад

This was awesome !! Sooo interesting, straight up going into this

@gabrielcesar568 3 года назад

Grad students from around the world are thankfull for this video.

@dishapandey96 3 года назад

Your way of explanation is very nice. Excellent video :)

@dmgeo 3 года назад

Thank you for your video. I work with satellite image processing and we are used to spend hours waiting for a particular processing task to finish... I for sure will be trying to use Numba

@vishawashkumar5159 3 года назад

This needs much more views.. This is a revolution..

@b992152 4 года назад

Great video mate. Really enjoyed it

@janewan1196 2 года назад

Thank you so much! I was trying and failing to turn to C++ for number crunching, but this has just saved me!

@dadogh 3 года назад

That's an awesome video. Thank you! I've been running orekit scripts which took me ~4h, and with numba there are taking ridiculously 2min!

@JackofSome 3 года назад

Oh wow those are amazing gains! Glad I could help

@shahnawazazam 3 года назад

Just make sure your cpu cores arnt too angy at you

@RealLionTion 3 года назад

YES, I LOVE PYTHON AND THIS IS PERFECT THANK YOU MATE

@evertonsantosdeandradejuni3787 2 года назад

Holy shit, I'm sold. Examples on point, mic indeed touched the ground. Thanks for the tip

@syoudipta 8 месяцев назад

After being skeptical about this library for a while, I've gotten used to it. I've watched many tutorials and read many articles and documentation. After weeks of determination to learn this technique properly,... I have become, comfortably Numba. 🤟

@Meodoc 2 года назад

I really liked that you stayed real and just said that you aren't qualified to comment on why the python loop with the numpy array took so much longer to execute than just saying something with the risk of talking bullshit and spreading misinformation in order to sound competent, a behaviour I have seen way to often online and which bothers me quite much. Very great, clear tutorial, truly one of the best python tutorials I have seen so far on RU-vid. Keep it up! :)

@JackofSome 2 года назад

Nice. Yeah it's always better for me to just say "I don't know" when I don't know :)

@Meodoc 2 года назад

@@JackofSome Love it 🙌

@nahiyanalamgir7056 Год назад

Amazing tutorial and Numba is a great software. I hope Python officially will one day ship with a compiler because the community got so huge that people often write performance-intense code in Python that goes against its nature. Moreover, it'd be amazing to ship a Python product statically compiled without gigabytes of libraries!

@MrRyanroberson1 3 года назад

this is absolutely perfect because i just found this while running a really oddly intensive random number process (something like one million terms per second? i need billions) and numba sped it up by a factor of 100 halfway through the video, and with njit it didn't actually speed up any further, it just got a more consistent time (randomly jit is 10% faster or slower depending i guess on the random values

@SomeFreakingCactus 2 года назад

These are some seriously impressive results. I wonder if this could give Python a niche in indie game development.

@SumitSingh-pu9hd 2 года назад

Your video was that amazing and well executed that I was hypnotized to hit subscribe and like. Keep it going.

@frodobaggins3974 3 года назад

11:45 - try this: np.zeros_like(input_list, dtype = np.int64, order='C') My experience with numba - it works great, but types have to be very precisely defined for every single object. Otherwise nopython will fail, and where numba sucks is generating traceback messages - in the case of complex functions it will take forever to debug. So type whatever you can - it is not a 'Pythonic' way of thinking and writing, but in this particular case it is necessary. Anyway, great introduction!

@JackofSome 3 года назад

Thanks for the tip, and best of luck on your journey to mount Doom. We're all counting on you.

@0.Andi.0 2 года назад

Hi, I have a question, how can I declare like a type so numba can compile it? For example I have a vector3 class that cannot compile with numba (class with x,y,z components)

@mariomuysensual 2 года назад

@@0.Andi.0 use numpy arrays instead

@dydufjfbfnddhbxnddj83 2 года назад

I didn't understand anything as a beginner to Python (after learning C,C++ and Java) but it looks really cool man. I will use this for python, especially in competitive programming

@Shane1994322 3 года назад

This video is concise and clear. amazing. Thank you!

@mkthakral 2 года назад

Fist visit on this channel, but you are the best ;-)

@MichaelBrown-gt4qi 3 года назад

This is amazing. This completely changes python for me.

@shkhrvarshney 2 года назад

Awesome tutorial! Thank you so much. I shared the video with my old Intro to Parallel computing professor. Hopefully he sees the email coming from non UW email id lmao. Video would be pretty helpful to any new(&old) students of the course.

@hunt9556 3 года назад

Nice tutorial, I'm gonna try numma in my plasma physics simulations

@satish-pokala 3 года назад

Very clean and informative video. Thank you 🙂👍🏻

@thestralspirit 2 года назад

Why wasn't this video in my RU-vid recommendations a year ago? Nonetheless, this is an amazing tool for python. I wouldn't have found it without you!

@muhammadanas1201 2 года назад

I had to submit a project in 2 days (that I had months to do but waited until the last minute) and in one piece of code, I needed to clean data of 6 million tweets. Regular python code showed runtime of 400+ hours (I used tqdm library for it). I got cold feet thinking I’m going to miss the deadline, but finally got the idea to search for ways to make the code faster. Found this library out on the internet and using it, I was able to run that piece of code in JUST 10 MINUTES. This library saved my project that day 👌.

@JackofSome 2 года назад

Numba: enabling procrastinators since 2012 😂

@ecielyang9719 2 года назад

I have struggle for Numba for 3 days. This Vedic is really clear

@maxmorehead1569 4 года назад

Really great video!

@KANGAR1982 3 года назад

Great video! subscribed instantly after seeing it!!!

@palermodpr Год назад

Awesome video man!

@culperat 2 года назад

Love the video, I'll definitely have to try this on some future projects.

@ianh5537 3 года назад

Awesome, thanks! Definitely going to start leveraging mumba!

@MrProzaki 3 года назад

oh nice, you just gave me lots of work to do xDDDD time to change the whole project code... thanks :'D , no really i mean it THANK YOU it was a great vid will def look into that 10% for sure.

@trizzzzlez 2 года назад

I came back to say a huge thank you. currently doing my Phd. my code was to run 300k iterations it took like 3days+. your short video changed everything, i took few hours to rewrite the code to suit numba as you explained. boom 2hrs i have results on the screen instead of 3days+

@JackofSome 2 года назад

That sounds amazing. Please be very careful and do consistency checks between the two codes. Don't wanna find out there was an issue when you're writing your dissertation 😅

@trizzzzlez 2 года назад

@@JackofSome yes I will jack thank you.

@adityanjsg99 Год назад

Presentation level is ingenious!

@mengzhu8385 2 года назад

Amazing video! I'm starting to think to give numba a try.

@PandemicGameplay 3 года назад

This is revolutionary. I always wanted a library to allow these but without the hassle of installing something like Cython, etc. Just a good ol' pip lib that can allow my code to speed up orders of magnitude. A nice bonus is the ability to release the GIL.

@smjure 3 года назад

Great video, learned a lot in 20 min, thanks!

@MVHiltunen 4 года назад

numba is really sometimes absolutely stunning in it's acceleration per extra work.

@JackofSome 4 года назад

That it is friend.

@AJ-et3vf 2 года назад

Heyy man! This is a very nice tutorial for numba! There are only few numba tutorials out there and I hope you could make a bit more. For example, I became interested in using numba to speed up my Runge-Kutta Algorithms that I coded myself, but I couldn't make it work. I would try to apply it on the for-loop part but it would throw an error that numba doesn't now the class function, something like that. Hopefully, an extensive numba tutorial would be made in the future.

@dyrevennen 2 года назад

Thanks for posting the video. I learned something new, and useful.:)

@Prithvidiamond 4 года назад

Very nice, I like this!

@harshraj22_ 4 года назад

Awesome ! Thanks for this video.

@michaelc3650 2 года назад

Damn this is really cool, just came across it and it should definitely help in speeding up machine learning or ai projects

@jamesmarchetti3305 Год назад

My new fav channel. Have you done a video talking about your Linux setup? If not I would love to see it/could you share some info?

@TunkFey 2 года назад

Thanks for sharing. This video is really a treasure!

@nacnud_ 3 года назад

Great work dude!

@pranavramesh7052 3 года назад

thank you i had a code which took like a whole 5 minutes to start running and now it is much faster

@Rotem_S 3 года назад

As someone who tried numba and was disappointed I appreciate you showing possible causes and solutions for that

@luisenciso1762 3 года назад

Very helpful video, many thanks :)

@nobody2937 2 года назад

Awesome explaination... Thank you ...

@pubgplayer1720 2 года назад

Nice, easy to understand.

@amir650 Год назад

Thank you for this excellent presentation.

@fadillahmaani7413 4 года назад

Great video, thank you very much ! I hope that you will make numba tutorial for class too.

@Epiphonication 3 года назад

that's fantastic, I used to resort to using my uni's HPC cluster to get some more speed but now it's so much faster even on my old thinkpad, thank you so much!

@JackofSome 3 года назад

Wow ... If you're able to go from HPC cluster to a laptop those are ... Those are some massive gainz

@Epiphonication 3 года назад

@@JackofSome truly massive! I'm going to try numba + HPC soon, let's see how fast that goes

@JackofSome 3 года назад

Them: nooo you can't write heavy workloads in pure python You: haha, numba go brrrrrr

@Epiphonication 3 года назад

@@JackofSome exactly my thought :D

@Guilfordust 4 года назад

This was great, thank you!

@vanviethieuanh2237 2 года назад

your tutorial is amazing, thanks!

@foodsscenes5891 3 года назад

Awesome work!

@sibyjoseplathottam4828 3 года назад

Thank you! You saved me a lot of time.

@Anonymous9683 4 года назад

Brilliant video, thank you

@sasimitra5871 3 года назад

I don't know why I need this. I'm still a High School student doing python for fun. But imma keep this tucked away in my mind. I have a feeling I'll need it soon enough.

@CrazyFanaticMan 3 года назад

When your'e working on projects that needs the program to do things fast, you want C like performance because low level languages like C/C++ are about as fast as you can get. Python is not a low level language. It's a high level dynamically typed language. This means it's pretty slow. So even though it's pretty fast to write a python program because it's basically the same as writing in English, you're giving up the actual execution speed (performance) in order to keep that fast typing speed (developer performance). Numba solves this problem by keeping the Python typing that we all love and getting the C like performance that is needed for projects where speed is very important.

@eskees9498 3 года назад

@@CrazyFanaticMan so I'm also in high-school just doing python for fun. Is number useful for smaller projects or only for large projects. My biggest code is only about 80 lines of code, but it uses my Webcam to track hands. Would it be worth it to implement numba for this low amount of code?

@CrazyFanaticMan 3 года назад

@@eskees9498 It doesn't matter how many lines of code you have, whether it's 80 lines or a billion lines. What matters is the problem you are trying to solve. If for example you say "I want my webcam to not only accurately detect the location of my hands but I also want my webcam to track my hands if I move them really fast". I would imagine a webcam tracking hands that are moving really fast would require very fast performance speed in order to keep up. So in this case, numba may help you a lot. It all really depends on what you want and what your expectations are. The size of your program does not matter, numba can you help you whether you're program is big or small

@eskees9498 3 года назад

@@CrazyFanaticMan thank you, I appreciate the info