Тёмный

Python Multiprocessing Tutorial: Run Code in Parallel Using the Multiprocessing Module 

Corey Schafer
Подписаться 1,3 млн
Просмотров 777 тыс.
50% 1

In this video, we will be learning how to use multiprocessing in Python.
This video is sponsored by Brilliant. Go to brilliant.org/cms to sign up for free. Be one of the first 200 people to sign up with this link and get 20% off your premium subscription.
In this Python Programming video, we will be learning how to run code in parallel using the multiprocessing module. We will also look at how to process multiple high-resolution images at the same time using a ProcessPoolExecutor from the concurrent.futures module. Let's get started...
The code from this video can be found at:
bit.ly/multiprocess-code
Threading Tutorial - • Python Threading Tutor...
List Comprehensions Tutorial - • Python Tutorial: Compr...
Exception Handling Tutorial - • Python Tutorial: Using...
Pillow (PIL) Tutorial - • Python Tutorial: Image...
✅ Support My Channel Through Patreon:
/ coreyms
✅ Become a Channel Member:
/ @coreyms
✅ One-Time Contribution Through PayPal:
goo.gl/649HFY
✅ Cryptocurrency Donations:
Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot
✅ Corey's Public Amazon Wishlist
a.co/inIyro1
✅ Equipment I Use and Books I Recommend:
www.amazon.com/shop/coreyschafer
▶️ You Can Find Me On:
My Website - coreyms.com/
My Second Channel - / coreymschafer
Facebook - / coreymschafer
Twitter - / coreymschafer
Instagram - / coreymschafer
#Python

Опубликовано:

 

19 сен 2019

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 930   
@coreyms
@coreyms 4 года назад
Hey everybody! I hope you find this video helpful. I'm finally moved into my new home, so I should be back on a schedule of regular releases very soon. Hope everyone has a great weekend!
@deadfool6105
@deadfool6105 4 года назад
Please make a video on asyncio too..
@param8378
@param8378 4 года назад
@@deadfool6105 yes yes
@beastgenv
@beastgenv 4 года назад
Congrats on the move
@wizard-28
@wizard-28 4 года назад
Corey, can you make jupyter notebooks on these tutorials also. By the way congratulations!
@admintest9291
@admintest9291 4 года назад
Bless you man!
@vicstan5983
@vicstan5983 4 года назад
Why do people give thumbs down to someone who teaches for free? And a small detail ... the man is a grandmaster! Thank you sir for doing this. Greetings from the Netherlands
@bartekz88
@bartekz88 4 года назад
There are jealous because they're programming in Cobol.
@darwinvinci7744
@darwinvinci7744 4 года назад
I think it's the Australians. You see, they have everything upside down there, when they thumbs up, it's thumbs down for us. Corey is the best.
@Bubelaschka
@Bubelaschka 4 года назад
I think it's some kind of obsessive compulsive disorder.
@josephsagotti8786
@josephsagotti8786 4 года назад
~0.78% of people who voted, thumbed down down.
@kyleyoung4974
@kyleyoung4974 4 года назад
.. because its not working for them.. so they wasted their time getting 'taught' something that didn't work. .hence thumbs down.
@ginadi9733
@ginadi9733 4 года назад
There are courses that take thousands of dollars for teaching what you're teaching. you're teaching very well and you give it free. Respect.
@gytisbl
@gytisbl Год назад
31:12 Mr. Schafer finishes with the theory and goes into the real-world example with processing images 35:37 implementing multi-processing to single thread/process operation 40:04 easy switch from Processes to Threads Love Your videos Mr. Schafer. Especially Your communication skills. No BS, just clear communication. I also love that You don't edit out mistakes, and summarize after several steps. Thank You for putting the effort and sharing.
@chinpokomon_
@chinpokomon_ 4 года назад
the longest deja vu in my life so far
@sensonpan7994
@sensonpan7994 4 года назад
same, lmao
@saikhamhein
@saikhamhein 4 года назад
Same LOL
@1OJosh
@1OJosh 4 года назад
I swear
@KhoiThinh
@KhoiThinh 4 года назад
I needed to check github repo first to confirm that I was not watching the same tutorial yesterday ;))
@frederikwaltherliberothchr757
@frederikwaltherliberothchr757 4 года назад
Hahaha! Same
@anhurtjv
@anhurtjv 2 года назад
I just started an internship in Germany and you have no idea how helpful these classes series are to young people like me that want to succeed in the professional world of today :)
@ronnetgrazer362
@ronnetgrazer362 4 года назад
It's one of those tutorials that has you reaching for the like button several times during the video, and once more after it's ended.
@krystianfeigenbaum238
@krystianfeigenbaum238 3 года назад
Well, then let's hope this urge posesses you an uneven number of times or it would be unfair to the author.
@carspeed50km
@carspeed50km 4 года назад
@Corey Schafer, Thank you very much for taking the time to make this video. I have written my first-ever parallelized code and the speedup is invaluable. For years I was apprehensive about learning parallelization methods because they seemed very complicated. You, my friend, have opened the flood gates. This work puts me one step closer to completing my Master's.
@tebbythomas1809
@tebbythomas1809 4 года назад
Another wonderful video. Thanks to your videos I've gained an understanding of how to build and deploy a Django web application, using matplotlib for data visualisations, python multi-threading, python-multiprocessing, exception handling, python unit test cases and more. Your content is consistently excellent because of how practical and well thought out your videos are. Extremely useful! Please keep making these videos! Thanks a lot! You're helping a lot of people with this high quality content!
@thedrei24
@thedrei24 4 года назад
oh man, you don't even know how long I was looking for a video on multiprocessing.
@theteleportercell6749
@theteleportercell6749 2 года назад
I have been looking for videos like this for like over 1 year, and today I randomly saw your video. Very helpful, easy to understand, thanks a lot Corey!
@ItzSenaCrazy
@ItzSenaCrazy 4 года назад
This video came on time. I work with AI with python and, at university, I'm seeing operating system. Now I can join OS and python for do faster algorithms. Thanks Corey. I really like your videos!
@dhurda471
@dhurda471 2 года назад
This has been incredibly helpful for me and my team. I'm a network engineer working on multiprocessing SSH connections and this made everything make sense. Definitely subscribed after seeing this.
@snoopyjc
@snoopyjc 4 года назад
Thanks Corey! Next please do async!!
@AnShett
@AnShett 4 года назад
Yes, this is concept I haven't used but would really like to learn about.
@Skaxarrat
@Skaxarrat 4 года назад
Yes please, async and when to use async/asyncio/threading/multiprocessing.
@user-be8ud2qd2d
@user-be8ud2qd2d 4 года назад
Yes would strongly appreciate this
@folorunso
@folorunso 4 года назад
Yes, async will a perfect next. Thanks Corey for the handwork.
@ashfaquechowdhury1343
@ashfaquechowdhury1343 4 года назад
Yes please and then about locks,race conditions and queues as well. Thanks a lot.Loved your explanation.
@KurtSchwind
@KurtSchwind 3 года назад
Seriously learning more from your vids on Python than the Python 'class' I took one time. Clear and helpful. Everytime I watch I learn something new.
@yuzhu2547
@yuzhu2547 3 года назад
You're a grandmaster. I really appreciate you put all stuff in ONE video. So easy to navigate. Thank you!
@eduardolpz386
@eduardolpz386 4 года назад
Great video! If you are running this in Windows and a later version of Python (I'm running 3.7), you need to run all multiprocessing under main(), then run if __name__ == '__main__': main() Make sure the do_something() function is declared outside main().
@yoyoliyang
@yoyoliyang 4 года назад
current
@caseymcmahan4834
@caseymcmahan4834 2 года назад
Thank you! Just the fix I was looking for. This is apparently true of macs as well
@Svkenapps
@Svkenapps 2 года назад
You fixed my Programm🤝
@drygordspellweaver8761
@drygordspellweaver8761 2 года назад
Only the code that spawns the subprocess needs to be guarded.
@thomasmayne2629
@thomasmayne2629 2 года назад
thank you so mutch
@fayfeld5900
@fayfeld5900 3 года назад
Outstanding video! Never learned so much about multiprocessing in python - in just 45 Minutes. Thank you, Corey!
@nguyennguyenkhoi9988
@nguyennguyenkhoi9988 Год назад
when i watch your vidieos i belive that if you can't explain something easy to understand, absolutely nobody on this world can do that. thanks you alot Corey, you make my life easier.
@spacecowboy2483
@spacecowboy2483 9 месяцев назад
This video is gold, every second of it. You are very talented at teaching complex concepts, you can't even begin to fathom how much I appreciate your work. Thanks!
@shawnalexander5704
@shawnalexander5704 4 года назад
Of all the resources out there that I have looked to get a start and grasp on this subject (some cost money), this video is on point!
@erik-martin
@erik-martin 4 года назад
Corey's channel has done more for me in 2 weeks than any other resource I have tried over the last 10 years.
@Mankind5490
@Mankind5490 3 года назад
Corey, thank you so much for such an amazing explanation. Even lingering questions that were happening in my head during the video, you managed to answer them right away.
@neolithian
@neolithian 4 года назад
We need at least a dozen of Corey Schafer in our programming learning life. I wish everyone would follow your method. You are simply the best! God bless.
@raghav-cx3qq
@raghav-cx3qq 3 года назад
I am not in a position right now to donate through patreon. All I can do is watch your videos without skipping ads and give it a like.
@coreyms
@coreyms 3 года назад
That helps a ton too! Thanks!
@bigbangdata
@bigbangdata 4 года назад
Thanks for the smooth and didactic intro to multiprocessing, I've done this a year ago during a fast and furious degree and needed a refresher. I have a case in which this will come in handy although it's hundreds of lines of code, but I bet I can wrap them all into a function to be the executor's target.
@drygordspellweaver8761
@drygordspellweaver8761 2 года назад
A function with hundreds of lines might not be such a good idea lol. Try learning Classes and Methods and I’m sure you can incorporate multiprocessing in a much more encapsulated way
@bigbangdata
@bigbangdata 2 года назад
@@drygordspellweaver8761 right you are, god knows what insanity I was up to 2 years ago.
@drygordspellweaver8761
@drygordspellweaver8761 2 года назад
@@bigbangdata haha must have been a blast
@KrzysztofSpikowski
@KrzysztofSpikowski 3 года назад
best explanation ever, I was really stuck when studying multiprocessing on coursera, could not get my head around, now is all crystal clear, thanks Corey
@TheGoldenPro
@TheGoldenPro 2 года назад
Best Python Multiprocessing Tutorial ever on RU-vid. I love it, you explain so well! Keep it up!
@robertlee6529
@robertlee6529 4 года назад
Just a reminder, on Windows you need to add "if __name__ == '__main__'"
@coreyms
@coreyms 4 года назад
Thanks! I usually run my videos through my Windows laptop before posting just to make sure it's the same for both operating systems, but I didn't do that this time. I appreciate the comment!
@ThePasupathi
@ThePasupathi 4 года назад
@@coreyms Somehow, me too facing errors without using main function. Even after using the if __name__ == '__main__', i got my output in different way . done sleeping 5 second(s)..... done sleeping 4 second(s)..... done sleeping 3 second(s)..... done sleeping 2 second(s)..... done sleeping 1 second(s)..... sleeping 3 second(s).... sleeping 2 second(s).... sleeping 1 second(s).... sleeping 4 second(s).... sleeping 5 second(s).... Finished in 5.5 second(s)
@whoa_that_ruled
@whoa_that_ruled 4 года назад
Just curious (python beginner), where would that go in the script?
@veganath
@veganath 4 года назад
@@whoa_that_ruled import multiprocessing import time start = time.perf_counter() def do_something(): print('Sleeping for 1 second') time.sleep(1) print('Done sleeping...') if ___name___ == '__main__': p1 = multiprocessing.Process(target=do_something) p2 = multiprocessing.Process(target=do_something) p1.start() p2.start() p1.join() p2.join() finish = time.perf_counter() print(f'Finished in {round(finish-start,4)} seconds')
@ahmedkhalil3015
@ahmedkhalil3015 4 года назад
Add a shebang directive and you wouldn’t need one.
@awndlr
@awndlr 4 года назад
thx so much, already implemented that into my webscraping scripts and the speed gain is tremendous. you’re explaining this so well an clear. thx a lot.
@sellosejake
@sellosejake 4 года назад
Hear trying to achieve the same
@rembautimes8808
@rembautimes8808 4 года назад
I joined the C# start to finish course last year and am so glad that you did this video. Hope you are keeping well. Thanks for this well explained tutorial
@rzufvideo
@rzufvideo 3 года назад
incredible video! you've covered more information than all articles all together that i've read so far - thank you for that!
@rakeshkumarkuwar6053
@rakeshkumarkuwar6053 4 года назад
Thank you, Corey, for such a wonderful explanation. The best part of this video is you have used the real-time problem, which helps to understand the concept in more depth.
@kylebaker8502
@kylebaker8502 3 года назад
Hi Corey, great video covered a lot, one thing I was hoping to see here which I did not was using Pool from Multiprocessing to pass multiple parameters to a class or function. I ended up finding it after some digging using starmap with zip instead of map. To pass constants with this method import itertools and use the repeat inside the zip. Hope this helps anyone looking for this.
@drmezo7
@drmezo7 2 года назад
This is the best explanation of that topic I have watched so far! Many thanks, Corey!
@nikitapetrenko1236
@nikitapetrenko1236 2 года назад
Even these years, this video is really informative about that topic
@IrshadKhan-vk8rn
@IrshadKhan-vk8rn 4 года назад
Thnx sir I got placed in a MNC all bcz of you learned python a lot and still learning a lot whenever I have any doubts stack overflow and you r my mentor thnx again preparing for higher studies now..
@venkatkumar8672
@venkatkumar8672 4 года назад
i pursuing a college degree and I'm learning python in different resources and i learned the basics , will you please recommend me a good resource for mastering python language and to go to next level??
@nadik4531
@nadik4531 4 года назад
Brah i was stuck at work tryna do this. Searched your name and braaaah! Your videos are so helpful.
@woddenhorse
@woddenhorse 2 года назад
same
@nikhileshsaggere
@nikhileshsaggere Год назад
Excellent tutorials, he teaches practical concepts with zero assumptions on the background and makes sure he gives real life examples and practical code to get you through! Brilliant 🙏🏻
@8koi139
@8koi139 2 года назад
I was pretty scared of touching this topic, but you made it so easily to understand!
@SHA256HASH
@SHA256HASH 4 года назад
This is the second video I've watched from your channel and was blown away at the quality of the content and the clarity of the concepts explained. Once I get into a data analyst role, I will support you on Patreon. Until then, I will watch every other video! 😁Thanks a million!
@coreyms
@coreyms 4 года назад
Thanks! That helps more than you know. I really appreciate it!
@SebaSalinas91
@SebaSalinas91 4 года назад
Thank you so much for this quality of videos on Python!... Cant tell you how much i`ve learned with them. Keep it coming! :D PS. Any GUI tutorials coming? PS2. Greetings from Chile!
@Jonix-redhat
@Jonix-redhat 3 года назад
thx Corey for this great video! I'm a biomedicine master student from Sweden. This tutorial enabled me to use multiprocessing in a program I write for dna analysis during my master thesis!:)
@varunn104
@varunn104 2 года назад
i cannot thank you enough for all the videos you've put out
@attilagyen1446
@attilagyen1446 4 года назад
In newer versions of Python instead of using a list in args you should use tuple. So instead of writing this args=[1.5] write this args=(1.5,). It's okay to use the list version, but you will get a warning. Good tutorial btw.
@salfredogonzal_
@salfredogonzal_ 2 года назад
Not anymore ... self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute ...
@sebastianbarriap.6871
@sebastianbarriap.6871 4 года назад
Excelent tutorial!, just for completeness, i'd love to see some tutorial on python coroutines, thanks a lot.
@amitdwivedi9951
@amitdwivedi9951 4 года назад
What is that
@aparnamoorthi1117
@aparnamoorthi1117 4 года назад
Me too
@kenhaley4
@kenhaley4 4 года назад
Thanks, Corey. One of your best tutorials IMO.
@MrEliyahilel
@MrEliyahilel 3 года назад
My python classes knowledge came from you and now multiprocessing/threading. Thanks a lot for AMAZING content!
@MrRobbi373
@MrRobbi373 4 года назад
That's the module I used to parallelize and optimize Mosaik smart grid co-simulations in my bachelor's thesis, which I finished this week! :D
@edwardwong654
@edwardwong654 Год назад
Corey is great. When you help people, and I mean really HELP people, he is just a saint. I am on Python 3.9.12 and my code does not execute properly, even though I am copying verbatim and I a stuck at the 10:20 mark. After I put in the p1/p2.start() statements, the script runs without printing the "sleep" statements. The final print statement does execute. But after I put in the p1/p2.join() statements, I get a runtime error: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ...
@sayhellotoroy
@sayhellotoroy Год назад
same here
@emmanuellelopez1002
@emmanuellelopez1002 Год назад
another one with same error
@j.caldeira
@j.caldeira 3 года назад
I do have to thank you @Corey Schafer your work in these videos. I learn a lot from them and in the end I use them on my work to automate some tasks. Some love from Portugal
@mahathirmohammad8635
@mahathirmohammad8635 4 года назад
I watch both of threading and multiprocessing video together without skipping a moment. I enjoy your video more than movie.
@anuprshetty6391
@anuprshetty6391 2 года назад
Hi Sir, As you mentioned, I am interested to learn advanced threading concepts like race condition, mocks, etc. Please make videos on that too. I will be waiting... Between really nice video. Thanks for the effort.
@johntamplin
@johntamplin 3 года назад
When faced with a python challenge, I ask myself "What would Corey Schafer do?" I then, somewhat ironically, ask myself "Why am I wasting my time asking myself rhetorical questions, when I could be watching one of his outstanding videos."
@churchillagutu7287
@churchillagutu7287 Год назад
This is A GREAT VIDEO!!!Thanks for explaining each step of the code
@RomanKozin
@RomanKozin Год назад
Still. After 3 years - it's still helpful. Even counting that I need to test this out - nothing that much has changed in Python for MP. Thanks for such an excellent explanation!
@piiumlkj6497
@piiumlkj6497 4 года назад
Any computer vision tutorial coming ? Would be awesome !
@khmsalhuas1
@khmsalhuas1 4 года назад
Dude you are amazing! Excellent excellent stuff! Truly great educator and we own you a lot. I have a question. What if my function takes more than one argument. Will the ProcessPoolExecutor instance (executor) be able to map the function to however number of arguments for every process?
@drygordspellweaver8761
@drygordspellweaver8761 2 года назад
It’s the map method that would handle that. The function then would naturally need to accept multiple arguments
@osyl8
@osyl8 2 года назад
Excellent hands-on walkthrough of Python with lots of examples. Thank you, appreciate your video!
@edgarlip2
@edgarlip2 4 года назад
by far the best out there ! waiting for locks and raise conditions !!!
@hacker2ish
@hacker2ish 4 года назад
You could have put the image processing and the reading and writing from disk in separate functions you could have used threads for the IO and procesess for the processing
@vipuljoshi7367
@vipuljoshi7367 4 года назад
31 people didn't find a piece of code they could copy and paste for their work.
@BeattapeFactory
@BeattapeFactory 4 года назад
#indians sorry
@dylanbruner8348
@dylanbruner8348 4 года назад
your sopposed to follow along
@aeroscience9834
@aeroscience9834 3 года назад
NΛVΛZ don’t be jealous
@tarunreddy7
@tarunreddy7 3 года назад
57 now
@MattPutt
@MattPutt 5 месяцев назад
my new favorite python teacher! TY
@scholar_sukhiya
@scholar_sukhiya 4 года назад
thanks a lot Mr Schafer!! your videos helps me a lot, plus the tips u give to keep the code clean are making my projects look awesome
@9SMTM6
@9SMTM6 4 года назад
The reason it finished the 1s future in the Multiprocessing part after the 2 and (just after) the 3s future is that the PoolExecutor limited the threads to 4 (you said yourself that it chooses the number of concurrent threads based i.e. your hardware). And your list of times to wait is reversed in duration. So it only started the future for the 1s duration AFTER the first future finished, the one with a duration of 2s. So the 1s future finished after 2+1= 3s plus overhead, so slightly after the 3s future. Edit: Aaand you realized that just shortly after. Was wondering why noone else wrote this before.
@drygordspellweaver8761
@drygordspellweaver8761 2 года назад
No one mentioned it because it’s literally spelled out in the video 🤦‍♂️
@9SMTM6
@9SMTM6 2 года назад
@@drygordspellweaver8761 I did wait some time to write that comment. You meanwhile were too lazy to read my whole comment.
@drygordspellweaver8761
@drygordspellweaver8761 2 года назад
@@9SMTM6 Literally everyone who watched the video understood what was going on. Only you saw fit to go out of your way to write a comment to "explain" what was literally explained in the video like 5 seconds after it happened. You are insufferably arrogant and dense.
@gerhardboenisch5618
@gerhardboenisch5618 4 года назад
I had to add if __name__ == '__main__': for it to work in Python 3.7.6 I did not get the print output, but the images were processed
@XinhLe
@XinhLe 3 года назад
The same for me! def main(): if __name__ == "__main__": main() and it worked, anyone can help to explain why?
@user-he4ef9br7z
@user-he4ef9br7z 4 года назад
thanks man , you are a savior , after watching your videos on multiprocessing and threading i could finally make a project i wanted to make for a long time - python which counts the number of times i press each key and store them in an excel file for each day
@c.c.1297
@c.c.1297 3 года назад
Thank You Very Much!!! I now understood how this works. Your way of teaching and explaining are very understandable unlike the other videos that I have watched.
@vincenzo3574
@vincenzo3574 4 года назад
What's the difference between time.perf_counter() and time.time()? Why should I use one over the other?
@TheOiish
@TheOiish 4 года назад
The resolution of perf_counter() is higher than that of time().
@Mayur7Garg
@Mayur7Garg 4 года назад
But he is eventually rounding off😅
@guardrepresenter5099
@guardrepresenter5099 4 года назад
for i inrange(1,1000000): print ("Thank you")
@jimmy2643
@jimmy2643 4 года назад
print("Thank you "*1000000) # :)
@sebastianjost
@sebastianjost 4 года назад
@@jimmy2643 wow. I didn't know this was possible... thank you!
@macieks3604
@macieks3604 Год назад
Great Video!!! As always, the best, most logical explanation on RU-vid.
@javedalam7383
@javedalam7383 2 года назад
One of the best video of Multiprocessing I have ever seen. Thanks for all your effort and teaching us.
@yandere_lover-tl9zs
@yandere_lover-tl9zs 4 года назад
Bro, can you tell me why this error pops up?? I'm trying to print the return value of the function. concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
@concert_music
@concert_music 4 года назад
Very good video, though yes, this is really frustrating me too!
@TheEpicPineapple56
@TheEpicPineapple56 4 года назад
@@concert_music You have to encase your code in that > if ___name___ == "__main__": thing. This is done to avoid unwanted side effects, and multiprocessing and concurrent.futures libraries both force programmers (at least on Windows; Corey is on Mac) to do that. Hope that helps! :D
@lethalfang
@lethalfang 3 года назад
Took me a while to find the issue: stackoverflow.com/questions/61860800/running-a-processpoolexecutor-in-ipython Basically I added those lines on top when I'm using MacOS on jupyter-lab: import multiprocessing as mp mp.set_start_method('fork')
@hongbingqin6876
@hongbingqin6876 4 года назад
It's great video. However, I receive the following error when run your code to print results in my windows. But the code works when I run it in aws environment. BrokenProcessPool Traceback (most recent call last) in 3 results = executor.map(do_something, secs) 4 ----> 5 for result in results: 6 print(result) 7 ~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\process.py in _chain_from_iterable_of_lists(iterable) 474 careful not to keep references to yielded objects. 475 """ --> 476 for element in iterable: 477 element.reverse() 478 while element: ~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py in result_iterator() 584 # Careful not to keep a reference to the popped future 585 if timeout is None: --> 586 yield fs.pop().result() 587 else: 588 yield fs.pop().result(end_time - time.monotonic()) ~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py in result(self, timeout) 430 raise CancelledError() 431 elif self._state == FINISHED: --> 432 return self.__get_result() 433 else: 434 raise TimeoutError() ~\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py in __get_result(self) 382 def __get_result(self): 383 if self._exception: --> 384 raise self._exception 385 else: 386 return self._result BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
@surajitchakraborty1903
@surajitchakraborty1903 4 года назад
@Hongbing , even I am getting the same error in windows. Did you manage to get around it - if so then how ?
@darkfantasy437
@darkfantasy437 4 года назад
@@surajitchakraborty1903 I'm getting similar errors in pycharm - not exactly the same, but similar (also on windows)
@mikeronni1
@mikeronni1 3 года назад
Excellent as always, Corey! You are truly a national treasure.
@dinhluongnguyen9758
@dinhluongnguyen9758 Месяц назад
when getting to learn the idea of data parallel and distributed data parallel, they stuck me a lot, but after watching your 2 videos, they help me a lot. Thanks sir, you're a life saver !
@unperrier5998
@unperrier5998 Год назад
At 40:40 you're omitting one important thing: GaussianBlur() is an extension using a library written in C. So it can processing the image in a separately managed thread in C and relinquish execution to the python interpreter to process python bytecode on other python threads. So it's essentially a bad example, because at the low-level it behaves like multiprocessing in the sense that it can run things outside the python interpreter (multiprocessing does that in a separate process, PIL does that in a private thread, in C). So your assessment that the workload is I/O bound is incorrect. It is mostly CPU bound, but the processing you chose released the GIL, that's an unfortunate choice.
@HarisJabbar
@HarisJabbar 4 года назад
Very well explained! With right level of complexity for someone looking to explore simple multi processing in python
@biswajit-k
@biswajit-k Год назад
Thanks, Corey! Great Tutorial
@TonyStark-cp3tj
@TonyStark-cp3tj Год назад
That was Brilliant! You are the best!
@TalkingMisc
@TalkingMisc 2 года назад
OMG ….Clear instructions. Thanks a lot ❤️
@drforest
@drforest 3 месяца назад
That last piece showing how to switch from processes to threads for testing was super cool. Thanks for this great video.
@karanparekh2329
@karanparekh2329 4 года назад
Been waiting for this!
@prateeksarangi9187
@prateeksarangi9187 2 года назад
Thanks Corey !! Learned something useful
@mikenesbitt8052
@mikenesbitt8052 2 года назад
Wow!! I kind of understood multi processing, but this polished up everything I know into a very sparkly vision.... Thank you!!!
@AnnWong1
@AnnWong1 4 месяца назад
Thank you Corey, it is brilliant to start from the basics. Really helpful approach and advice !
@iamicde
@iamicde 3 года назад
very clear and impressive teaching - thanks alot
@jonatan01i
@jonatan01i 4 года назад
Based on your results, I am gonna go with multiprocessing. Thanks very much for the tutorial!
@cetilly
@cetilly 3 года назад
Wow! The use of map() is brilliant way to automatically batch the jobs for each process. Thanks for showing this!!
@romabilibov7612
@romabilibov7612 10 месяцев назад
Awesome tutorial! Thanks showing both methods of handling multiprocessing.
@Lux1431996
@Lux1431996 Год назад
Absolutely great as always! Thank you a lot again!
@martinfultot5756
@martinfultot5756 4 года назад
Excellent. Perfect tutorial! I'll use this with pybullet.
@haroldthibault9921
@haroldthibault9921 2 года назад
Thank you very much sir Corey Schafer for all this good quality content !
@phoenix6909
@phoenix6909 Год назад
Amazing master!!! just love the way you walk through !!!! thanks you serrrrr
@_dark_horse
@_dark_horse 2 года назад
Glad I found this channel
@klyntonh7168
@klyntonh7168 2 года назад
Absolutely phenomenal video!! Thank you so much for doing this.
@mdfaiz4583
@mdfaiz4583 2 года назад
your video saved me.. Thanks a lot Corey
@danielsiqueiradeoliveira3929
You are an amazing professor! Really easy to follow your speech. Thank you so much!
@deepgrewal9862
@deepgrewal9862 3 года назад
You are the best teacher I have ever had. I wish I could give this video a million likes.
@BhanudaySharma506
@BhanudaySharma506 3 года назад
Man... You have explained so good.... Fall in love with your teaching style.
@mdmusaddique_cse7458
@mdmusaddique_cse7458 4 месяца назад
Fantastic explanation!
Далее
Лайфхак с колой не рабочий
00:16
Просмотров 271 тыс.
Unlocking your CPU cores in Python (multiprocessing)
12:16
God-Tier Developer Roadmap
16:42
Просмотров 6 млн
5 Good Python Habits
17:35
Просмотров 406 тыс.
10 Nooby Mistakes Devs Often Make In Python
24:31
Просмотров 48 тыс.
Python Generators Explained
28:37
Просмотров 146 тыс.
Compiled Python is FAST
12:57
Просмотров 100 тыс.