Understanding Python: Multiprocessing 

You hit on a really interesting (and contentious) topic with the GIL! So there is a lot of history with the GIL and its implementation. It largely came from a time where most people didn't have multiple CPU cores but still wanted to do parallel processing. It actually does a good job stopping most people from tripping themselves up, but imposes some limitations, namely stopping true threaded concurrency. Now, there are a few reasons why I haven't gone in depth with the GIL (it could use its own video), and one of the biggest of those is that it may be going away in future versions of python. We're already seeing some of that coming with Python 3.12 and the per-interpreter GIL implementation. But to answer the biggest of your questions, the GIL is effectively just a lock; pretty similar to what we used in the Threading video. Python is designed so certain objects or actions need to acquire and release that lock. Since Python runs in a single process by default, that act of waiting for the lock makes it impossible for true parallelism. This is because while one python thread is doing something, it has acquired the lock and it isn't until it releases the lock that other threads are able to acquire it. This is a giant subject, but if you think a standalone video on the GIL would be useful, then I'll add it to my list. I hope this helps and keep up the great questions!

@@JakeCallahan Interesting! Would definitely love a video about that sometime. Even the removal of GIL should have some sort of repercussion in the language design? The solution to solving this problem could definitely use its own video... Thanks for the clarification!

I'll add a GIL deep dive to my list, maybe even dig into the CPython source for examples.

That's my favorite part of programming, developing and testing possible solutions to interesting problems. Glad you enjoyed it!

Thank you for the kind words. I do have a video on threading that covers lock race conditions. There's also many more planned, including most of what you mentioned. Stay tuned

Interesting and tricky question! This is likely something you'll need external tooling for. The basic idea would be to snapshot your system's python processes before and after splitting off to multiprocessing. This is one way but, depending on what type of overhead you're wanting to measure, there would be other ways to measure.