The Fastest Way to Loop in Python - An Unfortunate Truth 

Comparisons are expensive and iterating is cheap. That's true in assembly, C and Python.

WRONG -- he just showed that numpy works fine

@@firstname4337 Yes, because if you use numpy you use C. That is the whole point of numpy, to not use python for operations that are repeated millions of times. Alex' point stands, because in the numpy case we use Python for everything that is not time sensitive, but for the thing that is time sensitive we use a small C monster wrapped in a thin fluffy Python blanket.

@@HiltownJoe Have you heard of PyPy. I ran his source file in pypy3 and got less than a quarter of a second for all methods except sum generator and sum list comp (0.72 seconds for those)

Clearly not a protip. A professional understands the constraints of their environment and the project they're working on. The choice of programming language is not (always) up to the individual writing the application.

Talk about understanding the letters, but not the meaning.

To loop,or no to loop that is the question

@@samuelhulme8347 Hahaha DEEP

@@samuelhulme8347 ok, nice job, now im lost ..lool

"a strange language. the only winning way is not to use it. how about a nice game of chess?"

So true, and something many eager-optimizers need to hear! They will certainly learn from experience as they discover where they spend their time.

Hi Peter, well I've been a software developer delievering a (Basic) scriptable technical measurement data PC-application NI DIAdem, and we put most of our development effort into giving our users like you (engineers & physicists) optimized script commands, which behave similar to that of Python. So using already implemented loop command may give you better performance than own loops as described here.

that's the thing tho, they're not mutually exclusive, you can have a very easy to write and a decently performant language (look at julia) why python is slow is not bc it can't go faster or bc it's a sacrifice for expressiveness (as there are python implementations that are way way faster but that lack total compatibility with libs) The reason why python is slow is because it's poorly designed. It uses linked lists for lists instead of vectors (log of n vs log of 1), it compiles to an intermediate language but it leaves it at that, there's no JIT in Cpython, the global interpreter lock impedes people from writing super performant code, etc, etc Python is slow bc of the choices they've made over the years, they're consistently made choices that affect performance, they never took the time to think about it and see how they could implement abstractions that would not take a big toll on performance, they just added things on top as they saw fit. But that's not a big surprise once you see that python was a pet project of Guido and he just made it bc he could and people happened to pick it up and spread it (even tho it's performance and features were worse back then)

You're using Python for simpler scripts, like what it was made for. That's a valid use case. Making large systems in Python is just stupid for the simple reason that it will be expensive to run.

That's depends on task, but when we talking about science - it's 100% true.

I always wondered what's the point of this? Are series we have summation formulas for really coming up in a real software? As far as I can tell the only application is to show it in a presentation about optimizing compilers. And frankly, this is even a wrong showcase of optimization, cause the useful techniques are general and pliable enough for compiler to reason about. This is just a hardcoded nonsense.

@@Czeckie this exact problem isn't common. But good compilers can recognize loads of (small) patterns in a codebase that, when combined, can amount to a large speedup. Sometimes some optimized chunks can again be combined into a new, smaller and/or faster, chunk.

@@oj0024 Yes, if you write shitty enough code the compiler cannot fix it for you because it can’t make assumptions about what you’re trying to accomplish. This is not a revelation or even a shortcoming, its a feature.

​@Matt Murphy He is right that compilers can do it, but if we look at the two most modern compilers right now only one of them can detect it and only if you write the code in a non-obvious way. The second part showed that compilers don't do it with constants either, they just execute the code internally. I'm not saying that compilers aren't doing great optimizations, just that the claim that "A good C compiler would detect this common pattern" is wrong. Especially since such optimizations don't gain anything (I'm talking about the pattern transformation, not the constant evaluation). They can only be done in very few cases, and you usually assume that the programmers know what they are doing. (e.g. the only reason to implement the summation code would probably be for benchmarking purposes)

@@joestevenson5568 clang didn't detect the pattern when using a normal for loop, it only detected it if you write the non obvious and usually not used while(n--) loop.

or C++, either choice is good.

C++ should be the better option here, cause of the OOP support.

Or any compiled language, really.

Assembler the only real choice.

@@Jonas_Meyer You're not a real programmer until you can program in assembly

I add secret lessons to all my videos!

Added in Python 3.6.

Same! I didn't know this too!

Same

Same

Thanks so much for the kind words!

Excellent content and testing summaries !

Then again, he's the only one that would place "just know your result ahead of time lul" as a viable suggestion.

Yeah... that's the price you pay for flexibility. But there are ways to overcome this. They all require some kind of hinting that the structures you use contain only elements of the same type.

> In a dynamically typed language like Python, every time the interpreter encounters an expression like c = a + b, it will first have to check a's type (and find out it's an integer). Then it checks b's type (and finds out it's an integer too). Then it checks if there's any known operation to add (+) two integer types (of course there is). A dynamically typed language implementation doesn't *have* to do things this way, that's just the naïve approach (albeit the easiest approach out of the box for new implementations). A smarter specializing JIT can create precompiled versions of functions and expressions for commonly used types (e.g. integers), and can determine that they are invariant and avoid having to re-check the types each time you hit the expression. The default CPython implementation does things the way you describe, but alternate implementations like psyco and PyPy take the specializing approach in their JITs. Not surprisingly, PyPy executes these for and while loops on the order of 50-100 times faster than CPython does on my machine (and ~10 times faster than numpy). $ python3 benchmark.py while: 5.254076757992152 for: 2.9640753349813167 sum_range: 1.1016407490242273 sum_numpy: 0.6697274879843462 $ pypy3 benchmark.py while: 0.05818561598425731 for: 0.07444041498820297 sum_range: 0.07667819698690437

I hope this problem can be partly solved if people declare data types of nearly all the variables as they need to do in C before hand like I do most of the time

@@sumnerhayes3411 There are tons of other reasons why dynamic typing is pointless and bad. It saves you having to *gasp* tell the program the type of a variable one single time; and it costs you dearly. All python projects should enforce static typing, even though that doesn't gain performance. All languages should be statically typed, period, because it is infinitely more readable and provides no development benefits at all.

@@kylehart8829 I would have disagreed prior to learning Python, back when it was just dogma in my head. Interesting how that works. However, the learning process was made easier by removing a step. I think a better solution would be to make it extremely clear how to toggle Python into enforcing static typing, and why that's a huge benefit.

Indeed! Never underestimate math! The feeling of deleting 30 lines of scratch and replacing it with 3 lines of math is indescribable.

@@mCoding it's the moments when you're lying in bed and it kinda hits you out of nowhere that are the best Of course, it is kinda a let down when the divine revelations are actually wrong though

Doing it on paper with a pencil probably triggers a different part of your thinking process from all the years of doing math that way.

You can actually be even faster than with math, if you know how CPUs work. If you instead of calculating (n * (n -1)) // 2 just calculate (n * (n -1)) >> 1 it should (in theory) be a few clockcycles faster. While most math operations are pretty trivial for the CPU(+, -, *) division is a pretty annoying task, so you might spare some time with just bitshifting.

@@delqyrus2619 Bit shifting is math too. No different than multiplying and dividing by powers of ten being easy in decimal. A good compiler will optimize it though, so it's probably best not to bother.

I cannot stress this enough! Benchmarking is such a difficult topic even though it seems so simple.

One thing you can do is to disable CPU boost clocks during benchbarks. This way, you get stable clocks regardless of load created by the tested function. For example on Linux, you can just write 0 to /sys/devices/system/cpu/cpufreq/boost to disable the boost or write 1 to enable it again. This’ll get you much more stable results that are comparable even if ambient temperature changes.

@@luziferius3687 that's a good approach! Myself I usually try to do perf stat whenever possible, and count the actual CPU cycles it takes. Run each command say 3 or 5 times and take the median!

@@mCoding Obviously the best way is to time it yourself with a stopwatch

imo it would have been better to use a smaller n in the function and a higher number of runs

Or just ditch python and go for java or other jvm languages or go or rust etc

@@NJ-wb1cz Eh, rust is likely one of the hardest languages to learn (after "modern" C++ with template metaprogramming). Julia seems quite easy to learn and pretty fast (if strongly typed it's close to C, maybe twice or three times slower, but not two orders of magnitude like Python). On the other hand I'm not sure how to structure code which is too complex for a single file and too simple to put it into package. It was basically developed for data crunching. Also year or two ago Rust was lacking some libraries, especially for plotting graphs (C++ lacks them too), painting (Cairo) and even exporting data as images (no support for writting of 16bit PNG files)

​@@NJ-wb1cz right, who would do something as stupid as just adding another library as dependency when they could add an entirely different development stack just to optimise one function.

@@julians.2597 why is adding libraries stupid? That's a strange stance. Sure, moving to another language is an option, but you'll likely use libraries there as well.

He actually did say in the video that Numpy uses a lot more resources, and with a bigger number, you may not have enough memory. Your original thought was correct, that you do save resources by not using additional modules. You may save time, but not resources. Also, if you compile your code, those additional modules will need to be compiled in, which will make the executable larger, and require more resources as well. As we are quickly approaching the limits of how far we can go with hardware (CPU technology is pretty much capped out, unless someone figures out how to make atoms smaller), we need to be more mindful of the amount of resources we use. Programmers need to stop being lazy. They have to stop thinking that they can bloat their code with hundreds of modules because CPU's will be faster by the time their code is in use. Just because speed and bandwidth are available, doesn't mean they should be used. For example, I know of some websites that include 50-100MB of Javascript code for each page view. Sure, dial-up is a thing of the past in most parts of the world, and people have faster internet connections than ever before. The page loads pretty fast on a gigabit connection. What about the people who have much slower connections? Is it really necessary to design your website where every single page view uses an entire day's worth of bandwidth on a dial-up connection? If you have several pages of the same website open in different browser tabs, your memory utilization goes through the roof! I've seen a single browser tab on one of those websites using over 2.5GB of RAM! I haven't even checked how much unnecessary Javascript is downloaded when using something like Facebook, but I've seen a single tab with Facebook open using over _5GB RAM_!!! This kind of waste needs to stop. Even if it does save a second or two here and there, it could cost several more seconds (or longer) in other areas. I've always hated importing modules in any language I've ever programmed in. I would much rather write my own code to do what I need done, whenever possible. I realize that's not always possible, but it's always better than adding bloat to a program that's not needed. Look at what's known as the "demo scene". It's something that started LONG ago, where people try to make the most exotic program possible in assembly language that fits within a 64KB executable. People have fit 10 minute long full screen videos with music into 64KB! They have created complete first-person-shooter games in 64KB! These programs run plenty fast on hardware from over 30 years ago! If more of todays programmers took the time to write good code, we wouldn't even need computers as fast as we have. Sorry, that started off as a 1 line reply, and ended up going 100 directions. There's still soo much more I want to say about the topic, but who is going to read it anyway?

@@eric_d I will! I'm very new to programming and I really appreciated reading that. Thank you for the reminder :)

Thanks for the support!

Thanks for the support!

Awesome, thank you!

I guess I’ll be first then...

Your guess right

Its not incredebily slow, compared to others its slower yes but it can acomplish the tasks its meant for with no issues Machine learning, Automation, Data Science and Cyber Security

@@soupnoodles Yes but you can do all of that much faster with C, you're better off using C.

@@soupnoodles Also why did you like your own comment?

In my experience, I've never hit a RAM limit because of this, but in memory constrained systems this is definitely something to watch out for. I wish numpy has something like std::ranges::iota to use as an array-like.

​@@mCoding There is a library called dask, is used in distributed computing and has support for numpy arrays. working with blocks consume less memory but might be a bit slower.

i think numpy.arange(...) creates a potential memory issue (it allocates the a large array in this case), but the Python3 builtin range(...) doesn't allocate a large array - it is a lazy iterator that simply tracks the last value generated so it knows what to generate next.

People too often leave out the speed of development in their calculation of speed!

Python isn't special in this regard

People oftentimes forget lua.

A faster processor is cheaper than another developer to speed up development. That’s why I use python.

@@HydratedBeans unless you depend on oython-specific libraries I don't think the development speed is noticably greater compared to, say, kotlin. And a loosely typed language will always have more mistakes and harder to statically check

There is a language called Lython, so you don't try to loop in Python; instead you poop in Lython.

I appreciate that!

I appreciate that!

me too

Thanks, will do!

You should also check out Jax, Numba, and Cython for potentially enormous speedup for little effort if you are doing ML.

you're not supposed to do ML with for loops in theory you can and it's very undestandable if you do it that way but any solution using matrices/tensorflow will be faster. Like he says in the video numpy is fast

I appreciate that!

And he came up with this at such a young age too!

wait isn't it: (N+1)*N/2?

@@bFix He takes sum to N-1 in the example

@@laughtale1607 Yes, so it is n*(n-1)/2

@@Samsam-kl2lk yup

really?

Python doesn't have tail-call optimization, so you probably just overflow the stack.

I'm always open for suggestion! What do you want to see?

@@mCoding Nothing in particular just any fancy pythonic things which help improve performance and code readability. Also, one thing you can do is make a mess of a code and start refactoring it. That would be amazing to see.

You left out one important aspect; the teacher wanted to keep the kids busy for a while and Gauss came up with the solution within minutes.

Another important aspect, the formula is actually (n+1)*n/2.

@@thomas-sk3cr Exactly. I was looking for this correction.

@@thomas-sk3cr which actually illustrates a pitfall of using external math formulas instead of letting the computer do the work, that they are going to be less clear about what the code is supposed to do. getting the wrong answer really fast is worse than getting the right answer slowly

@@adrianmizen5070 not sure if I understand you correctly. I do think that it's definitely better to use a formula if there is one. But it should be properly documented of course and you have to make sure that it's the right one. Maybe we can agree on that? I don't think you should reimplement well-known formulas with loops just because it's more clear to the reader at first glance :)

Great suggestion!

yes, graph the size of n against the time it takes different strategies to compute the sum >:3 make the computer dance for my entertainment!

Thanks for the support!

@@mCoding Very interesting video indeed, something I never really thought about … but the math-guy in me needs to say it: You got the formula wrong… 😬

Flooring the division is very much recommended, because otherwise python will convert the result to a floating point number which may cause you to lose precision

@kurogami2771 it does in this case as we are adding up a series of numbers from 0 to n, which by definition excludes negative values. As for flooring, it is true that we should avoid using floats when not needed even for just one operation so we can either floor or int() the results.

@@aioia3885 Either n or n-1 is even so it's never a float. Using floor isn't required.

@@abebuckingham8198 edit: "either n or n-1 is even so it's never a float" is not correct because in python division with / will always result in float conversion (see type(1/1)). the rest of this reply is just trying to clarify what I now realize was not necessary to clarify in my comment I said "flooring the division is very much recommended" but what I really meant was not using the floor operator on the result but instead using the integer division operator. in python if you do a/b the result will always be a float bu if you do a//b then the result will be an integer if both a and b are as well. doing a//b is basically like doing int(a/b) except that there will be no conversion to a float and so no precision will be lost. for example, int((10**18+2)/2) evaluates to 5e17, because the division was a floating point one, but (10**18+2)//2 does result in the expected value of 5*10**17+1

Glad you enjoyed it!

Glad you liked it!

Other than that, the sum itself can be computed parallelly and fused together later, thus we are not forced to iterate serially element by element!

In compiled languages loops are often optimized on compilation stage, such as being unrolled where necessary, with particular emphasis on efficient cache/prefetch from memory utilization. One should look on original Fortran code of LAPACK routines where some loop optimizations were done 'by hand'

@@dmitripogosian5084 loop unrolling is never efficient when cache and prefetch is the topic unless you explicitly wrote "from 0 to 4" just to avoid copy pasting code 4 times by yourself, and even that may not be worth unrolling depending on body of the for loop which most compilers can decide by themselves if you do profile guided optimization pass. There's a reason why -O2 more performant for most software than -O3 and unrolling is one culprit of that.

Comparing the sum with the counter limit certainly won't give the correct result in s: We are calculating the sum of all integers and analyzing various ways to do that. I. e., if the limit where 10, with 1,2,3,4 we reach it, way before reaching the correct sum 55.

Most like myself use Python for the libraries, nothing else comes close except for maybe C++. It would take a lifetime to recreate all the libraries in Julia.

@@incremental_failure there has a lot of ongoing work to make native Julia libraries (mostly in the scientific and statistical communities), but Julia can also call C, C++, and Fortran code, and the PyCall package already allows you to embed Python in Julia, so you're never left in the lurch if you choose to use Julia. I've already seen Julia be hugely faster and more flexible when doing a lot of ODE modelling than when using R and calling deSolve which is written in Fortran.

Thanks!

Thanks for the tip! I have no idea what I'm doing video editing so tips like this can really help improve my video quality!

I have many video ideas in the works, this is definitely a contender.

Usually recursion is slower and takes up more memory. And many "big" problems cannot be coded recursively due to stack size limitations. But if compiler/interpreter can do tail recusion optimization, it's a very nice option. Unfortunately, not all problems are applicable to it.

The CPython interpreter doesn't do tail recursion so even if you write it out correctly it's still faster to loop, save the recursion for when you forget how to convert your code into a loop correctly

@Sec Coder Check out Lisp or Prolog. The recursion is conceptual in terms of the problem or model. Imagine a family tree. The definitions and relationships are recursive. Mom and Dad produce offspring each of which in turn can become Mom's or Dads who produce offspring. Comes into its own when these Relationships are highly complex ( Imagine every leaf on a real tree was in fact another tree and so on and so forth. The Mantlebrot set is recursive ) Essentially they become huge lists or recursive data structures. Those languages became prioritized and optimised around lists and recursive data structures.( With a few constructs and language primitives not found in the likes of C or Pascal such as "is a type of " "is connected to" "is related to", "is derived from" etc etc ) Pythons modelling capabilities Tuples Lists etc etc Means most of it can now at least be modelled easily before being translated to something else for speed. Ps for fun check out Occam . It's editor was indent sensitive just like Python It also let you mix sequential and parallel programming .!!!! Oh lordy 😁

@Sec Coder It lets you express common algorithms in a purely functional way, i.e. without needing to mutate variables. Doing so makes your code easier to analyze mathematically and makes it more composable.

Glad it was helpful! You are very welcome!

That isn't the point. The point is the efficiency of raw Python.

This is a good point that everyone should keep in mind!

Is slower than other languages but still enough because computers are very fast

On any CPU capable of many teraflops, 20 seconds for 100 million numbers means eons. It's like walking and taking a step only every 5 seconds, short distances It's not that bad, but will badly scale once you need to travel farthest.

You're very welcome!

Exactly!

You're right, I totally forgot about some "optimizations" one could try. Maybe a later video. Thanks for the support!

Unroll to 100,000,000 lines of additions, then reduce them to a single value and have the function just return that value. Boom, orders of magnitude faster!

@@mattmurphy7030 Unrolling is more about reducing dependencies than iteration overhead, the multiple accumulators are summed in the end. Branch predictor doesn't play much role here as this loop is fully predicable, only the last iteration flushes the pipeline after the loop gets up to speed. Data dependencies can potentially stall the ALU, but if the CPU has a really short pipeline it won't have any effect.. this is highly dependent on the microarchitecture so arguing one way or another is a bit pointless unless we specify the hardware..

You are welcome!

Thank you! I've seen many of Corey's videos, so I'm glad to hear that!

Awesome, thank you!

Absolutely, if you can use Numba to @jit your problem, them it will very likely speed up the solution! I'll probably cover this in a future video.

@@mCoding I did a small test. from numba import njit @njit def int_sum(n=1_000_000_000): s: int = 0 for i in range(n): s += i return s timeit.timeit(int_sum, number=1) 5.373000021791086e-06 timeit.timeit(int_sum_p, number=1) 44.33714248199999 int_sum_p is the same function, without decorator. The result is very close to C speed. But obviously, the best improvement comes from doing the maths properly.

Numba is amazing, if you can limit the code in the @jit’ed routine to simple loops and math and/or numpy operations. You can’t use things like lists or dicts

Should do all the examples with numba this is also interesting. Cython also. Masterclass in squeezing loop performance, could also show sum(x for x in range(n)). Ideally use matplotlib to show the data for different sum sizes and prove the obvious that it is linear

@@gregorymorse8423 When trying to make a comparison with compiled C++ code, I discovered that clang removes the loop altogether, applying the maths that the programmer should have done in the first place. And I suspect that numba applies the same optimisation, otherwise the total time does not make sense with the clockspeed I have.

Awesome, thank you!

A C compiler would probably compute the answer at compile time so there is nothing to compare! A Python to C transition would be a huge undertaking even for a small project.

Here are the results on my pc: python with for loop: 8.74s c with gcc and no compiler optimizations: 0.235734s c with gcc and full compiler optimizations: 0.048770s (gcc doesn't detect the pattern and actually executes the entire code) c with clang and full compiler optimizations: 0.000001s (clang detects the pattern and optimizes it using the Gaussian formula)

@@oj0024 wow, there is absolutely nothing to compare. I knew python was slow and C was fast, but I didn't know that the difference is to this extent

@@mCoding Nuitka will take Python code and transpile to C (which uses the Python standard Library to do the object iterations) - on some applications it is 2x faster than normal Python (and it is a work in progress).

I personally like to start with python. Its vast libraries and simple structure is easy for me to test programs. Later when I decide on a specific implementation, i'll code that in C++.

@@akshaybodla163 Yeah, I can see that, it's nice, but how do you transfer a python library to C++?

@@andreiiaz2097 A lot of C/C++ libraries have Python bindings so you can just use the same ones a lot of the time.

In reality, python is the first thing most people will use because it is easy to use. Python is only usefull because if works as a glue for other languages. Easy access to libraries written in more difficult languages.

Such a fact. I do statistics research using Stata. My rule is Stata where I can, Python where I must.

Numba would make it closer to the unoptimized C/C++ code. I used it to render fractals and I got it to run 50x faster than unoptimized Python, and almost the same speed as the C code. The thing is ... that you can make the loop a lot faster in C too because you can use SIMD intrinsics, so you could do 4 or 8 or even more operations in a single instruction.

Just tested.. The JIT execution time is similar to the math version. To all Numba / LLVM developers: Great job!

Watch out for those null pointers though, wouldn't want your missile program to CRASH!

Usually Ada or SPARK are used in high reliability systems like aircraft instead of c since they are much safer

I had heard about this before, thanks for sharing, it makes a lot of sense in safety critical applications. I know NASA still uses C for (as least some?) space shuttles!

@@bamberghh1691 I knew ADA is used for missiles... But is there a specific reason why.

@@mwanikimwaniki6801 Ada has been specifically designed to catch as many mistakes at compile time. It’s static analysis tools are extremely thorough, and in general, it’s designed to prevent undefined behavior, from its type system, to its syntax. It’s an interesting language.

btw it's doing 100M iterations so it won't affect too much in small iterations

Thanks!

It serves the purpose to explain the problem of loops and also the process of finding different solutions to a problem It’s a nice exemple

Yeap, I was thinking the same thing. These tests really don't tell you anything meaningful.

Such an honor! Thank you!

@@mCoding THANK YOU for creating such quality content.

Glad you like it!

Good point! Maybe some day I'll do a comparison of which C library for Python is the fastest... so many choices though.

@@mCoding Would be interesting to watch. Good video btw. I did not expect the "i = i+1" to have such a huge impact when iterating.

Thanks!

Glad it was helpful!

Welcome aboard!