Nice presentation. Discovered polars only recently and already like it. Feels way more lightweight than spark and I don't often need a whole cluster to compute stuff.
Nice seing "the industry" apply vertical scaling stuff that has been done in games (for world data) for at least the last 20-30 years (since cache fetch fail is order of magnitudes more expensive than a cache normal read and SIMD/MMX intructions are available on x86. I thought that compilers took care of these processor "bowels" stuff since thewlast two decades, and probably they did until now. It seems today the data sets are so huge that we must micromanage memory access and vertical paralelism explicitelly again. For a litle while at least. So it in hindsight, explicit optimisation of code data access patterns was outrunned by processor might, and then was further rendered obsolete by multi-core CPUs: Iin theory, of course, because good usage of parallelism is still in its infancy. But now it seems data set size growth has outrunned CPU growth 🤣 Low level 3D engine game programmers, you have a whole new market opening!