Hey @ArjanCodes, can you create video series in python instrumentation for observability i.e. metrics, log and trace at (application level, container &pod level and inter microservice ) I love watching your video
Spills to disk very well when you have bigger than memory data, not a strength of Polars. You can use all sorts of different languages with it, not just Python. Lots of people know SQL. It is a db with db features like constraints and indexes.
Influx DB looks VERY INTERESTING! We use RRD for this function and it has the most awful, clunky API you can possibly imagine. I think learning Flux Query Language would be easy-peasy-lemon-squeezy compared to navigating the tortuous documentation of RRD. :)
Thanks for this Video. I always like content that makes you reflect about architecture decisions. Another Database that seems interesting to me is ArangoDB
I would love to see a video about non-typical SQLite use cases. It's so flexible and lightweight and I feel like people are sleeping on it just because it's not for a client/server role. I started using it as a local K:V store because I didn't wanna bother with something like redis, and I'm quite impressed.
I don't like the implicit nature of duckDB. Constantly grabs objects that exist in a local scope. Polars on the other hand is much more stable because it is very explicit. I have had to fix data scientist's code many times because they didnt realise secondary effects of many duck db operations. Also duck db absolutely messes up the linter and static type checking tools.
These days, Postgres is very very good. You need a good reason not to use it. It is free, mature, scales, has good IDE support, good python support, extensions for everything, and great Docker packages. And if you want third-party support, it is easy to find at every level.
@ArjanCodes, man thank you so much for these contents you upload for us, very helpful, well described, and when you explain things, you make them look very easy, please keep up the amazing work
@ArjanCodes - Would you mind exploring Mojo more, for those who are looking to harness the power and speed it can provide for the standard Python user? There seems to be many topics related to ownership, life cycles, traits, and pointers which are foreign concepts and not a part of the standard Python paradigm.
I have never used guess it time to give it a try, can we get your views on using typesense in python projects using fastapi or postgres full text search.
I have a project that coukd benefit from duckdb i think, data isnt important enough for long term storage, but good to see at a glance as a technician or team of technicians. Perfect
RE: Rediculous DBs - did you know Python has a built-in DB? No, not SQLite! It's called dbm. It's not even relational - it can just store dicts for you! 😂
Nice video Arjan. I think session management with openAI is already implemented through the newish OpenAI Assistants API. Just use the same assistant with the same thread ID, and enjoy your key value store!
I don't understand why people say duckdb is cool ... feels just like sqlite but with the flexibility to work directly over dataframes or files ... but why would i use that instead of just loading the files with some specialized dataframe package like pandas, polars or vaex? It would be cool to see a video on it!
Can be quicker to than Polars and definitely is quicker than pandas. It is really useful when you work with team that are sql heavy/mixed and where there is a lot of legacy sql code to integrate. It's also lighter to setup (I sometime just use the cli or the exe). You can also take creative approach to your pipeline and apply the transformation that are clearer in sql using DuckDB and then continue using your dataframe package. I'm not saying it's a good idea but I did it for a few transformation and it worked really well. I feel like for some bigger than ram dataset it can be better than Polars and also is more mature for the moment if that makes sense. I also find that the "ergonomics" of DuckDB is really where it shine:DuckDB is the easiest way to use sql from python IMO not saying that other tools are difficult but DuckDB is dead simple.
Spills to disk very well when you have bigger than memory data, not a strength of Polars. You can use all sorts of different languages with it, not just Python. Lots of people know SQL. It is a db with db features like constraints and indexes rather than another dataframe lib.
Was that an official endorsement of hitting interns with mechanical keyboards??! Watch out, you'll get cancelled with talk like that! All joking apart, this was very timely and useful information for me. Thanks!
