Great job building this out and showing Python developers how easy it is to work with streaming data. There are so many different ways to build anomaly detection pipelines and this time it's great to see Databento, Redpanda Data, Streamlit and Elastic with Quix in the mix 🍹
Appreciate the effort. would definitely try this project personally. But can you come up with full data engg architecture that companies actually use. For example companies like uber,lyft,doordash etc. I would like to see how they design their data architecture to process huge volume of data 24*7. right from types of data collected to visualisation. Also pls try to include industry standard tools like airflow even though there are many simpler alternatives to airflow like mage ai. Cause many of the interviewees seek for these regardless of any better tools u know. Hope u found this review useful. Big fan ❤
There are a lot of videos on this channel, not all are best suited for beginners. You might need to have some background at DE or programming to be able to follow along
will i be able to make your projects on windows too? i dont have a mac and ive noticed that the CMD and terminal commands in mac are quite different from windows and i think i might not be able to find that specific cmd command for windows terminal what should i do?
In industry, anomaly detection is extremely common. In addition to finance, it's also deployed on a large scale in factories and companies that use sensors (agriculture, robotics, manufacturing, energy, etc). Sensors are everywhere 😎
Quix is absolutely suitable for production applications. In fact, it’s already being used by Formula 1 racing teams and major energy companies, where real-time insights from high-volume telemetry data are critical to their operations.
@@yash-ri2lgWhile they share some similarities, they’re fundamentally different. Quix isn’t necessarily a direct drop-in replacement for Spark Structured Streaming, but it can definitely serve as a strong alternative depending on your use case
@@yash-ri2lg Fundamentally, Spark applications need to be compiled, packaged and submitted to run in a cluster so there is complex infrastructure involved. Quix Streams applications take the microservices approach and run anywhere Python is installed, with no need for server/cluster software. Quix is true real-time stream processing, whereas Spark Structured Streaming is actually micro-batching so isn't true streaming. That said, if you're already using Spark for batch processing, as a first step I always recommend trying Structured Streaming out first. It's more important to identify and solve real-time use cases to see if there's value before selecting a new technology
