This is really a great tutorial. Not just jumping into setting up airflow using "docker pull apache/airflow" but building the image from scratch. I'm also trying to replicate the image but hitting a wall. Do you mind sharing the "dockerairflow" folder so I can use it as a reference? The timeline. is 19:06 Thank you
just a heads up to anyone on windows- airflow is absolute trash on windows machines. literally rather spend 2k on a mac than work with airflow in windows
Love docker / helm / k8 but have not gotten into it with a data pipeline. Would be cool to expand on this video and see high level of docker/k8/py or c# / azure / snowflake
Do you know how to switch http to https for the web-server? I’ve already configured docker compose with SSL configuration like environment and volume. Can you please make a video
Thank you for the quick tutorial. I am a junior data analyst trying to manage better my data pipelines. Do you think the way you did the configuration there has some security issue? Because I'm going to replicate part of these video at my work and I am always afraid to mess up
I'd recommend using a managed service like MWAA or Cloud Composer if you're planning to manage this for a company. I am unsure about the scale of your project but Airflow in the wild takes a lot of effort to manage. If you get to a certain scale, it's just too much for one person to handle and it should be managed by the DevOPs or TechOps team. Here are a few articles that discuss the challenges and how-tos of managing airflow at scale shopify.engineering/lessons-learned-apache-airflow-scale tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html
Hey Ben, thanks for this video, it's been helpful! I was trying to run exactly same code but ran into an issue with "airflow-webserver / command". I guess it's because I can't see complete line. Would you mind sharing with us the code? Appreciate it! :)
