Airflow Dynamic DAGs: The powerful way with Jinja and YAML

Подписаться 26 тыс.

Просмотров 15 тыс.

50% 1

Airflow Dynamic DAGs: The powerful way with Jinja and YAML
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🏆 BECOME A PRO: www.udemy.com/course/the-comp...
🚨 My Patreon: / marclamberti
Airflow dynamic DAGs can save you a ton of time. As you know, Apache Airflow is written in Python, and DAGs are created via Python scripts. That makes it very flexible and powerful (even complex sometimes). By leveraging Python, you can create DAGs dynamically based on variables, connections, a typical pattern, etc. This very nice way of generating DAGs comes at the price of higher complexity and subtle tricky things that you must know.
Ready?
Lets go!

Опубликовано:

8 мар 2022

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 26

@MarcLamberti Год назад

Hey folk 👋 Here is the updated video with the sound fixed: Dynamic DAGs in Apache Airflow for beginners ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-6eHuOd96unQ.html Enjoy ❤

@litan1106 2 года назад

thank you for the great tutorial again. I love the airflow series.

@ricardorodriguez4180 2 года назад

Awesome content, thank you.

@timerelaxingsound 2 года назад

awesome! that's what i was looking for)

@rajdeepsinghborana2409 2 года назад

Informative ❤️

@skviknesh 6 месяцев назад

Lovely!

@egehanyorulmaz4965 2 года назад

I believe, this method is much more reliable if you are going to generate multiple dags with different schedule intervals. I was using dynamic dag generation with globals()[dag_id]=dag, but if you are working with APIs or clients, scheduler & worker don't work as expected. Also in the documentation, isolated DAG files are suggested, therefore Airflow community must be using your methods as well. Generator.py can be scheduled to generate the files using cron job, and it will be good to go :) Thank you for the content you have shared and informed Airflow users. It was extremely helpful for me :)

@rajhindocha9728 Год назад

Hi Marc, thank you very much for sharing great information! one quick question though, this could have also done by using normal replace method as well where it will replace the values it receives from yaml in dag template and generate new dag for each yaml config files. Any specific advantage of using jinja2 templates instead?

@user-lw4io9cg4p 9 месяцев назад

This is informative but I think the piece that I always struggle with is whats the best way to keep the load on the scheduler down if I need to pull data to build the dag based on whats in a database.

@tehmoonrulz Год назад

Hi Marc, can you please elaborate on the risks of using config with a loop and globals? I would agree that having one file per DAG is more ideal but you also linearly increase your codebase for each config so that con needs to outweigh the risks/downsides of the loop/global approach

@shiroyashazoro 2 года назад

Hey. Can I access the airflow instance which is deployed on astronomer using kubernetes?? I mainly want meta data to access

@claudee8736 Год назад

Hi! Thanks for your video. Very helpful. Just a comment, you're recording your voice through just one channel, and music in the other, which makes it a bit quieter than normal.

@RajeshSamson 2 года назад

Do you have any git repo this example?

@Fizzility 2 года назад

Hi Mark, I am new to airflow and I was hoping you’d be able to help provide me guidance on this: I want to create a job that retrieves 1.api key based on params (ie params=api id) 2. make api call 3. load data However I need to do this for many different Apikey (ie do this for 100+ different api keys, but function is the same across all, only difference for each job is params of step1) how would you approach this? Would you dynamically create individual dags based on params for each api key? Would you have a parent dag to generate subdags for each apikey? Or any other ideas? Thanks a lot in advance!

@mukundreddy9374 Год назад

Hi did you find a way to do it?

@97Arshan 2 года назад

Hi Marc, first of all, thanks a lot for teaching us airflow! I also got your course on Udemy but I had a question, there you ask us to use virtual box but I didn't and I used your RU-vid video to setup airflow using docker in 5 minutes. That shouldn't be a problem for the rest of the course, right? Edit: fixed tracing to teaching

@MarcLamberti 2 года назад

Not at all and I'm actually thinking of removing this VM

@97Arshan 2 года назад

@@MarcLamberti that's great, thank you very much!