Тёмный

5 MUST KNOW Airflow debug tips and tricks | Airflow Tutorial Tips 1 

coder2j
Подписаться 12 тыс.
Просмотров 14 тыс.
50% 1

Опубликовано:

 

1 окт 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 27   
@nicpolyakoff
@nicpolyakoff 2 месяца назад
Thank you for the video! Exactly what I was looking for
@konradjablonski285
@konradjablonski285 Год назад
For Scheduler in docker alias start command. You can start/stop it when needed and instant refresh.
@coder2j
@coder2j Год назад
Valid point. 👍
@abdiyerzhan
@abdiyerzhan 4 месяца назад
can you explain bout trigger?
@machinimaaquinix3178
@machinimaaquinix3178 11 месяцев назад
DAG Shows up to slow section is for a local airflow setup only. For an airflow docker setup, use the following command in the airflow project directory wherever you've created it: export AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL=60
@coder2j
@coder2j 11 месяцев назад
In airflow docker, you can set the corresponding environment variable in the yaml file.
@kobrowsky
@kobrowsky 9 месяцев назад
- Hello, everybody! - Hello, Cleveland Jr.!
@shri2398
@shri2398 Год назад
Could you please tell me what this parent dag and child dag has
@amitaniket1502
@amitaniket1502 Год назад
I have started docker container. Now when I create dag files in my folder- airflow_docker, import airflow is not recognised. What to do?
@coder2j
@coder2j Год назад
In you Dev tool, select the python env in your docker container as the interpreter.
@Suresh-pk3sx
@Suresh-pk3sx Год назад
No task found how to debug this
@airflow578
@airflow578 Год назад
Thank you so much for the excellent content! In the airflow UI (docker), I could not see log history and got the following messages: *** Log file does not exist: /opt/airflow/logs/dag_id=........ *** Fetching from: :8793/log/dag_id=..../run_id=manual__2022-12-17T18:49:13.457497+00:00/task_id=data_crawl/attempt=1.log *** Failed to fetch log file from worker. Request URL is missing an '' or '' protocol. Do you know if I am missing anything in setup? Thanks a lot in advance!
@coder2j
@coder2j Год назад
Can you see your log in the logs folder?
@airflow578
@airflow578 Год назад
@@coder2j could not find the relevant log about the run in the logs folder, no matter the run was successful or failed😇
@ahmedshalaby9343
@ahmedshalaby9343 Год назад
greetings from Egypt you are awesome
@santoshkdwh
@santoshkdwh Год назад
Airflow scheduler is getting down due to deadlock error. what need to do to fix ?
@coder2j
@coder2j Год назад
Did you try restarting the airflow scheduler?
@xmagcx1
@xmagcx1 Год назад
regards
@hectorherreraespinola6399
@hectorherreraespinola6399 Год назад
Hi, this is a great video. I tried all day long today to fix a error. My airflow is running, muy dag is in place but when I push refresh, nothing happen. I did manually but is like the task is not running. The date I wrote in the code is tomorrow. Any advice? Many thanks
@coder2j
@coder2j Год назад
What are the start date and schedule interval?
@hectorherreraespinola6399
@hectorherreraespinola6399 Год назад
@@coder2j Start_date =( 2023,3,4) schedule_interval="@daily" I don't see an error but it's like the taks is not running. I am not able to see the dails of the taks. I when I go to graph I cannot click on the taks neither.
@coder2j
@coder2j Год назад
It will only run when your local time past 2023-03-04 00:00. Try to change start date to 2023-03-03.
@hectorherreraespinola6399
@hectorherreraespinola6399 Год назад
@@coder2j Thanks for your time. I changed the date and know I can access to the Log but it shows me the following error: *** Log file does not exist: /opt/airflow/logs/dag_id=our_first_dag/run_id=manual__2023-03-03T23:25:26.672754+00:00/task_id=first_task/attempt=1.log *** Fetching from: :8793/log/dag_id=our_first_dag/run_id=manual__2023-03-03T23:25:26.672754+00:00/task_id=first_task/attempt=1.log *** Failed to fetch log file from worker. Request URL is missing an '' or '' protocol. By the way, is the example of your airflow tutorial
@hectorherreraespinola6399
@hectorherreraespinola6399 Год назад
I did it! it was a issue with the version of python. I was using python 3.10 and I changed to 3.8 and now is working. Thanks!
@angadjitsingh6144
@angadjitsingh6144 Год назад
Thanks for this great content 🙏
@coder2j
@coder2j Год назад
You are welcome!
@moyeenshaikh4378
@moyeenshaikh4378 Год назад
is subdags covered in this tutorial?
Далее
▼ КАПИТАН НАШЁЛ НЕФТЬ В 🍑
33:40
Просмотров 473 тыс.
КВН 2024 Встреча выпускников
2:00:41
To Debug a DAG: The Airflow local dev story
20:03
Просмотров 1,4 тыс.
How to Debug Your Airflow DAG's!
10:19
Просмотров 2 тыс.
Airflow DAG: create dag with bash operator!
12:50
Просмотров 29 тыс.
Implementing Event Based DAGs with Airflow
20:59
Просмотров 11 тыс.
Airflow AWS S3 Sensor Operator: Airflow Tutorial P12
13:52
Dynamically Generating DAGs in Airflow
55:36
Просмотров 13 тыс.
▼ КАПИТАН НАШЁЛ НЕФТЬ В 🍑
33:40
Просмотров 473 тыс.