Airflow DockerOperator: The Basics (and more 🤫)

Подписаться 26 тыс.

Просмотров 16 тыс.

50% 1

Airflow DockerOperator: The Basics (and more 🤫)
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🏆 BECOME A PRO: www.udemy.com/course/the-comp...
🚨 My Patreon: / marclamberti
The Airflow DockerOperator is a very powerful operator.
It executes your task within a docker container. There are multiple advantages of using the DockerOperator such as:
- Easier way to test your task
- Control over the resources needed by your task
- Avoid dependencies conflicts
and more.
Even if you shouldn't use ONLY the DockerOperator, knowing how it works and what you can with it will truly help to make more reliable data pipelines.
Ready?
Lets go!

Опубликовано:

12 окт 2021

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 40

@dr_flunks 2 года назад

it's actually super helpful that you display the folder/file structure of everything you're using as you go along. very well thought out!

@saritkumarsi4166 2 года назад

Thanks Marc for the video on one of the operators I use extensively :)

@anthonyloganhall 2 года назад

This is how we have our environment setup and it works very well.

@joshuabodyfelt1239 Год назад

Wonderful job Marc! If I could amend this - would be awesome to have a followup video discussing the variety of different Docker registries, and how to connect to them with the Docker Connection.

@aerobot6571 Год назад

Merci Marc, premiere video de toi que je regarde : c'est clair, utile et complet. Ben je vais voir les autres ;D

@MarcLamberti Год назад

Ben merci :)

@abhishekacharya5069 2 года назад

Hi Marc, Thanks for the video and it really helped me in understanding airflow. Actually I'm trying to pull the docker image by using dockeroperator on Apache Airflow. But wheneever I trigger the dag again and again, it pulls the image from docker hub. Can I save the docker image somewhere , so that whenever the dag is triggered it should not pull from docker hub. It will be very helpful, if you help me with this. Thanks

@lokeshkumar1365 2 года назад

Could you make video on best practices for kubernetesexecuter on k8s deployment and different tasks can run parallel?

@user-ep8sj9te3m 11 месяцев назад

Docker Operator doesn't seem to work when I run airflow in docker containers (using a docker-compose) how can I fix this ? edit: the only solution i found was adding to volumes: - /var/run/docker.sock:/var/run/docker.sock and setting user: root instead of user: "${AIRFLOW_UID:-50000}:0" but apparently this isn't the safest way??? Anyone got a cleaner way to fix this ?

@jagadishlucky1793 2 года назад

Hi Marc, thanks for the videos it really helped me to understand airflow effectively. Actually, Iam trying to generate dynamicness in tasks creation. Based on the config parameter from UI the dag has to run the tasks. For Ex: If conf parameter has t1,t2 as true, there should be two tasks running. And if I increase the tasks in config params(t1,t2,t3,t4) it should run that many tasks. I tried multiple approached using operators,its not happening. Can you please suggest an approach? ---> Thank you

@Ayush_1908 Год назад

Hi Marc, is it possible to use dockeroperator for running java code on airflow? Or any better option?

@kimted3272 2 года назад

hello Marc, thinking of listening yout lectures on udemy. are there any lectures that covers kuberenetesPodOperator? think operartor reference is the closest, but asking just in case if u already have a video. thanks :)

@yuricastro522 Год назад

If I'm using an airflow container to call another container, how can I mount volumes generated inside the airflow container to the other ? I'm getting errors trying this with this source parameter

@data-freelancer 3 месяца назад

Hi sir, can this work on production like cloud composer?

@user-ep8sj9te3m 11 месяцев назад

when running a DockerOperator on Airflow running in a Docker Container, the mounts have to be between the DockerOperator and the actual Host machine.. is there any way to avoid this? Can we create mounts betweeen the Airflow Container and the DockerOperator Container ?

@mbkhan1000 Год назад

I understand we can use the templating to pass variables/xcoms/connections to env variables in the docket container, but is there anyway to push values to xcomms from within a docket container? I understand that the process running in the container is isolated from airflow (unless it connects through the rest api?)

@gregh6586 Год назад

Why can't you use `retrieve_output_path`? What exactly are you trying to do?

@mbkhan1000 Год назад

@@gregh6586 trying to know if we can push to xcoms within a Docker operator task

@736939 Год назад

How to send data via XCom from DockerOperator? It there any better way than just print the values? how to run PythonOperator like script from DockerOperator let's say I want to run not the whole file but the function inside the file, how to do it via DockerOperator?

@emanuelgiannattasio3366 Год назад

Marc, in your opinion, in which cases would it be convenient to use DockerOperator over PythonVirtualenvOperator?

@MarcLamberti Год назад

IMHO the dockeroperator is great as your run a docker image so you can encapsulate your task in it. That can help for testing and versioning. Otherwise, go with the python virtual env operator

@vaib5917 Год назад

Hi, I really need to know if we put the python script into a container and run it using DockerOperator, how can we pass the values of Variables from AirFlow Admin UI t the container ?? Please help.

@mbkhan1000 Год назад

Templating into docker container environment variables

@user-ow4dv3cp8p 9 месяцев назад

I am trying to use DockerOperator but ger en error: PermissionError(13, 'Permission denied') I shoul set chmod 666 /var/run/docker.sock to avoid it. Can i use DockerOperator with chmod 660 /var/run/docker.sock?

@user-ot8bh3xm9j 3 месяца назад

Please tell me, have you solved this problem? I have the same problem

@ReenanOFC 2 года назад

Is it possible to set different schedules based on tasks?

@MarcLamberti 2 года назад

Nop

@danielpapukchiev3754 2 года назад

split in multiple DAGs

@trench6118 2 года назад

With a branchpythonoperator you can - for example, I have some tasks which have changed from daily to hourly within a client DAG (all sources for that client are in the same DAG). What I did was add a function to check the execution_date.hour and if it was a certain time, I would return all extract task IDs. Otherwise, I would return only the hourly task IDs. The result is that my daily tasks are skipped each hour unless it is say 13:00 UTC, and my hourly tasks run each hour. It makes the DAG a bit messy though because of so many skipped tasks

@eduardocarrerah3704 2 года назад

is this a replacement for k8soperator?

@MarcLamberti 2 года назад

Nop

@user-pr2kr1ts9i Год назад

getting this error requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

@PrakashReddyK 2 года назад

Hi 👋

@MarcLamberti 2 года назад

it's been a while

@vladdank9158 Год назад

Anyone figure out how to get this to work with Airflow itself running on Docker? Kind of lost. It's mentioned in the video around 8:03. I'm on Windows so it's kind of horrible LOL

@user-ep8sj9te3m 11 месяцев назад

the only solution i found was adding to volumes: - /var/run/docker.sock:/var/run/docker.sock and setting user: root instead of user: "${AIRFLOW_UID:-50000}:0" but apparently this isn't the safest way??? Anyone got a cleaner way to fix this ?

@slimebwoy 2 года назад

I know this is 6+ months after this has been posted but I think your teach would be a bit more effective if you took the time with your hand writing. If I saw the diagram at 3:25 after listening to you talk about it, I'd have a hard time understanding what it means. Just wanted to provide a helpful tip!

@MarcLamberti 2 года назад

thank you for the helpful feedback :)