Running Airflow 2.0 with Docker in 5 mins 

Wow, its amazing how far the Airflow team has come with this. Thanks Marc!

arrgh .. spent about 3h trying to figure this out, basically all the online-instructions missed one small bit or another ... with your instructions, le voila, it works straight up. Thanks a lot!

I finally got it up and running! Thank you, Marc!

Awesome! Thanks for the tutorial, Marc!

I struggled to get airflow running for a long time and this short video helped me SO MUCH, thank you!!

This instruction really helps me thank you so much !

Thank you so much, Marc. Great content! TIL Using YAML aliases and

Thank you VERY MUCH!! Marc. This video is very useful.

The only of many tutorial that actually helped! Thanks

Thank you! very efficiently and clearly explained !

Thank you Marc. I was in hurry to find out how to run airflow and kept failed somehow. However with your nice clear explanation, nothing is mysterious anymore~

Awesome Marc, thanks for sharing

Thanks marc to sort the installation part of airflow👏👏👏👏

Thanks Marc, It helped me a lot!

super amazing job thank u!

this is so much fun and informative, thanks.

I love this!!! thanks man!

Fabulous! Thanks Marc! I have installed Airflow2.0 successfully. The webserver was failed to start in my MAC. But after I have increased the memory to 4GB ...it works.

This is an awesome tutorial ! Thanks a lot~

excellent walkthrough,,, Thanks :)

I always try to find a docker image to perform experiments. Thanks for providing a reference that I can refer anytime in future.

Awesome man!

Thanks Marc ! awesome tutorial.

Great job boy. keep it up.❤

Thanks for the video!

Thanks Marc!!! Great Job!

thanks. you help me a lot.

Thanks, simple and clear ;)

Hi Marc, great video. Just wondering if you could show us how to install a triggerer into your airflow stack using docker compose? Thanks!

Amazing! Thank you very much!

Hi Marc, thanks a lot for this!! :)

Thank you so much!

VERY EASY TO UNDERSTAND

neat & clean! thanks!!

super video Marc je m'abonne !

Great video Marc. It's sort of crazy how easy that was (even on WSL2)... Thank you Some things I'm still considering afterwards: 1) Is this enough for a production deployment of Airflow if the database was decoupled from the rest of the container? If the container crashed for whatever reason all of the connections would be lost, so separating is a good idea. 3) For local testing/debugging of an instance I'm going to try and mount DAGS that exist in another project folder instead of the one that we created. 3b) For local testing I might also try and store connection details in environment variables in the .env file rather than relying on the persistence of the database.

Marc, Big fan of your content. Can you make a video for deploying Airflow 2.0 (with Celery executor) on Azure Containers?

Thaaaaanks, i have been having issues with running airflow and now it worked!!! Ill now be able to automate tasks and be lazier lol,

THANK YOU

This is super useful... Thank you. One question: Can I use it on an AWS instance. How should I configure the security group and firewalls.

Thanks Marc for this video. Question: Do I need to run everytime I spin the containers?

Hello Marc thanks for the videos it's great, I have a question for you how can we version the dag in production ?

Nice 😊

Great! but where can i locate the requirements.txt to add for example the apache-airflow-providers-snowflake?

Hello Marc, Your videos are always great and helpful and with your video I get Airflow running quite well. The only trouble is that I need to run java within docker and I have not found any good description of how to get this working. I am starting a shell script that starts a java runtime within the terminal. Could you give me some help on how to get this running? Thanks, Armin

For anyone coming here from the 2024's and beyond, in Linux, specifically Ubuntu, remember to use: `docker compose up init-airflow`

Thanks a lot Marc. Can you also please make a video for deploying Airflow 2 using helm chart? and go over the options on values.yaml file? Thanks in advance

Great tutorial Old but relevant. Thanks! Marc, I am using Visual Studio Code and everytime I want to save my dag file, I need click a button "Retry as Sudo". Can you tell me what to do here... it is quite annyoing! Regards!

Hi Marc, thanks for your sharing! I'd like to know how to install third-party modules. When I installed yfinance module, there was a dag import error : no module named yfinance.

Thanks for the wanderful tutorial. I understood that the DAG file I stored in DAG folder will be added to Airflow. But what happen if Airflow is running in remote docker that I only have web access? I can upload DAG from my local disk to remote? Or is there other way to do it?

Hi Marc, great tutorial. Airflow is running w/o Problems. I tried to use vs code with airflow and found your new video "Configure VS Code to Develop Airflow DAGs with Docker at ease!" However, I don't understand where the Dockerfile come into the picture. Can you please elaborate! ---> Reopen in Container looks totally different as in your video. Thanks

First of all, thank you so much for this *awesome* video. It is really helpful. I followed this tutorial and was able to access AirFlow seamlessly. But I want to have apache-airflow-providers operators. So, I tried giving them in _PIP_ADDITIONAL_REQUIREMENTS and also building using Dockerfile. But nothing worked and I still see "error: command 'gcc' failed with exit status 1". I changed airflow image to 2.1.2-python3.7 as slim versions don't include extra libraries. But no luck. Could you help me resolve this issue?

Kindly demonstrate on Teradata and keycloak containerisation

It works on Windows and Mac, I tried both and it works, thanks (on Windows with some tricks)

Thank you for the awesome tutorial. I do have one question though: how can I install python packages with docker-compose when creating the containers? for example I would like to install Pymongo.

Hello Marc, I have recently installed airflow 2 using docker compose file as suggested in this video. But, when I enhanced the dag with mutiple connections i.e., Gdrive->S3, S3->Snowflake,Snowflake->S3 operations using pyspark and sql scripts, the webserver keeps restarting and at times shows unhealthy. Can you please suggest or advice what could have gone wrong or should I consider increasing docker memory?

It was great video thanks. How would i push my custom airflow python file into docker container?

For those that have a Mac and install Docker Desktop, you will not need to install Compose separately. It comes with Docker Desktop

Hello sir, how can we launch every task of etl in a different container as we do via k8s pod operator to launch every task of dag in a different pod?

Hi Marc, I have used docker compose to install airflow. However, the sample dags seems not to work for me and I found no logs.

Hi mark. I bought your course but I got an error trying to run the bash operador that insert data into the user table. I have tried the comand alone in the console and it works but when I used inside my dag in my bash operator I got this error bash command failed. The comand returned a non-zero exit code. I have tried a lot but I still can't found a solution for this

Is this usable in production? Could you create a production setup?

I had to increase the amount of RAM available to Docker to 6GB for this to work on my Mac. Also had to enable permissions to the folder i worked in with CHMOD.

Great tutorial! Short and sweet. I followed the exact same steps and checked the containers status, redis and postgres were healthy but airflow-scheduler, flower, worker, webserver and triggerer were unhealthy then I deleted all the containers and repeated all the steps and now I'm getting error as "database "airflow" does not exist". Redis and postgres containers are running without any problem. I would appreciate if you can help me understand the error. Thanks.

Question: what's the recommended way to increase the number of celery workers using docker compose? Say from 2 workers to 10 workers? Copy&paste worker keys in docker compose yaml files?

For my Mac with M1 chip: I had to increase the amount of RAM available to Docker to 8GB, and swap to 2GB's.

Is there available docker-compose with mysql? Can u please share link if you have

how do you create a celery executor if you haven't specified a dedicated backend mysql or Postgres db in the yaml file?

Hi Marc I installed docker desktop at windows using Ubuntu wsl. I changed the dags directory path in .yaml file to my c:\ drive folder in windows. When I start web UI, it doesn't pick my dags.py file. what can be the issue.

While running Airflow 2 via Docker Compose(Just like the above video), I am unable to successfully execute DockerOperator tasks. Can you enlighten with a video reference or doc reference about how to properly configure Airflow Docker compose file or Docker Operator to run tasks

Hi Mark! could I perform these steps without problem on a raspberry pi?

Any tips for this issue: AirflowException('Celery command failed on host:

When i followed the steps and installing, I am getting the error " manifest file not found". I have seen this error is reported by others as well; I changed the image to 2.0.1 and it worked by later I get an error about old version used.

How do we install providers after installing airflow on docker

Localhost:8080 aint opening for me. How to check the logs for any issues?

How can I install required packages to docker or how can I mentioned required packages in .yaml file

When I got to the docker-compose up airflow-init, I get the following error: "Python-dotenv could not parse statement starting at line 1 Traceback (most recent call last): File "docker\api\client.py", line 214, in _retrieve_server_version File "docker\api\daemon.py", line 181, in version File "docker\utils\decorators.py", line 46, in inner " A few dozen more error lines afterwards, but I can't make it work so far

Can someone explain to me why we are running "docker-compose up airflow-init" and then "docker-compose up"?

How can we add and setup airflow.cfg file inside project folder?

Very cool! Unfortunately, I got an error saying 'port 5555 is already allocated..' but I am pretty sure there is nothing on there. So, not sure what's going on.

I have many problems using PythonVirtualenvOperator or ExternalPythonOperator inside docker because you must include system site packages as True (it creates conflicts between venv and base python libraries) or otherwise you will get "ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv"

I installed everything and could not open the localhost:8080. Tried many times and safari said that "safari cannot open the page. The server dropped the connection. This happens when the server is busy" why does that happen?

The container for the webserver seems to be restarting continuously every minute or so. Any idea why this may happen?

Great video with details! I am following your steps but I constantly get WARNING - Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'. when I did docker-compose up airflow-init.

I got this error "Error response from daemon: manifest for apache/airflow:2.6.0.dev0 not found: manifest unknown: manifest unknow" after I typied "docker-compose up airflow-init" Why?

where to place my custom Python file into the docker container? Thanks BTW for good video

I followed the steps mentioned here, but getting no response from gunicorn master within 120 seconds and the webserver keeps getting restarted. Can anyone help with any lead here please?

Anyone facing problem in getting the logs displayed in the UI? Clicking a task --> Log --> gives me a blank log frame. However, it allows me to download it to my machine.

I got all of this running, but once I add a new py in dags it doesnt show on the airflow interface. Anyone had the same issue?

Hello guys. Just a tip for everyone: Do not try to create the airflow folder outside your folder user... You will run into permissions problems (a tried to create a Airflow folder o /opt/, but i strong don't recommend that).

I don't know why but it is giving me some python error when i am executing docker compose up airflow-init. Any suggestion ?

Hi, I'm facing the issue PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/20201-03-16' when running "docker-compose up airflow-init" . Any idea? Thanks

where do I need to write the command at 2:07?

For me it does not work. Docker compose is creating path "./local" and I can not access it. Airflow can not read my DAGS. it is very frustrating. I have been installing airflow for 8th time and none of them worked...

The only issue is that I can't import anything to my dag from other folders (not dag folder). I don't know why but I get a Import Error

getting error : Import "airflow" could not be resolved while importing 'from airflow import DAG'

the only commands that worked for me was the mkdir one everything else gave me an error

I am importing by from airflow import DAG in a file inside my directory but vscode is unable to recognise airflow

anyone know why I can't access the installed airflow docker-compose in ec2 instance via browser? I have installed airflow using docker-compose in ec2 instance, all containers running, I have set inbound rules security group TCP 8080 port to be accessible. But when I open ec2dnsaddress:8080 on the browser, it shows This site can't be reached. I have check it also in docker-compose logs airflow-webserver, it doesn't capture access from the outside and it only logs healthcheck

When I try to run the official 2.0.1 docker-compose.yaml file at airflow.apache.org/docs/apache-airflow/2.0.1/docker-compose.yaml on my Ubuntu 18.04 LTS I get the following error: ERROR: The Compose file './docker-compose.yaml' is invalid because: Invalid top-level property "x-airflow-common". Valid top-level sections for this Compose file are: services, version, networks, volumes, and extensions starting with "x-". You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1. For more on the Compose file format versions, see docs.docker.com/compose/compose-file/ services.airflow-init.depends_on contains an invalid type, it should be an array services.airflow-scheduler.depends_on contains an invalid type, it should be an array services.airflow-webserver.depends_on contains an invalid type, it should be an array services.airflow-worker.depends_on contains an invalid type, it should be an array services.flower.depends_on contains an invalid type, it should be an array Changing the version to 3.4 removes the first error but I still get docker complaining about depends_on. How can I fix it? My docker-compose version is docker-compose version 1.17.1, build unknown docker-py version: 2.5.1 CPython version: 2.7.17 OpenSSL version: OpenSSL 1.1.1 11 Sep 2018 while docker version is Client: Version: 19.03.6 API version: 1.40 Go version: go1.12.17 Git commit: 369ce74a3c Built: Fri Dec 18 12:21:44 2020 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 19.03.6 API version: 1.40 (minimum version 1.12) Go version: go1.12.17 Git commit: 369ce74a3c Built: Thu Dec 10 13:23:49 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.3.3-0ubuntu1~18.04.2 GitCommit: runc: Version: spec: 1.0.1-dev GitCommit: docker-init: Version: 0.18.0 GitCommit: Thanks in advance, Flavio