Hi all! Please suggest how to implement the following scenario: Have a dag with sshoperator that executes shell script. Shell script returns different return codes. How to analyze which return code is received??
Nice video but is there a way to do it between 2 DockerOperator? And if not with the DockerOperator creating some kind of CustomDockerOperator to make it able to do it?
Thanks, Marc. Great info Can You Please Confirm This I have Two Tasks in DAG1 Task1 - Normal Function Which does some process Task2 - TriggerDagRunOperator (Trigger DAG2) Can I Use the return value/XCom of task1 in Task2 and use this value in DAG2? Thanks
Sir, I tried to do the needful to do good and nice Airflow in the XCom, but my hit chance was still saying 99% chance of to becoming the hit sir, but it still missed most of the time. Sir I do believe you should do the needful by improving the Airflow by using the Minecraft Mushroom Soup Windtunnel.
@@MarcLamberti It would be awesome if you can share some knowledge by taking this tutorial further by adding (if possible) a new service and assigning it a task via DAGs within the Airflow container. Just to give you some background on what I am currently trying to do --> Similar to your compose file, I have Postgres, Airflow Webserver and scheduler services defined in docker compose file and running perfectly. However, I have an extra service (say an Ubuntu server) defined in the same compose file which basically takes an .sh command and runs it. I am trying to send an sh command via DAG DockerOperator, to this 4th container. I took help from your video on DockerOperator as well :). If I understood correctly, the Airflow DAG in that video is running on the host instead from within a docker container. I searched for help on the internet, but haven't come across any. Do let me know if such use case is possible with airflow. Appreciate you help :)
@@MarcLamberti thank you for the info!!, you are right the execution dates (YYYY-mm-dd, HH:mm:ss) are different, I see it has to be executed at the same time to overwrite that XCOM.
Hi @@MarcLamberti I have the same issue. I do want to overwrite the value from the last run. I have a DAG that will run every 15 mins and it will push a key to XCOM. Since this key will be stored in the DB, the number of records will keep on growing. Hence, I want to overwrite the value of the key from the previous DAG run. please let me know if that is possible. Thanks!!
