Please, do some "deep dive" tutorial on managing multiple airflow instances in a cluster like dev/prod. Never used git branching with airflow btw. What are best practices?
I'd like to see more about setup and teardown tasks. I'm using airflow intensively but we create our pipelines based on a Yaml file. (custom framework) so our dependecies are taken from the structure of the yaml
Great question. I think the best answer is to look at the following tutorial: docs.astronomer.io/learn/airflow-setup-teardown setup and teardown tasks have specificities that make them much more convenient to manage resources than a task with all_done. I will make a video about it very soon.
i am currently have an issue with subtasks in apache airflow. EXAMPLE: let say there are 3 subtasks (in parallerl) which are already completed and now two new tasks are added.. HOw do i make sure that only these two new subtasks are run? (instead of the entire 5 subtasks running) Attempts: from GUI perspective we have option called "queue up new tasks" that does the job. I am coming from the script point of view , about how to activate this button/option Could you please provide me with a hint/idea/suggestions (literally anything )
@@MarcLamberti earlier there was an option where we can click on task ( task box in the DAg visualization) and go to the logs of particular task. With the UI upgrade how can I navigate to the logs ?
