Hi Marc, great video yet again! You mention around 3:35 that you can have multiple replicas of the metadata DB, I thought this was not possible by design. How did you set that up?
Airflow needs a database to work and there is nothing that prevents you from making your database highly available. If you use postgres, you can look at the following doc: www.postgresql.org/docs/current/high-availability.html To be clear, it's not an Airflow thing, it's db thing and it depends on the database you use :)
Hi Marc, I took your course last year, and I don't remember any discussion around dynamic workflow and data lineage. Are these new concepts? Do you have any videos about these concepts? Thanks in advance :)
Hi Marc, I love the way you make Airflow interesting to jump at for a beginner like myself. Am completely new to Airflow world, as a matter of fact, your video on online retail is my first bite. But unfortunately, i got choked 😔 along the line 😭 tried to figure things out but couldn't. The task the error was coming from was in the task that upload gcs to bigqery, this task failed with error decodedencoded utf-8. But when i check BigQuery the raw_invoices table was created but no dataset 😢 wish i could fix this or find way around this. Could you as well be kindly make a video on dynamic task maping as mentioned in this video. Maybe from API to data warehouse. Thanks for the great and concise videos. 🎉
