Developing Airflow DAGS in Isolation With lakeFS 

15:20 - 15:50 | DATA ENGINEERING THEATRE
DEVELOPING AIRFLOW DAGS IN ISOLATION WITH LAKEFS
WEDNESDAY 20 SEPTEMBER 2023
SPEAKER: GUY HARDONAG, lakeFS
As Apache Airflow establishes its stronghold as an essential tool for orchestrating complex data workflows, it is increasingly recognized for its critical role in data engineering ecosystems. However, if you're familiar with Airflow, you know that testing and debugging DAGs can be both intricate and cumbersome.
In this session, we will delve into how lakeFS, with its robust and scalable data versioning capabilities, can become a game changer when testing and debugging DAGs.
We will demonstrate how you can test your DAG in isolation on up-to-date production-scale data, all without disturbing the end user and consumer. lakeFS will also demonstrate how introducing schema or infrastructure changes becomes a non-issue when the power of version control is added to your data lake.
Join us as we demonstrate the capabilities and full potential of lakeFS to help you revolutionize your Airflow workflows.