Originally presented at Big Data Utah Meetup on January 2023.
www.meetup.com...
Abstract
Data teams often use dbt and Airflow together as complementary tools - dbt (data build tool) is an open-source command-line tool that enables data analysts and engineers to transform data in their warehouse using SQL. And Airflow, traditionally used by Python savvy data engineers, is a tool that allows for the orchestration of data pipelines across various systems. Users of dbt are traditionally analytics engineers, folks who sit closer to the business and are skilled in SQL, but are perhaps less well-versed in Python. And due to the multiple ways in which Airflow and dbt can be integrated, sometimes confusion and challenges can arise, leading to a loss of data observability and/or dependency conflicts.
Cosmos is an Apache 2.0 licensed OSS project from Astronomer designed to generate DAGs (directed acyclic graphs) dynamically from other frameworks. Leveraging Cosmos with dbt in Airflow results in a first-class ETL authoring experience, allowing for native Airflow connections and virtual environment management, as well as native Airflow operators to run dbt commands.
This talk will go through an introduction to Apache Airflow as well as the pros/cons of using different options for integrating Airflow and dbt. We’ll end with an introduction to and demo of Cosmos.
Chris Hronek Bio
Chris Hronek is an experienced Analytics Engineer with a proven track record in collecting, storing, analyzing, and presenting data to meet organizational needs. His professional background includes experience in senior data engineering, data engineering, and business intelligence engineering roles with expertise in Apache Airflow and SQL. Chris also enjoys backcountry skiing, mountain biking, and backpacking in the Uintas.
Ben Garrison Bio
Ben Garrison is a Field Engineer at Astronomer, serving as a technical advisor to folks desiring to improve how they manage, scale and leverage Apache Airflow for all their data pipeline needs. Ben is a recent transplant to Salt Lake City and loves hiking, mountain biking and snowboarding in the glorious Wasatch mountains. Ben is also an avid board game enthusiast and lover of barbecue.
7 сен 2024
