Data Engineering with the Open Source Modern Data Stack (From MDS Fest '23) 

The Modern Data Stack gives us a wide range of free technology for data ingestion, data storage, data transformation, data orchestration, and data visualization.
Pedram Navid, Head of Data Engineering and DevRel at Dagster, walks us through one data pipeline design using a number of open-source solutions, including Dagster, dbt, PopSQL, and the Mastodon API, DuckDB, dbt-duckdbt, Evidence, Sling and Steampipe.
The pipeline is used to analyze bird observation data from the Cornell Lab.
Github Repo: github.com/dagster-io/mdsfest...
Source Data: feederwatch.org/explore/raw-d...
Try Dagster today with a 30-day free trial: dagster.io/lp/dagster-cloud-t...

Наука