Nick Schrock (founder and CTO of Dagster Labs) shares details on two new powerful features of Dagster.
Read the blogpost on Dagster Pipes here: dagster.io/blog/dagster-pipes
Read the blogpost on External Assets here: dagster.io/blog/dagster-exter...
1) External Assets - the ability to tap into Dagster's control plane observability, lineage, and data quality capabilities without adopting Dagster as the orchestrator across all environments. You can now expand Dagster's span of control to legacy systems without having to do an infrastructure migration.
2) Dagster Pipes is a protocol to invoke remote computations seamlessly in hosted execution engines and remote environments. This reduces the incidental complexity of dealing with external computations and lets you bring in computations in any programming language.
00:00 Introduction and recap of Dagster Launch Week
03:10 Deconstructing Dagster:
05:05 The three layers of Dagster's system
07:06 Separating the layers of the stack (External Assets and Pipes)
08:54 An example use case for External Assets
14:34 Dagster Pipes overview
28:06 Coding example
35:57 What's launching this week?
Try Dagster today with a 30-day free trial: dagster.io/lp/dagster-cloud-t...
10 июл 2024
