Тёмный

Dagster Crash Course: develop data assets in under ten minutes 

Dagster
Подписаться 2,9 тыс.
Просмотров 21 тыс.
50% 1

If you are looking to get up and running with Dagster in 10 minutes or less, this is a good place to start. Buckle up.
This video is a companion piece to the tutorial found at dagster.io/blog/dagster-crash...
Sourcecode is found on GitHub: github.com/petehunt/dagster-g...
You can launch an instance and follow along on GitPod: gitpod.io/#github.com...
Try Dagster for free for 30 days: dagster.io/lp/dagster-cloud-t...

Развлечения

Опубликовано:

 

9 окт 2022

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 28   
@roberthedblad3044
@roberthedblad3044 Год назад
Would love to have more of these! Unfortunately there isn't a lot of tutorials and resources around to get inte dagster as beginner. So thank you for making this!
@dagsterio
@dagsterio Год назад
Take a look at dagster.io/blog/dagster-google-sheets-tutorial which is an introduction tutorial for Dagster.
4 месяца назад
Joining other comments, I'd love to see more step-by-step tutorials and use cases. It took a few videos to grasp the concepts, and this one is a good one to start with. Docs are good, but videos are even better. I would love to see more of duckdb / dagster and ingestion cases.
@gw1284
@gw1284 Год назад
Thank you for this great tutorial on dagster. Will explore dagster.
@zuesbenz
@zuesbenz 3 месяца назад
fk the learning curve on this shit.
@krtmlry2719
@krtmlry2719 2 месяца назад
is it really hard? Im planning to learn this too. lmao
@jesperbagge2504
@jesperbagge2504 Год назад
Great example of how to actually do meaningful stuff with actual context (no pun intended) completely free from “foo=bar”. Love it!
@raticus79
@raticus79 Год назад
Aha, I was wondering if we could use context with assets like that - the docs had examples for using context with ops but not assets. Good demo, thanks
@ttehir
@ttehir Год назад
Thanks for the very interesting demo. I'm still struggling to grasp some concepts and how they apply to my use cases. Elaborating on your demo, my use case would have additional requirements - the data can't be fetched in its entirety on every run of the schedule, like if GitHub had a volume limitation on the amount of data you can fetch, or rate limiting so that you can't call their API for every day of data - some days, the API might not respond but we still want to retry to fill that in - I also want to run this pipeline for, say, multiple repos. All the logic is exactly the same, but the repo changes. I may have to tweak some extra info other than the name of the repo I've looked into the docs but I'm still confused whether partitioned assets, multi assets for example will allow me to run my use cases, in a way that will solve our problems and remain within the asset based paradigm of dagster. We've used DAGs, partitions and scheduling on a 0.* Version of dagster which worked but came with some issues (partitions growing too large). Now that dagster seems to be moving away from that approach, I'd like to realign our work with the paradigm of the tools we use to ensure we're not using soon to be deprecated features. Thanks a lot if you find the time to help out :)
@kosmylo
@kosmylo 7 дней назад
What is the difference between dagit and dagster-webserver?
@mansibhatt2392
@mansibhatt2392 Год назад
can you make a video how to define sensors in your dagster project..
@hiteshmusale7895
@hiteshmusale7895 Год назад
Good Demo. Actually I am running dbt jobs on dagster in k8s Where i have scheduled some dbt assets which runs fine when it is materialized manually but gets failed sometime when tried to run on scheduled time. Though Dagster daemon is also set up. Do you have any inputs on it ? Appreciate your help.
@dagsterio
@dagsterio Год назад
Hi - Join us on Slack at dagster.io/slack and we can support you there.
@user-if2kq8nh8m
@user-if2kq8nh8m Год назад
Why was it decided to showcase this demo with "@repository" as opposed to "Definitions"?
@selimrbd
@selimrbd 4 месяца назад
Repositories were the previous way of defining code locations (at the time this video was made). The new way is using Definitions
@jesusmartinalcala4730
@jesusmartinalcala4730 Год назад
When i execute dagster project scaffold --name ....., in the created directory does not appear workspace.yaml, how can i fix that? Thank you in advance
@dagsterio
@dagsterio Год назад
As long as you have `[tool.dagster]` pointing to the right Python module in `pyproject.toml`, you don't have to use `workspace.yaml` since Dagster 1.1.7 Details: We introduced `Definitions` to replace `@repository` and `workspace.yaml`. `workspace.yaml` is no longer the preferred solution for convenient command-line-free usage of `dagit` and other tools. Instead we just rely on a `[tool.dagster]` of `pyproject.toml` a Python standard design with extensibility points with these uses cases in mind. However, if you do want to use the older pattern, we're still supporting `@repository` and `workspace.yaml` for the foreseeable future. You can find more details in github.com/dagster-io/dagster/discussions/11167
@datalearningsihan
@datalearningsihan Год назад
I am trying to install it. It does not work. Had to spin up docker, still does not work. Whats wrong. Can I get an actual non buggy guideline?
@dagsterio
@dagsterio Год назад
Hi Sihan, Feel free to join us on our Slack channel (dagster.io/slack) and we can help you out. Hundreds of users have successfully completed this tutorial so we are confident we can get you unstuck! We will just need some details on what your issue is.
@TheEsisia
@TheEsisia Год назад
Am I the only one finding it weird that you basically coding inside the _init__.py file?
@dagsterio
@dagsterio Год назад
It's definitely a shortcut we sometimes use in demos to keep things simple. If a fully built-out project, you would use a full file structure facilitated by the scaffolding process.
@aamirahmadansari2721
@aamirahmadansari2721 Год назад
I am disappointed with the content as it was rushed. As a beginner, I would have loved a more structured and detailed tutorial/ crash course.
@dagsterio
@dagsterio Год назад
Hi Aamir. Sorry if the 'crash course' seemed a bit rushed. Were you watching the video embedded in the blog post here: dagster.io/blog/dagster-crash-course-oct-2022 ? If not, the written version will help. Also we would recommend the following tutorial which is more details, and geared towards people brand new to Dagster who want to take things one step at a time: dagster.io/blog/dagster-google-sheets-tutorial
@aamirahmadansari2721
@aamirahmadansari2721 Год назад
@@dagsterio Hey, I will definitely have a look at it and would request the team to please add a detailed version of the video for mlops. I like the premise of Dagster and would like to pitch it in my company. Your documentation is real good but a video tutorial would be very helpful.
@dagsterio
@dagsterio Год назад
Agreed. We have lots of Dagster users running ML pipelines, and we hope to bring more case studies to the community in the coming weeks. In the meantime, did you see this blog post from Sandy (Lead eng on the Dagster project)? dagster.io/blog/dagster-ml-pipelines Join us in the Slack channel and we would be happy to discuss the use case: dagster.io/slack
Далее
Converting an ETL script to Software-Defined Assets
26:16
СКАМ НА TWITCH
14:07
Просмотров 289 тыс.
아이들은 못말려 〰️ With #짱구
00:11
Просмотров 567 тыс.
когда мучает жажда // EVA mash
00:58
SH: Intro to Golang!
2:13:31
Просмотров 244
Intro To Data Orchestration With Airflow
53:56
Просмотров 8 тыс.
тгк: katylazarevaa
0:16
Просмотров 1,8 млн