Databricks Asset Bundles: Advanced Examples

Подписаться 2,5 тыс.

50% 1

Databricks Asset Bundles is now GA (Generally Available). As more Databricks users start to rely on Databricks Asset Bundles (DABs) for their development and deployment workflows, let's look at some advanced patterns people have been asking for examples to help them get started.
Blog post with these examples: dustinvannoy.com/2024/06/25/d...
Intro post: dustinvannoy.com/2023/10/03/d...
* All thoughts and opinions are my own *
References:
Datakickstart DABs repo: github.com/datakickstart/data...
Data & AI Summit Presentation: www.databricks.com/dataaisumm...
Data & AI Summit Repo: github.com/databricks/databri...
More from Dustin:
Website: dustinvannoy.com
LinkedIn: / dustinvannoy
Github: github.com/datakickstart
CHAPTERS
0:00 Intro
1:52 Monorepo project setup
3:09 DABs basics (review)
6:32 Deploy and View in UI
11:28 Config Re-use (YAML Anchor)
16:09 Shared Python Package (Wheel)
19:04 Serverless Compute
20:54 Modify in UI
23:10 Create in UI
25:37 Extra tips
27:38 Outro

Наука

Опубликовано:

11 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 11

@NoahPitts713 10 дней назад

Exciting stuff! Will definitely be trying to implement this in my future work!

@etiennerigaud7066 13 дней назад

Great video ! Is there a way to overide variables defined in the databricks.yml in each of the job yml definition so that the variable has a different value for that job only ?

@ameliemedem1918 14 дней назад

Thanks a lot, @DustinVannoy for this great presentation! I have a question: which is the better approach for project structuration: one bundle yml config file for all my sub-projects or each sub-project have its own Databricks and bundle yml file? Thanks again :)

@gardnmi 16 дней назад

Loving bundles so far. Only issue so far I've had is the databricks vscode extension seems to be modifying my bundles yml file behind the scenes. For example when I attach to a cluster in the extension it will override my job cluster to use that attached cluster when I deploy to the dev target in development mode.

@DustinVannoy 16 дней назад

Which version of the extension are you on, 1.3.0?

@gardnmi 16 дней назад

@@DustinVannoyYup, I did have it on a pre release which I thought was the issue but switched back to 1.3.0 and the "feature" persisted.

@fortheknowledge145 16 дней назад

Can we integrate Azure pipelines + DAB for ci cd implementation?

@DustinVannoy 16 дней назад

Are you referring to Azure DevOps CI pipelines? You can do that and I am considering a video on that since it has been requested a few times.

@fortheknowledge145 16 дней назад

@@DustinVannoy yes, thank you!

@9829912595 15 дней назад

Once the code is deployed it gets uploaded in the shared folder can't we store that some where else like an artifact or storage account because there are chances that someone may deleted that bundle from shared folder. It is always like with databricks deployment before and after asset bundles.

@DustinVannoy 15 дней назад

You can set permissions on the workspace folder and I recommend also having it all checked into version control such as GitHub in case you ever need to recover an older version.