34. Change Data Feed Demo 02

Подписаться 19 тыс.

Просмотров 6 тыс.

50% 1

Follow me on Linkedin
/ bhawna-bedi-540398102
Instagram
www.instagram....
A medallion architecture is a data design pattern used to logically organize data in a lakehouse, with the goal of incrementally and progressively improving the structure and quality of data as it flows through each layer of the architecture (from Bronze ⇒ Silver ⇒ Gold layer tables). Medallion architectures are sometimes also referred to as "multi-hop" architectures.
Change Data Feed (CDF) feature allows Delta tables to track row-level changes between versions of a Delta table. When enabled on a Delta table, the runtime records “change events” for all the data written into the table. This includes the row data along with metadata indicating whether the specified row was inserted, deleted, or updated.
Data-bricks hands on tutorials
• Databricks hands on tu...
Azure Event Hubs
• Azure Event Hubs
Azure Data Factory Interview Question
• Azure Data Factory Int...
SQL leet code Questions
• SQL Interview Question...
Azure Synapse tutorials
• Azure Synapse Analytic...
Azure Event Grid
• Event Grid
Azure Data factory CI-CD
• CI-CD in Azure Data Fa...
Azure Basics
• Azure Basics
Data Bricks interview questions
• DataBricks Interview Q...

Опубликовано:

11 сен 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 9

@dpatel9 5 месяцев назад

How come you know that I need to add 2 and 5 as commit version? How to make it dynamic like real world scenarios?

@AshokG12 9 месяцев назад

gr8 video, i would suggest to implement delete as well to gold table, currently deleted record from silver table is still showing up in gold table

@ravikusma3598 2 года назад

Do you have Git repository of your Notebooks

@ShwetaGarg-s4w Год назад

Can you share how to do this dynamically? Also what about rows which are deleted from source? How does this change flow to target?

@chetanambi 11 месяцев назад

Wonderful channel! I have watched all the videos on Databricks. Keep going! Just one question on this video. You are manually mentioning the versions. How this will be done in dynamically in production scenarios?

@DevarshShah-zo3qm 10 месяцев назад

If you are asking about the versions that she is mentioning here Select * FROM table_changes('table_name', ) here we have the option to mention the timestamp as well instead of version like this getting all the changes between the given timeframe SELECT * FROM table_changes('tableName', '2021-04-21 05:45:46', '2021-05-21 12:00:00')

@t-bar-mranch1074 4 месяца назад

@@DevarshShah-zo3qm This did not answer the question. I had the same question. Finding the versions or timestamps insnt the question. The question is about her hardcoding the versions. in a production environment, you wont know how many versions there are so there has to be a way of checkpointing to know how to get just the changes dynamically