29. Schema Evolution in delta Tables

Подписаться 18 тыс.

Просмотров 7 тыс.

50% 1

If you need any guidance you can book time here, topmate.io/bhawna_bedi56743
Follow me on Linkedin
/ bhawna-bedi-540398102
Instagram
bedi_foreve...
You can support my channel at: bhawnabedi15@okicici
Schema evolution is a feature that allows users to easily change a table’s current schema to accommodate data that is changing over time. Most commonly, it’s used when performing an append or overwrite operation, to automatically adapt the schema to include one or more new columns.
A schema defines the data structure contained inside the dataframe.
Data-bricks hands-on tutorials
• Databricks hands on tu...
Azure Event Hubs
• Azure Event Hubs
Azure Data Factory Interview Question
• Azure Data Factory Int...
SQL leet code Questions
• SQL Interview Question...
Azure Synapse tutorials
• Azure Synapse Analytic...
Azure Event Grid
• Event Grid
Azure Data Factory CI-CD
• CI-CD in Azure Data Fa...
Azure Basics
• Azure Basics
Data Bricks interview questions
• DataBricks Interview Q...

Наука

Опубликовано:

18 мар 2022

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 17

@okachovik 2 года назад

You are good at this keep up the good work .explanation is crisp and clear and content is bang on .

@samk_jg 2 года назад

Thanks alot for such awesome tutorials !!👋

@leelakrishna8069 2 года назад

Thanks for your videos madam. It's giving good insights.

@padmsrikanth1672 2 года назад

Nice explanation, thanks

@fratkalkan7850 2 года назад

thank you so much

@TrendingSoftwaresTrainin-bl4gu 2 месяца назад

how to maintain complete history for daily insert data into target delta table? scd type -2 , can you provide separate video

@thotarohith2060 7 месяцев назад

As you mentioned that mergeSchema helps when we have change in schema while writing. Suppose I have a use-case which I need to handle the dynamic schema on read. How can we handle that without using inferSchema,(because as we know that when we use inferSchema spark will run extra job ). Can you please help me in handling this situation

@arundasari9025 3 месяца назад

Is their a way to do schema evolution only on delta table without writing the data to delta table

@gyan_chakra 2 года назад

Another great video mam! 👍 One query - Can we use this mergeSchema option during Upsert operation? If yes, what will be the syntax for that?

@mahesh.khatai93 Год назад

In case of Delta Schema Evolution - will it preserve the history of data in new table along with the newly added column ? i.e is it re-creating the target table again in background with new columns and then loading the data to it . Ex - Existing 10 cols + 1 new col = 11 cols in new table . Also how does it handles removal of column as well from existing table . Or its only restricted to adding cols .

@rahulmittal116 5 месяцев назад

Revision - option(mergeSchema,”true”) for Schema evolution so that there is no error in case of schema change while overriding/appending Delta tables in Data bricks.

@rakeshgopidi4066 Год назад

well explained again if the code snippet has been shared would have been much helpful github , one questions can we perform all the tasks using Community edition databricks instance ?

@susmitabiswas1977 9 месяцев назад

Inside cell 7: loans = sql(" ... ---> does this sql stand for spark.sql or databricks SQL API

@iamramanr4s Год назад

Error in SQL statement: AnalysisException: The schema of your Delta table has changed in an incompatible way since your DataFrame or DeltaTable object was created. Please redefine your DataFrame or DeltaTable object. I'm getting above error can u please provide any solutions for it, or anybody