Crystal Clear Explanation. You have hard coded the version of the table while merging , when you schedule in a pipeline , how to make it dynamic ? is it like get max ( _change_type) and then use it in merge?
Hi Bhavana, great explanation for CDF. Now when update happens on a table , table_changes contains two records update_preimage and update_postimage. I am more curious about how you handle these cases while taking table into next merge. because i can see you take table name and version only.
Hi Thanks for the video on new feature of Data bricks. I have one question , you have hardcoded the version number in code. how can we pass the latest version dynamically in merge part. Thanks in advance
Great video and explanation. I have a question - In Cmd22- For Merge stmt we need to keep updating the version number from 2 to 3 and so on for each incremental load right based on the latest version. How do I do that automatically?
At 18:08, 2nd insert of 7 rows, Six rows are on date "4/1/2021", One row on date "3/1/2021" and they are inserted, how come at 18:28, showing 7 rows inserted, but Six of them are on date "3/1/2021" and One row is on "4/1/2021" ?? I'd expect them to be the same dates as of 18:08 ? Am I missing something or I got something wrong ? Can someone explain to me please ?
