Mark walks through a demo of the new top-level ADF resource for CDC (Change Data Capture). Build a Change Data Capture process in #Azure #DataFactory in minutes without a pipeline, data flow, or a trigger!
This is an amazing feature to add. With this in place it's a case of comparing the performance of the CDC with the need for a compute (in most cases a merge) to update the existing records with the new/updated record brought across and how this will perform against a simple copy statement which would use Polybase (in the case of synapse dedicated or azure SQL DW). This needs to be added to the options in synapse pipelines (via synapse studio) considering this is intended to be the home for future data development.
Are there plans to add full load capabilities directly within the CDC Resource? Currently, I have to establish a distinct pipeline / data flow for full loads. Incorporating this feature would greatly simplify the process.
Hi - I have setup as SQL Azure Database source and Managed SQL Instance target but it does only capture inserts and not delete or modifications on rows. So just replicate only inserts
Hi, I am using a Parquet source and an Azure SQL sink, it is unable to detect deletions between files. I have applied a key to the ID and this still cannot detect them. Is this something that will be resolved soon? I think there are many businesses ingesting from third party sources facing a similar issue where capturing incremental changes including deletions is of huge importance.
If I am not wrong, We can't use this when we are ingesting data from external source like other RDBMS servers like Oracle, and API based ingestion like salesforce?
