hey awesome vid! thanks I follow you! with dataflow can i aim a specific schema on the warehouse destination? or i should combine pipeline and dataflow for that? thx
Thanks for the wonderful videos on Microsoft Fabric. I see that all the data imports are "one time activities" from a source. How can we get the delta data (eg. new records, deleted records) in the source syncd with the lakehouse periodically? i.e., What about the continuous CRUD operations being done on the source (eg. Sql DB) ? How can that be synced with the data in the lakehouse?
Thanks fo your comment! Yes you're right, they are one-time activities (but you can schedule then to run every day/ hour etc, but it's never going to be 'real-time' The feature you are describing is more like OneLake shortcuts (which you can create a real-time link to FILES in external locations (ADLS and Amazon S3). Microsoft are also releasing soon a feature called Database Mirroring which will do the same thing, but for databases (Snowflake, CosmosDB, Azure SQL etc) - this feature is currently in private preview, I believe they will release it soon for public preview!
Hey! Could be a number of reasons! Does the dataflow run ok outside of the pipeline? I would look in Monitoring Hub and analyze the error message. Good luck!
Thank you for sharing! I didnt load my data into the azure storage account - i used DFg2 to read/upload the CSV file. Do you guys experience slow delta writing to the lakehouse? [Can I do something to speed it up]?
Hi Yes in general the Dfg2 is quite slow at the moment during the public preview, I’m sure the write speeds will increase as we move closer to GA (general availability) of Fabric
