Great video guys, keep them coming. Not sure why your confused over the Lake explorer tool etc. Its horses for courses. People might not want to explore Delta files in a Lake explorer tool, but the Lake explorer also shows you the other FILES section, where you can access non delta files (csv, txt, images etc). People might use it to upload/download stuff. Random client data that might need to analysed or processed etc. Or, you might have some desktop tools that you want to connect to the Lake files that cannot use abfss urls, so those tools will see files on the local desktop path that happen to sit in the lake (if the Lake explorer tool appears as a drive in windows?). You questioned the value of seeing the names of delta files behind a table, and why anyone would need that? In SQL Server you can get the name of the mdf/ldf storing all the table data? What can you do with that information? Its just completeness I guess. Seems like the abstraction is where the benefit lies. Tables (SQL), Tables (delta), datasets(powerbi), all underpinned by delta files that you dont need to concern yourself with, it appears. This is a SaaS service designed to work across a multitude of personas, from highly skilled to unskilled. Not sure always critiquing the concept with a data engineers view will always be valuable. But thx for getting out this content, really helpful to see it being demonstrated.
The ability to see those things isn't the area I'm questioning - from the engineer's standpoint it's always useful to be able to view and analyse the underlying files when performance tuning etc. My concern is for the personas who DON'T want to see those things, or for the people who absolutely SHOULDN'T see those things. As long as we can apply security properly so users can still create new tables (through SQL, Lakehouse, whatever) but can't drop new files or delete files within the formal delta structure, then we're all good! For local IDE/Library use cases, it's super useful, for maintenance and governance it's super useful. I see the value, I'm just old and cynical :) - Simon
hi there, thanks for the amazing content! when reading data in databricks with abfss path, how did you get authenticated? what sorcery is this? passthrough?
Great video, like you keen to under how backup and restores work on all the stuff in onelake. Until there's a disaster recovery plan for the Synapse datawarehouse we can't move our dedicated pools over.
Thanks for the video :) This is just a more convenient version of azure storage explorer. BUT like storage explorer, bet it logs you out every single flipping 8 hours. Our lake consists of pretty small datasets for the most part. Some ad-hoc python on local windows in vscode is often all i need.
Great video guys 👍 If you import another adls G2 account into your one lake, what does the abfss link look like? How do onelake and the onelake explorer manage permissions & ACLs, particularly in imported adls G2 accounts?
I'd like to get into an example implementation of a lakehouse with pipelines in fabric. How does a standard Databricks lakehouse implementation look in fabric?
Don’t you got issue with your Fabric Workspaces named « UK Sales » « EUR Sales » ? Since in Fabric the workspace name is a Gen2 Container it’s doesn’t like too much the UPPER case and space I noticed
