Wow! This was the best explainer so far! I'm interested in tracking when other external sources will be available via shortcuts because my organization keeps data warehoused in other (non-Amazon) sources.
This is so cool! I can see this being a quick but powerful win for small businesses that want to harness their data without having a ton of overhead in getting services up & running!
What's the performance impact of having data as shortcut compared to having these in one lake when we use it in different engines especially direct query on power bi
At a guess, copying the data into one lake is better for data manipulation. Transformations, and selects etc. Referencing the tables as a shortcut is akin to using a server less SQL instance with external tables. Physical data moving, indexing, and keys may not be possible. I'm guessing here though based on limited knowledge currently.
There's an on prem SQL Server option to copy data into a lakehouse in fabric. I'm not sure how that relates to one lake as I've just started this video and still getting used to Fabric terminology
Does anyone know if DirectLake is as performant as the import mode? As far as I know, each workspace is located in one region: do we pay cross regions transfer costs when using a shortcut between two workspaces located in 2 different regions?
Is my understanding correct that structured data is stored in Onelake in .Parquet files? If so how efficient is it as compared to storing data in RDBMS like SQL server?
It looks fantastic. I have a question. I use to store my data in Azure SQL. Is it better to migrate my data from Azure SQL to OneLake or it's better to stay like it is now.
All files are converted to parquet and then delta parquet. So basically you have to build an ELT framework for handling this... Really anoying it isn't done automatically and then generate scripts.. Like 10% extra developer time comñared to a storage blob into sql database - Sigh 😢
Amazing vid. Very informative. Question: Is it possible to have a deeper dive into the inner workings? All the Marketing and Sales sessions/materials/etc. repeat the line (or something similar) that: "it's one OneLake and you can have one Fabric on top". Josh mentioned in this video that whoever owns (uploaded) the data, say to your store DB, continues to own it and the OneLake consumer gets a copy. I'm curious on how this is actually working. The last thing I want is our data and analytics team "connecting" to our application data stores and running crazy queries on our operational data. Given Josh said it's copied data, I'm curious to know more and what that means for debugging missing data, eventual consistency, atomicity, etc. Thanks!
Do you see anything on the horizon to allow an ISV to provision a onelake, setup domains and grant external guest users (the ISV's customers) access to different paritioned datasets? In otherwords is there anyway for an ISV to grant lakehouse to their SaaS customers?
Hey Adam, does using one lake and direct lake storage mode in power bi semantic model allow to connect model to really big tables (hundreds of millions of records) while keeping visuals and dax fast and efficient? Thx in advance!
I like the shortcut table feature! Is there also a way to create shortcut tables to REST APIs or OData web APIs so that I can get the latest data from source systems like Sales Force or Business Central without copying the data on a schedule into OneLake where it will eventually get outdated or is missing columns I need at all?
If we use a different source other than S3 like ADLS for example can we still load the datasets directly from those external sources within our Power bI REPORT.
When clients (say power bi) consuming a shortcut data source, data is transferred to one lake first, then forwarded to clients, right? If it's true, it sounds like copy the data eveytime consumes.
Thank you so much for the demo!! I have a question regarding a potential use case for this. I have a client that has multiple data locations, they're still moving into the cloud scene however they currently utilise an Access Db for a particular set of data. If this db was hosted on a vm within azure, could OneLake potentially reference data from Access Db?
Slightly out of sync "Yo" at the beginning of the video. Literally unwatchable. p.s. brilliant content, thank you so much for providing so much insight and material into such a new product
not convinced about this yet, shortcuts are nice in theory, but performance will never be as good as if the data was imported.. secondly, look at the video at 9 min and 25-30 secs , the insert took about 5 seconds. i know we are nerds and we like new shiney things, but lets balance that with being practical.
Datalake and delta lake sucks imo. It's not drag and drop really. If you load incremental changes from a source like SAP or Oracle or API. You have to build an integrator in Data Factory, SQL scripts in Synapse to convert to Parquet files, do Transformations, Implement a Delta logic and Save in Delta Parquet, Then transform the data again, Build views, and then you can consume the data. I much prefer Data storage gen 2 or Storage Blobs and a regular SQL Database and do the transformation in Synapse.. Now you are reliant on Data Factory+Synapse+Parquet logic+OneLake+Synapse+Synapse or SQL serverless And It is super slow to read Parquet since It's basically thousands of KB size files to merge like heap pages in an SQL database.
