I also love that the guy says "you don't need to wait for an engineer to build out a data warehouse"...then proceeds to show a star schema model based on clearly cleansed and normalized tables.
Yes. Though it is simpler because it work like Power BI (PQ + DAX and if you want SQL). You still will need someone who really understands both the source and propper datamodelling.
The "power" of the Datamart demoed here is that someone who isn't able to write code or wait for others can now build or contribute to building robust analytical solutions rapidly. Cleaning and prepping data, and then building out star schemas can now be done without writing code and through a visual interface, enabling folks with varying degrees of technical expertise to self-serve with Office like familiarity.
This is so powerful. I've been struggling with a query in Power BI desktop that was brining my laptop to crawl, maxing out my CPU, 16gigs of memory, and also writing to my disk 100%. This query ran for over 3 hours before I abandoned it and had to get other work done. I moved the query into a Datamart last night (took a while as I encountered some errors) and the process time was about 20 minutes. Pushing all of this calculation and compute into the cloud is, like others here have said, GAME CHANGING. I can now do tasks that were either impossible or completely impractical. Thank you so much!
There's so many exciting things here. One that stands out the most is the ability to do all of these typical tasks in a browser whereas previously restricted to PBI Desktop..? Amazing.
That's right. It means that you can kick off processes from the web service and not worry if for example you lose connection or accidentally close the tab.
So now instead of spreadmarts we are going to have a bunch of ad-hoc Azure SQL Database out there. :( Hopefully this feature can be disabled for enterprises that don't want all of these Azure SQL DBs eating up IT's Azure budget. I don't know of many analysts/business users that are going to be able to create an efficient star schema or even want to build a bunch of Power Query. I get the need for small teams that don't have a structured IT or an IT group that is too busy, but this can open up a whole new mess of issues with people creating their own databases without IT's knowledge/oversight. MSFT loves this, more $$$ with all of these new Azure SQL DBs.
My thoughts exactly. Not all of us "Data Engineers" take weeks to turn stuff around for our users. And a little bit governance does go a long way toward ensuring some organizational consistency. Methinks copies of Tragedy of the Commons should be distributed and made required reading over there in Redmond.
what's the difference with Dataflows? I mean, if someone is able to work with this, is also able to access and create a dataflow. I don't clearly see a filled gap here.
Also interested to see how dataflows fit in - I don't think they do - this can replace them entirely methinks if you can connect and choose individual tables to bring into your model.
Since it can do T-SQL codes in the background, can it also perform Table Views, Functions, and Stored Procedures inherent in an On-Premise SQL? And if so, (1) how can I trigger the Stored Proc? can I still use the old-school technique of batchscript-vbscript/python combo via task scheduler, use an ETL tool (e.g. SSIS, Lavastorm etc.), or do I use the Power Automate for it? (2) If I create table view to store a complex SQL script can I create a new table and use Select statement on that table view? This is an interesting development and thanks in advance. Cheers.
We are still in the process of rolling this feature out in several regions. So you should be able to start seeing the new Datamarts capability by the end of the week
Maybe I missed it, but where is the data stored? Does data persist? Can you do upserts? Keep deleted records? Slowly Changing Dimensions? Snapshot Facts? Aggregate Facts? conformed dimensions?
Awesome. Does somebody know if and when insert/update/delete into the underlying Azure SQL tables will be available with SSMS/Data Studio? Currently I only see the imported exposed as views in SSMS. Would be really great to be able to insert data into it programmatically with external tools.
Thank you for this video!! However, I have questions: - Do we need a gateway for datamarts syncing? If no, that will be great in fact - Can we do ETL on an existing/external Azure SQL database?If yes, this feature is a revolution !! I think the only situation we need on-prem gateway is when use have data behind a VNET, please correct me if I am wrong! Thank you again!!
Dataflow is your old school ETL like experience where you can connect to it innstanard pbi dekstop and datamart is kinda like full process (ETL, modeling etc) so its broader experience
Dataflows allow self-service users to build reusable and shareable data prep/ETL. Dataflows load the data into the Data Lake. Datamarts allow self-service users to build a Datamart/analytical SQL database that can be useful to do relational analytics, both within and outside of Power BI. Dataflows = ETL/data prep Datamart = Destination (fully managed Azure SQL database that we provision for you) You can connect to an existing dataflow and load that data into a Datamart if others in the organization want to query the data for ad-hoc analytics using SQL/no-code in addition to Reporting.
I am a PBI user through and through and have solved a lot of my ETL problems with rather advanced SQL queries. Through my work I've recently taken a course in ODI Developer. This seems very similar but more streamlined and better UX. Should I just give up on ODI and try to nudge our data warehousing towards this (we do have premium)?
@Gustavo leo - We're still rolling this out and you should get this by the end of the week. You should see the option from your Gen 2 workspace to create a new artifact using the +New button (right next to Dataflows, Datasets etc.). No support through PowerShell Cmdlets yet. Datamart isn’t currently supported in the following azure regions: • Sweden Central and South • West India • UAE Central • Japan West • Southeast Asia
Ok this is awesome, but who do you control if every Deparment in a company has a particular datamart build by them, and how Data Governance is impacted by this? Does this mean no more DW - Centralized DB?. Who is going to maintain order in this open field feature?
Does anyone know if I could create a Datamart using PPU in one Workspace, produce a Shared Dataset in a Pro Workspace based on the PPU Datamart data and then share that Dataset with Pro users? Basically I have a PPU license but the majority of the end users in the business have Pro.
You can create this Datamart in a PPU (Premium per user) or PPC (Premium per Capacity) workspace (Gen 2). You can share this with a recipient and the recipient will get an email. The recipient who clicks this link (to access the Datamart for downstream consumption) needs to have either a Pro or PPU license to view the Datamart info page. There are 2 things that the recipient can access: a) Access to the Datamart's SQL endpoint (from Power BI Desktop or SQL client tools such as SSMS) b) Access to the Datamart's underlying auto-generated dataset so that the recipient can create reports within Power BI Service or from Power BI Desktop.
This sounds incredible. Love to try it out. But on my Premium Workspace I don't see the option to create a Datamart. Is there something I need to do to activate it? Thanks in advance.
Thanks Alexander. You should be able to try it if you create from a Premium Gen 2 workspace. Check the settings to ensure its Gen 2. (won't work from My workspace)
@@watchthewaj Thanks for the answer. It's Premium Gen 2 (P1 plan) and I'm looking in normal workspaces (not "my workspace"). But it doesn't show :( Is it US-only preview maybe? I'm in Germany. Or are there any other prequisites like Azure Data Lake or similar?
@@DocKeul Same issue here. We have P1 and Gen 2 workspace, but i cant see the new datamart button on new menu :( Hopefully its still beeing rolling out to all tenants.
@@DocKeul - There are no other pre-requisites. We are still rolling out Datamarts to more regions and you should be able to access this feature by the end of the week. Thanks!
@@watchthewaj you may need to ask your tenant admin to enable as well. I'm the admin and it was on by default, but looks like I can turn off datamarts for the org.
