Watch the video for the new Warehouse Destination Microsoft Fabric Part 18: Dataflow Gen2: Native warehouse Destination- Power Query Transformation - Pivot, Unpivot, Transpose Data ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-U6fsHSiNpz4.html
Nice. quick question, why we are using Warehouse and also in another video we are using Lakehouse. in which use cases we have to decide on this? Also is this warehouse dedicated SQL pool like we jhave on synapse or is it just a lakehouse platform where warehouse is created?
Microsoft's Fabric architecture offers several data management options that cater to different needs and scenarios. Here's a breakdown to help you decide between using a Data Warehouse and a Lakehouse: Lakehouse Architecture: Scenario: When you need a unified platform for both big data analytics and machine learning. Data Stages: Typically divided into Bronze (raw data), Silver (cleaned data), and Gold (business-ready data). Tools: You can use Spark SQL for SQL queries or Spark Python for ETL processes. Data Warehouse: Scenario: Best for scenarios where you require traditional SQL databases for transactional processing or when specific SQL performance optimizations are needed. Tools: SQL (like in DB SQL) is used. Data Flow: For instance, you might move from Lakehouse (Bronze) to a hybrid Lakehouse/Warehouse (Silver) and purely Warehouse (Gold) for optimized querying and performance. Choosing Between Lakehouse and Warehouse: Full Lakehouse: Ideal for complete big data and machine learning pipelines, utilizing tools like PySpark for transformation. Warehouse: Preferable if your focus is on leveraging SQL capabilities for complex queries and transactional data processing. Tools and Integration: Dataflow Gen2: This can be used for data transformation in both Lakehouse and Warehouse setups. Notebooks/Spark: Currently, these are more aligned with Lakehouse setups and do not directly interact with traditional Warehouses. In summary, the choice between Lakehouse and Warehouse depends on your specific data processing needs, the scale of data, and the analytical tools you prefer. Lakehouses are versatile for handling large-scale data and machine learning workflows, whereas Warehouses are more suited for intensive SQL querying and transaction-based operations.
Thank you very much Sir Amazing video with a detailed explanation. I have covered each piece of information from this video. Looking forward to learning more from you.
Hey Amit, I am trying to use Dataflow Gen2 to update a Warehouse (not LakeHouse). I'm following exactly what you did, so using the SQL Connection string with target Azure Synapse (SQL DW). However I get this weird error: "An exception occurred: Microsoft SQL: This function doesn't support the query option 'EnableCrossDatabaseFolding' with value 'true'. " I cannot find anyone who had such an error...I tried different options, no luck. Did you get that by any chance?
I tried again and able to save the Dataflow Gen2. Hope you are getting error before that. Make sure you have taken correct URL and opted for Organization Authentication. Also make sure a warehouse is created; add a table using create statement and check it is working . If everything is fine, connect to me on LinkedIn and share screenshot. I would check the issue www.linkedin.com/in/amitchandak78/
Please refer to full series. Most of the options are explored Mastering Microsoft Fabric 31 videos: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-p-v0I5S-ybs.html
Warehouses are like databases, where data is stored in a structured format and can be accessed and modified using SQL queries. Lakehouses are like data lakes, where data can be stored in a variety of formats, including structured, semi-structured, and unstructured. Structured data in a Lakehouse can be accessed using SQL queries(Read Only) . I am loading the same data to show case various features. Refer first and third video of series ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-p-v0I5S-ybs.html
Hello Amit... When you created a table using SELECT INTO or created a view in Warehouse by copying query from Lakehouse.. I believe it is not actually pointing to Lakehouse. You have the tables with same name present in Warehouse and it actually sourced data from there... Not sure but you can try appending Lakehouse before schema name and see if it works... Though it doesn't work on Azure Synapse
Yes, seem like tables are there at both places. But with the same workspace, you can query tables across LakeHouse and the warehouse, for example ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-KIbt9GzAdWk.html
It looks like the Azure Synapse Analytics (SQL DW) option as a data destination for a Dataflow Gen2 has been replaced with Warehouse, making the process more straight forward as we only need to choose the Warehouse name now.
Yes, Agree. New one is much better. Discussed in the recent video - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-U6fsHSiNpz4.html Hope you will like the full series Microsoft Fabric: A Brief Introduction- ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-p-v0I5S-ybs.html How to enable Microsoft Fabric; on Power BI Service| Start the Fabric (Preview) trial ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-8kIMHmkQ9m4.html Microsoft Fabric-What it is, Why Use, Who Should, When and How to use ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-sbWNJMaspeA.html Microsoft Fabric: Various Components| Microsoft Fabric Create Workspace, Lakehouse, Warehouse ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-pVvvb-MzxKw.html Microsoft Fabric part 5: How to load data in Lakehouse using Dataflow Gen2 ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-aPI6NT3lfsE.html Microsoft Fabric Part 6: How to load data in Warehouse using Dataflow Gen2 ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Mn1GB8rSeoQ.html Microsoft Fabric Part 7: How to load data in Warehouse using Data Pipeline | End-to-End Scenario ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-idBrer4NrnY.html Let's have a spark in fabric- Microsoft Fabric Part 8: How to load data in Lakehouse using Spark; Python/Notebook ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-jWDFcpK_k9I.html Microsoft Fabric Part 9: Integrated Machine Learning- k-Means clustering: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-lsnvQ4UoTwo.html Microsoft Fabric Part 10: Create a Date table in Dataflow Gen2, and use in Lakehouse, and Warehouse| Time Intelligence with Power BI ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-g3BEVXQdk-g.html Microsoft Fabric Part 11: How to load data in Lakehouse using Data Pipeline | End to End Scenario: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-jn_D2-b5YpY.html Microsoft Fabric Part 12: What are the options to load local files in Lakehouse: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Pi5IvkHxJBs.html There is no shortcut to success. However, in MS Fabric, you can shortcut to Azure DataLake Microsoft Fabric Part 13: Shortcut- within Fabric/ Azure Data Lake: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-KIbt9GzAdWk.html Microsoft Fabric Part 14: Import Azure SQL Data to Warehouse | Multiple tables using Pipeline- ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-4wj4b3EvFpI.html Microsoft Fabric Part 15: Incremental Data Load(ETL) for Warehouse using Dataflow Gen 2, SQL Procedure, Data Pipeline: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-mpFRnZBXsvQ.html How about incremental without Dataflow, Only using pipeline and SQL procedure for Microsoft Fabric Warehouse Microsoft Fabric Part 16: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-qsOIfTzjCSQ.html Microsoft Fabric Part 17: Load local SQL server data in Lakehouse using on-premise Gateway and Dataflow Gen2: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-oEF-jHVmvdo.html Microsoft Fabric Part 18: Dataflow Gen2: Native warehouse Destination- Power Query Transformation - Pivot, Unpivot, Transpose Data ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-U6fsHSiNpz4.html Microsoft Fabric Video 19: Load Azure SQL Data to Lakehouse & Warehouse simultaneously with Dataflow Gen2: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-40KK-dquvLE.html Blogs Microsoft Fabric: New Age Analytics medium.com/microsoft-power-bi/microsoft-fabric-new-age-analytics-ba90b894633f Microsoft Fabric- What, Why, Who, When, How medium.com/microsoft-power-bi/microsoft-fabric-what-why-who-when-how-638b391df4f1
