the amount of work the developing community has put into it. Think of it as beta vs stable. Databricks is way more stable, has been developed through iterations to catch bugs and implement fixes already. Synapse Analytics is newer comparatively and is going through that iterative process now, so in time its reliability will catch up to that of Databricks.
I would say maturity is the level of knowledge and skills an organization has to support these tools. You are not going to give a graphing calculator to a 6 yr old child. You are not going to give databricks to a company that has everything in spreadsheets.
Azure Data Factory is similar to SSIS and doesn't have a data store to persist the data, but Azure Databricks and Azure Synapse has a database engine to support the storage of data. Azure Data Factory is only an ETL/ELT tool. But for the other two there are ETL/ELT and database. In case this, Azure Data Factory shouldn't be compared to a database.
Its a seperate tool that's true and as many people use ETL with Data Factory they do have doubts about Should I use Azure Synapse / Azure Databricks for my ETL or I should continue using Azure Data Factory. Noting don't know code can leverage UI with little extra cost and who knows code can save little too.
Migrating to Databricks can offer you a bit more flexibility, but you would have to migrate all the pipelines to code. Alternatively, you could use both tools, and make your new flows in Databricks. Notebooks and packaged code in databricks can easily be kicked off by ADF, making it a cool orchistrator!
If SQL Server is the target, you can indeed just connect Power BI to SQL Server and do your aggregations/data loading with ADF, no problem! My point was more regarding to connecting ADF to Power BI. In synapse and Databricks you can create tables and use these definitions directly in Power BI by connecting these tools. ADF has no such thing.
Here's a few off the top of my head: 1. Databricks clusters are more flexible. You can choose the cheaper Compute Optimized VMs for append-only incremental processing, or Storage Optimized VMs to enable caching on the local SSDs, among other VM types. In Synapse, you can only use Memory Optimized and GPU Optimized VMs; 2. Databricks clusters allow you to use Spot VMs for the workers, which are significantly cheaper as well. Synapse does not support Spot VMs; 3. Databricks allows for better cluster sharing, as the same cluster can have multiple Spark sessions active at once. Synapse reserves slots for each Spark session, and those slots will sit idle when the developer is not running any code -- they can't be used by other developers while they are reserved; 4. The notebook file format in Databricks lends itself better to git diffs in Pull Requests, as they are regular code files (e.g., Python code) with some comments for special cells. Synapse notebooks, on the other hand, are saved as JSON files which are much harder to review in a git diff interface; 5. Databricks has exclusive features such as Auto Loader and identity columns, which are really helpful for data engineering and framework development; 6. Databricks is the flagship product of the company founded by the creators of Apache Spark, and as such it will always have an edge in supporting new Spark versions and features. Meanwhile, Synapse is a PaaS offering from Microsoft, and Microsoft is now clearly focusing a lot more in their SaaS offering: Microsoft Fabric. If I had to build a data platform on Azure today, I would use Databricks as my transformation engine. Hope this helps! :)
lake house is just the architectural approach. as of my knowledge, every analytical, cloud based solution is built on top of some kind of cloud data storage (adls, blob storage, aws s3 etc.) and this is only a data storage layer
as per my understanding, databricks and synapse store data in azure blob storage, and give you a database/ datawarehouse like model on top of that, so that you can do easier analytics or other stuffs. Even some projects creates data integration and pipeline in ADF to trigger databricks jobs/notebooks and synapse can do analytics and use BI tools over delta lake in databricks.
