This video explains the concept of data mesh and its characteristics. What is a Data Lake: • What are Data Lakes | ... What is a Lakehouse: • What is a Data Lakehou...
very nicely explained, Thank you! I have one doubt though, since we are storing data domains separately managed by different data owners/teams, do we still have one pipeline? or every domain works on their own pipeline?
But once these data domains and it's corresponding ownership is created, how will it serve data scientists who need collective raw data to work with.. I see it's similar to have different schemas per domain in a data warehouse.. I kind of see a lot of overlap.. probably the only difference is the ETL can be handled by the domain team or can still be managed by a centralised team based on needs.. also how can it complement with some thing like a data lakehouse?
Nice introductory video about #DataMesh...You explained about how the data is managed here, but can you provide some idea on how the data will be served or retrieved from diff domains?
I suppose data marts are built on top of a CDS which is a common database, extracting only the required information specific to the domain. Wheares, here in mesh, there is no such CDS from which individual data gets pulled, rather the data is sourced directly to the domain (Only the required data specific to that domain)
hey chandra , we creat GUID to join different domain data .if a consumer would want data from different domain for specific then based on guid the data is extractes from different domains
this is DDD concept which says to have own copy of data in each domains suitable for this domain, since 2012 by Evans. It's not new concept "data mesh".
Can you pls make a video of implementation of data mesh, as per comments I see people still have a lot of doubts like how will data inter domain will be accessed, will there be duplication etc
Thanks Shreya for the knowledge shared. Truly helpful. Had one question though. Is that only with the help of global governance team, one domain data can be accessed/understood by a different domain in case required. Or they have to create/duplicate a separate metadata on their own domain despite the presence of the same in a different domain instead of reusing that
Thanks sriram. Data duplication may occur as there may be common datasets across domains. Metadata can be managed centrally. Here we are moving away from centralized storage to domain driven disparate data storages.
@@BigDataThoughts , Thanks for the quick response. If metadata is central, will that Entity hold all the associated attribute from each of the domains? Or only the common ones, leaving the domain specific attributes to be defined individually
Hi Shreya, what about having different schema and storing data as different domain(sql DB) compared to having data domains that you are talking about ? Any example pls to correlate and understand more ? (Thanks again for video on Monday, I thought no video for today. But you haven’t missed it. Thanks once again)
We can have data in different schema or different storage. But the essence is that we are storing data domains seperately managed by different data owners/teams rather than centrally with one ELT pipeline. The concept is to not have a central storage that may not be flexible in terms of adding new data source.
I am more confused - in the past, various "islands" of data was causing all kinds of headache for the business to consolidate reports....with the consolidated data domains, things were much easier to rely on as they were catagorized as "single version of the truth"... well....now comes Data Mesh which is encouraging (so to speak) to seggregate the single verson of the truth and distribute is across the enterprise?
