Data Mesh Logical Architecture

(This summary is prepared by S Kannan, Editorial Assistant at The Fifth Elephant. Read the summary before proceeding to watch the talk.)

The Data Mesh principles facilitate building data products with a product thinking approach and a rigour to data sets. With a growing number of data sources and application use cases in your organization, how do you better learn the shape and form of data for your business customers, and scale the access patterns? A domain team better understands the integrity rules, constraints, operational and analytical data to make meaningful insights. It is thus important to push the data ownership towards the domain experts. Data Mesh performs this heavy lifting to define the capabilities and necessary abstractions for product teams to define, declare and build their data products with the least amount of friction. It is thus essential to have a deeper understanding of the principles of Data Mesh and domain oriented decentralisation.

In this talk, Vanya (Head of Technology, Thoughtworks and Global Lead, Data Mesh Guild) and Sarang (Lead Consultant, Thoughtworks) demonstrate the logical architecture of Data Mesh using a ratings data product for a media streaming company in the podcast domain. All this is accomplished while still ensuring global interoperability standards, security, and meeting Service Level Objectives (SLO). The Data Mesh platform planes enable both producers and consumers of the data to interact with each other, and help data product developers to test and validate their hypothesis. The data product plane gives a complete bootstrap repository with a working CI/CD pipeline, and the data product experience plane accelerates the development and deployment cycles.

As a data analyst or data scientist, Data Mesh allows you to even programmatically explore the data sets and data products, and also review data lineage and data quality. The possibility for automating interfaces, defining quality metrics for the data, and proper auditing of logs exists. The platform helps seamlessly deploy, train the model with new data, version control and monitor the model on the experience plane. The data product owners, platform architects, compliance and risk personnel, and chief data officers establish the standards for the business domains. The logical Data Mesh architecture greatly simplifies building data products for both product and domain teams while still complying to regulations and standards.