Data Mesh - The Four Principles

(This summary is prepared by S Kannan, Editorial Assistant at The Fifth Elephant. Read the summary before proceeding to watch the talk.)

How can organisations modernise their software and infrastructure for AI and data operations? Can companies respond gracefully to the volatility and uncertainty in business while still being compliant with regulations? The impediments in today’s organizations in moving towards a data driven culture are people, process and culture. The Data Mesh platform from Thoughtworks provides a decentralised, socio-technical approach to unlock the power of data while operating and managing analytical data at scale. In this talk, Vanya (Head of Technology, Thoughtworks and Global Lead Guild, Data Mesh) and Manisha (Lead Consultant and Data Engineer, Thoughtworks) explain the four key principles of Data Mesh - decentralised domain ownership, data as a product, self-serve data infrastructure and federated computational governance.

The data ownership is created around domains in a decentralised manner which brings data closer to the teams who handle business use cases. The data product is treated as a first-class citizen and necessary abstractions are provided to make it easier to build them. The platform also provides the capabilities in a domain agnostic and self-serve manner that decreases the lead time for development and deployment. The metadata of data sets, data product attributes and Service Level Agreement (SLA) definitions can be defined in the data products to inherit the governance in the design itself. The four key principles facilitate each other in a way that hold the entire model together helping organizations to make wise decisions from data.

The Data Mesh platform provides a well defined ownership and is built on trust. It does the heavy lifting of the infrastructure for setting up data products in the form of three planes using the separation of concerns principle. It effectively closes the gap between the operational and analytical world. Thus, data engineers and AI scientists can train a model from aggregated data products and operate microservices seamlessly and thus create more value for the business. As organizations move away from monolithic, centralized bottlenecks, the Data Mesh platform accelerates the value in their return of investment while addressing the business needs.