Data Mesh — Should you add a webhook?
--
The concept of the data mesh revolves around the fact that data is perceived as a product and is shared in a standardised manner throughout an organisation in a mesh structure. Data should be accessible and consumable in a standardised manner which complies with enterprise (or industry) wide standards for both the technical implementation as well as the definition of the data itself.
Data mesh is a sociotechnical approach to build a decentralized data architecture by leveraging a domain-oriented, self-serve design (in a software development perspective), and borrows Eric Evans’ theory of domain-driven design and Manuel Pais’ and Matthew Skelton’s theory of team topologies. The main proposition is scaling analytical data by domain-oriented decentralization. With data mesh, the responsibility for analytical data is shifted from the central data team to the domain teams, supported by a data platform team that provides a domain-agnostic data platform.
Exposing data products
One of the core principles is that data is exposed / provided to the wider data mesh for consumption by other domains. The common implementation used is to ensure relevant data (commonly business event oriented) is being transferred from the application sub-domain into a data node sub-domain.
The data node sub-domain takes the responsibility for exposing this data as a data product to the wider data mesh. The data exposed by the data node can be the as-is data from the application sub-domain in a rationalised and standardised manner which adheres to a enterprise wide standard, it can also be a new or enriched data product based upon the data from the application.
An example of a new or enriched data product could be for example detected anomalies in original data or forecasting data which can help to improve business processes and support better data driven decision making.
A common misunderstanding is that a data product is a REST API, reality dictates that a data product is not a predefined and single technical solution. A data product is an agreement based data definition which…
