Data Mesh — Anomalies as Data Product

Johan Louwers
8 min readSep 5, 2022
Anomalies as Data Product

The concept of the data mesh revolves around the fact that data is perceived as a product and is shared in a standardised manner throughout an organisation in a mesh structure. When talking about “what” data could become a data product the initial thinking is that all business events should become data products.

While this being true, data describing business events are valid data products, other types of data can also provide meaningful and valuable insights to the wider organisation. A potential data product which is commonly not considered in first instance is the data describing anomalies in events.

Anomalies in business events

Whatever type of business your company is in business events are happening constantly. Being a logistical company will involve business events related to goods arriving and departing from warehouse locations and the movement of goods within the warehouse and on the road. Being a financial institute business events related to transfers between accounts and customer interactions are an ongoing daily stream of business events.

In normal day to day operations business events will all follow a relative standard and predictable pattern and all events can normally classify as valid business operations. However, in some cases an event might classify as a anomaly and can be an indicator for upcoming business disruption, human error or pure fraud.

Being able to automatically detect anomalies in your daily business operations, being informed about them and being able to act upon them lowers the overall risk profile and protects the daily operation and the business as a whole against undesirable situations. Being able to detect, prevent and mitigate against undesirable situations protects companies against fraud, human errors as well as loss of money and reputation.

Events clustering and anomaly detection

Regardless of the type of business event, regardless of the specific data describing the business event, events can be clustered using cluster analysis. Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense) to each other than to those…

--

--

Johan Louwers

Johan Louwers is a technology enthousiasts with a long background in supporting enterprises and startups alike as CTO, Chief Enterprise Architect and developer.