The real-time data
Best practices in a box
Real-time data allows you to make better business decisions but also to react when the value of your reaction is the highest: closing an on-line sale, detecting a fraud attempt or addressing a risk in an automated production chain. Data science and predictive analytics are pushing the boundaries of what is possible with real-time data.
Even for more traditional scenarios that would not necessarily need real-time, using a streaming approach gives you the certitude that all your data gets continuously updated so there is no need to manage timing of updates or synchronisation. You can just consume the data in the stream with the guarantee that it is up-to-date, which greatly simplifies the process.
Digazu is effectively an out-of-the-box implementation of a state-of-the-art data hub architecture. It fully decouples data extraction from data consumption, and can provide data scientists with self-service data
The latest thought leadership on architectures is to go even further by putting the requirement on data providers to package their data from the start as data products. This data mesh architecture requires a significantly higher organizational maturity and Digazu, coupled with a data catalogue, can fast-track this transformation.
of Kafka and Flink
These underlying technology choices are at the heart of Digazu’s non-functional excellence: real-time, low latency data processing at scale.
Kafka is managing the data collection and distribution aspects (stream processing) while Flink focuses on the manipulation of data in streams (data transformation). Flink is particularly efficient at managing stateful computations which are a must when you want to combine several high-volume sources in real-time and make aggregations typically used for real-time dashboards for IoT.
Both frameworks have extensive communities and ecosystems and are there to stay in the foreseeable future.
As the data architectures become more distributed, the need to understand data lineage and associated permissions becomes increasingly important. Digazu supports the definition of domains in which the data can be managed end-to-end.
In certain jurisdictions, personal data privacy is a strong legal constraint. In Europe in particular, the GDPR regulation imposes strict rules on consent and the enforcement of these rules in a decentralised data infrastructure can be very complicated. Digazu, created and validated by European customers includes native support for GDPR features.