The Evolution of Data Architectures

There are different possibilities for organising data depending on your needs. While some are starting with a simple data warehouse, others are reaching the capacity of utilising a data mesh. We will define some concepts that will help understand the different organisation options and how it can benefit your business.

Data warehouse is a first step in storing data and it is known to be a good option for reporting, sustaining decision-making through quantitative and qualitative facts. In essence, a data warehouse is a consolidated and structured repository for your data assets, through ETL (extract, transform, load) data modelling principles. As a result, this method can transform the data based on predefined reporting schemas, supporting the business in various way:

• centralise data collection
• format to simplify data usage
• assist the definition of data subsets for specialised reporting
• bolster data governance
• facilitate data quality assessment

However, data warehouses are limited in storing non-SQL databases and processing algorithms that require direct data access. In addition, in all ETL structures, data needs to be transported to feed the warehouse, making it impractical and compromising its overall efficiency.

Data lakes can to some extent overcome such limitations as your capabilities will be extended to much more than just reporting, by integrating raw data usage (data from images, internet, sensors, live events, etc.) The ability of storing a large amount of raw data in one single repository will deviate from the lengthy usage of multiple ETL systems. Fundamentally, using a data lake will:

• generate added value by helping your team leverage data from multiple sources, regardless of the data type (unstructured, structured or semi-structured)
• define data subsets for a specific use case through quicker extraction of data sets
• remove the transformation process from the ETL process (data is now extracted from operational systems and directly loaded into the lake)

When dealing with a great amount of data, it is likely that you will not need all the data stored. With improper management, a data lake can easily turn into a data swamp filled with irrelevant data while relevant data can be very poor in quality. Formatted only for operational needs and unsuitable for any sort of analysis, a data lake will not be able to keep up with processing capabilities for modelling execution.

One way to overcome this challenge is by implementing a data hub. As a strategic intermediary between producers and consumers of data, hubs allow real time data management (streaming and feeding data to applications in real time). Resulting in a boost for any data lake and rapid integration with pre-existing data warehouses through its streaming capabilities. With a data hub you manage to comprise all data warehouse and data lake advantages in one go.

Taking data architecture a step further by distributing data management capabilities to different data domains, can be achieved by adopting a micro-service architecture: data mesh. The most appealing part about such an architecture is its self-service approach. Consumers of data like data scientists and BI engineers can focus on their added-value tasks and keep the momentum of their projects. By effectively decentralising the tasks while enforcing enterprise-wise standards (through federated governance), data mesh removes all the common bottlenecks of more traditional approaches to data.

 

Investing into a modern data management platform can not only provide the necessary tools to govern your data but also scale with it side by side without compromising resources or expertise. When moving from one type of data storage to another, security and corruption can pose a threat, resulting in compatibility or scalability issues. Opting for a platform that supports SQL queries can make historical data more accessible than ever. Consequently, implementing such a platform will ensure a smoother integration of your data within the desired infrastructure.