Our marketing analyst, Selima Triki, unpacking the idea of real time and real-time data. In this article she covers  what streaming data is, how it works, how it differs from batch processing and some of its benefits and applications. 

The world generates an unfathomable volume of data, that keeps on multiplying every minute of every day. IT-powered innovations, the Internet of Things and new generation customer demands are fueling real-time data creation. In this new data landscape, businesses from all industries are quickly shifting from batch processing to real-time data streams to match the new imperatives. 

Real Time and Real-time Data ?

A practical example of a real-time system is the collaborative real-time editor enabling live and simultaneous editing of the same digital document, such as an online spreadsheet or a word processing document. With real-time editing, multiple users, on different computers or devices, can access the same document at the same time with almost instantaneous synchronisation of their edits.

Another example is social media reactions such as “likes or comments”. They are by definition real time. Why? Because they appear almost instantly.

Processed using real-time computing or processing, real-time data in the new generation data. 

Real-time data comes in the shape of streams and live feeds, but rather than consumer content, the transmission is data. It is delivered immediately after collection, with no delay in the timeliness of the information provided. Such data is not designed to be kept back or stored, but rather passed along to the end user as quickly as it is collected. 

Real-time data is enormously helpful for all sorts of analytics projects and for keeping track of your environment through the power of instant or near-instantaneous data delivery. Delivering practical and meaningful real-time insights, mapping customer behaviour and shopping patterns, detecting fraudulent activities in real-time or optimising advertising campaigns, the power of real-time data is limitless. 

An illustrative example would be ad placements online or automated ad placement through programmatic advertising. Based on optimisation algorithms that act on parameters such as segments, prices and purchasing power, digital ads are almost automatically placed in an optimal way, targeting the right person with the right message at the right space and time. 

Streaming Data

As mentioned before, real-time data comes in the form of streams. Let’s explore what that means in more detail. 

Let’s start with the term “streaming”. This term describes the unceasing data flows with no beginning or end, that brings continuous data feeds which can be used and utilised instantly. 

Streaming data, also referred to as event stream processing or real-time data streaming, describes the constant flow of data. Data streams are designed to be processed, analysed and leveraged in real time. 

How Data Streaming Works ?

Data comes in various forms and formats. It is generated by a high number of source systems – servers, applications or even IoT sensors. It is nearly impossible to control its structure, velocity or volume and needless to mention its integrity. While current systems are built to ingest and process data, streaming data architecture brings in the capability to store, enrich and act upon data in motion. 

Some examples of streaming data include use cases in almost every industry, encompassing social media, ride-sharing apps, fraud and banking or even inventory management. 

For example, when you call an Uber, real-time streams of data merge to build up a consistent user experience. Using that data, the application pieces together real-time traffic tracking, location coordinates and pricing information to simultaneously match the passenger with the best possible driver, find optimal pricing and appraise time-to-destination by crossing historical with real-time data. 

Thereby, streaming data becomes the cornerstone of any data-driven use case, setting the stage for big data applications and real-time analytics. 

Real-time Processing Vs. Batch Processing

One of the fundamental questions to ask oneself when we talk about data architecture is the question of batch or stream processing: Should you wait for data to pile up before we process it or should you process it as soon as it arrives, in real time? 

If you are still wondering about the difference between batch and real-time processing, the following might help: : 

  • Under batch processing, high volume data is collected over a window of time, then headed towards a storage layer where it awaits to be processed. 
  • Under the streaming model, data is streamed continuously and simultaneously, processing is done piece-by-piece in real time. 

While most businesses use batch processing, we can clearly see a tendency towards data streaming. Streams of data come in all shapes and sizes, at different times and velocities, from various sources and locations adding up more complexities to an already complex landscape. 

Data Streaming and Stream Processing

The evolution of the data landscape created a virtuous circle of data analytics leading to new insights and action. To thrive, businesses need to anticipate what’s next and react in real time. This has spurred all kinds of personalised services that combine digital information from consumer data to social network graphing. It has brought more efficiency to fields like supply chain and consumer marketing to name just a few. 

Let’s understand how this works through a simple business use case; customer intimacy. 

Customer intimacy is a measure of how customer-centric a business can be. It combines thorough customer knowledge with operational flexibility allowing a timely response to almost any need. 

A perfect scenery to project this case would be at an Amazon Go checkout-free supermarket offering a new kind of technology-powered shopping experience. 

As the customer walks through the store, passing by different shelves, his behaviour is observed and analysed in real time. From the moment he takes a bottle of shampoo in his hands, to the amount of  time he spends reading the ingredients label, a series of recommended services and products are waiting to pop up in the buyer’s smartphone. How is that possible you may wonder. Easy ! 

Data streaming combined with stream processing. Sounds complicated ? Let me rephrase. 

All the actions and purchase behaviours that compose the customer experience are captured in streaming as events or data streams (data streaming), which are in turn analysed and acted upon using stream processing.  

Stream processing brings the capability of instant computation of data streams to enable better functioning from more effective processes, enhanced customer experience to generating new revenue streams.  

How ? By making all the needed computations, combining historical data with real-time data and applying predictive analytics or machine learning to create new information which in this case are real-time recommendations or customer incentives. 

Practically, to be able to do the above, you need to follow this to-do list: 

  • Manage all data formats coming from different sources
  • Harmonise data structures, and apply needed transformations
  • Control data access to avoid the loss of information
  • Ensure freshness of data and models used
  • Ensure the authorization to access and process data
  • Track data operations lineage 
  • Give the ability to erase and delete data

Luckily, pre-packaged solutions that integrate all of these tasks are making it easier for businesses to start real-time/ stream processing and make it part of their data strategy.

Real Time Across Industries

Any business of any industry can highly benefit from real-time technologies and create synergies thanks to real-time capabilities. 

Think about the synergy of retail and banking for example as a next generation real-time-powered application. 

Now, on your way to work at the local coffee place, not only can you get a cup of coffee that you don’t have to pay for thanks to your loyalty points through the retailer’s mobile application but you can also pay for a salad using your digital wallet. Another example would be joint mobile payment schemes that allow seamless access to bank accounts with instant alerts triggered by a series of real-time events such as shopper behaviour or change in consumption patterns. Here, real time data is captured, analysed and acted upon to give recommendations or optimise the shopping experience. 

Value-added services for both retailer and bank can deliver endless benefits. The synergy of services ensures an integrated customer experience at every touch point and in real time. 

To win in the modern-day digital world, businesses have to deliver quality in real-time speed, exceptional customer service and data-driven, backend operations. Achieving this real-time performance requires the ability to react, respond and adapt to continuous, multi-faceted and ever-changing data from across the organisation. 

Today’s digital systems have given businesses a wealth of new tools to capture and measure the minutiae of user behaviours with connected devices, on-demand applications and other real-time platforms. With digital data being generated at staggering speeds, data engineers and analysts need the right architecture and systems to quickly operationalise data and process it for analytics and data science. A number of technologies are making real-time streaming possible, allowing businesses to act on up-to-the milliseconds data, before it becomes stale.