IDG Contributor Network: Real-time data pipelines: pairing message queues and databases


In a connected world, real-time data pipelines power applications and insights, providing the digital infrastructure for active data. This helps data-driven companies understand how their customer base is behaving in the moment, especially when it involves live operations. With an ongoing pursuit of new digital growth opportunities, data leaders need to stay on top of these leading architectures.

A stereotypical real-time data pipeline might look as follows:

Real-Time Data Source > Message Queue > Database > Application

Data sources and applications can be unique to specific industries. Choices for message queues and databases apply horizontally and are the primary focus of this article.

Real-time data sources

Modern applications drive real-time data in areas such as: