Enterprise architectures are transitioning to event-centric to serve the real-time needs of customers
Michael Stonebraker of MIT and Ugur Cetintemel of Brown University published in 2005 an important paper about databases titled ““One Size Fits All”: An Idea Whose Time has Come and Gone.” For the previous 25 years, the relational database management system (RDBMS) dominated the enterprise database market. The prevailing thinking was that all enterprise applications could be served by a relational database. Many companies spent years attempting to normalize their data and build integrated applications that both used such data and integrated to each other through such data.
The Stonebraker paper argued that this one-size-fit-all database approach was limited, had run its course, and was soon to be replaced by a proliferation of built-for-purpose databases.
Another implication of the paper and subsequent proliferation of databases was a movement away from (either explicitly or implicitly) the RDBMS as the center of the application universe. This architectural paradigm was perhaps most notably represented by enterprise resource planning (ERP) systems. The original ERP architectural approach was to house and control all data and continually add functional modules that used such data and integrated to other modules through such data. In practice, this never materialized, with data spread across many different databases and non-databases (flat files, Excel) throughout the enterprise.
As we know by now, the Stonebraker paper turned out to be right. The past 15 years has seen a proliferation of databases optimized for specific use cases. Relational databases continue to handle many workloads, but they are a shrinking percentage of overall enterprise workloads. In addition to relational databases, we now have NoSQL, cloud, columnar, wide column, object-oriented, key-value, in-memory, hierarchical (which pre-dated relational), document, graph, and time-series databases all with offerings from multiple vendors. Add to this analytic and stream processing engines such as Hadoop, Spark, Storm, Flink, and their various commercial derivatives.
It's no small coincidence that this movement has parallel movements in consumer product and delivery choice and a movement away from one-size-fits-all supply chains towards segmented supply chains. In fact, this movement away from one-size-fits-all databases has enabled the movement away from one-size-fits-all offers to consumers and one-size-fits-all supply chains that serve consumers.
Architectural Implications
One of the architectural implications of this movement is that the database is no longer seen as the center of the application universe (like planets rotating around the sun). Databases are simply consumers and sources of data to be used by, and shared with, applications, users, and other databases. This has important implications for enterprise data architectures. One central implication is that enterprise architectures are transitioning from database- and data query-centric to event-centric. What does this mean?
Every data update in an enterprise represents an event that someone or something may be interested in. Let’s consider the database-centric approach with two applications: Application A and Application B. In a database-centric approach, Application A puts data into database A. Application B can access this information by either 1) directly retrieving it from the database A, or 2) indirectly by retrieving it from database B after it has been batch transferred from database A to database B. In enterprises, millions of such transactions occur daily.
In an event-centric approach, Application A updates its data; this becomes an event. It publishes the event on an event bus. Other applications and databases subscribe to this event. For example, database A, application B, and database B may all be subscribers. As subscribers, they all get updated with the same data. This is known as a publish/subscribe (pub/sub) architecture.
Another implication is that data pipelines and data engineering have become central to modern enterprise applications; in fact, these pipelines are becoming the center of architectures, with data sources (including databases) becoming sources that simply “hang off” the pipelines.
Factories Were First
Most factories have something like an “information bus.” In the 1980s a lot of time was spent on standardizing on factory communication protocols. The manufacturing automation protocol (MAP) was the most visible such effort. It failed because it was a committee and standards-driven effort; meanwhile every device manufacturer was developing to the TCP/IP protocol, which was supposedly inferior to that which the standards committees developed. The point here is that factories are action and event centric. Things happen fast in factories; this has always been the case. In factories, as Patton said, “you either lead, follow, or get the hell out of the way.” (Note: this quote did not originate with Patton, but it created an enduring image when he said it).
Enterprise information architectures can learn a lot from factory information architectures. Enterprise information architectures have historically been batch- and database-centric. This is because 1) the clock cycle of the enterprise allowed for it; 2) event-centric architectures didn’t scale to the enterprise; and 3) thirty years of inertia have perpetuated the database-centric approach.
The clock cycle of factories did not allow for this; thus, a different, event-centric information architecture evolved in the factory. Now, in the past five or more years, the clock cycle of the enterprise has converged towards the clock cycle of the factory. There is less and less time for batch and more and more need for processing of events in real-time (see “What is Real-Time?” for a definition of real-time).
Financial Industry
The financial industry, with its need for processing large volumes of real-time data, has been a leader in transitioning its information architecture from database-centric to event-centric. Here, as in many cases, necessity has been the mother of invention. All industries now have shrinking time windows for just about every business process; thus, there is an increasing need to transition to architectures capable of processing real-time events.
As industries transition more towards event-centric architectures, there will likely be a hybrid architecture for some time in the future. Architectural patterns take time to fully develop; costs and inertia are significant impediments to rapid change.
Supply Chain Visibility Leads the Way in Goods-Producing Industries
The rise of supply chain visibility platforms is providing event-centric architectural leadership in many goods-producing industries. The proliferation of IOT devices with GPS capability has enabled real-time tracking of shipments. SCV platforms provide the ability to process these event signals and collate them with other API-based transactions to provide visibility of end-to-end product movements in the supply chain. This is a case where supply chain problem sets are leading the way in driving IT architectural change. This has not always been the case in the past, with supply chains being later in the adoption curve of information technologies.
At the same time, companies are adopting digital twins (active, time-phased models of their supply chains) to drive myriad business processes, including managing demand and supply, rapidly responding to variability and volatility, and providing tailored customer service. The lifeblood of the digital twin is data; there are many data tributaries that lead to the digital twin. In the past, the data source for these data tributaries were databases; event data would come in from the physical world, be deposited into a database, and then get updated into the digital twin, typically through a batch process.
Today, the digital twin data update process is increasingly direct, event-driven, and real-time. The source of this data is increasingly a supply chain visibility platform.
The adoption of SCV platforms is ushering in the new event-based architecture in many goods-producing industries. CIOs initiating SCV projects can link these projects to broader goals for transitioning their information architectures to event based. These projects can serve dual goals for both the business and IT.
Summary
Enterprise architectures are transitioning from database-centric to event-centric. The one-size-fits-all database approach that started in the 1970s and 1980s has given way to use-case-driven databases, and in turn, event-centric architectures. This data architecture mirrors similar architectures in physical supply chains, as companies have moved away from one-size-fits-all supply chains towards tailored supply chains with specific characteristics and personas.
The database-centric era is most notably characterized by ERP systems, which sought to house all data in a relational database and then serve it up to multiple applications for direct use and for cross-application integration. Future architectures will increasingly serve applications and databases through an event bus. Databases will increasingly be sources and consumers of events that travel across the bus.
In the supply chain application world, supply chain visibility platforms are leading the way towards this architectural pattern. These platforms are providing visibility capabilities directly, along with serving up visibility information to consuming applications such as S&OP and control towers. CIOs seeking to transition their architectures towards event-centric and real-time can leverage their SCV projects towards dual goals of providing business value and at the same time promulgating their architecture goals.