What we see in this example is that the execution sequence of transactions defines the outcome, and this is driven by the fact that each individual transaction is dependent on the result of a previous transaction. The price charged to the customer when the order is made should be related to the price reported to the customer after the initial price request, if that is what the company has stated as its definitive policy.

Because any master object can be either read or written, we want to ensure that the data dependencies are observed within any transaction. The combinations of read/write dependencies by different applications can be characterized as follows:

This last point is the critical one: maintaining consistency and synchronicity is related to ensuring that all participating applications see a consistent ordering of data dependent actions.

Different aspects of coordinating the management of product WID-3IN's virtual master record show how reliant our applications are on particular systemic support for data synchronization. For example, the result of the customer's query could have been different if supplier B's product update had arrived earlier in the sequence, as would be the ultimate cost per widget, which highlights how the timeliness of master data transactions impacts the business process. In fact, the intermediate update to the product's master record in the middle of the customer interaction demonstrates the importance of having the most current data available.

The amount of time that it takes for the specific actions to be communicated may also impact the final outcome, because if supplier B's update takes a long time to be made generally available (in other words, has a long latency), the customer will be charged the higher amount. At the same time, the view to the customer displayed one lowest price, whereas the updated master data set contained a different lowest price—a lack of consistency and coherence that could impact the end result. Lastly, had the customer requested the price twice, it is possible that the result would have been different each time, exhibiting nondeterministic behavior with respect to data access. In general, within the DBMS, all of the peculiarities of concurrent interactions between these different aspects are handled as a by-product of transaction serialization.

The fact is that almost any high-level transaction in any operational system is composed of multiple subtransactions, each of which is executed within a defined sequence. At the same time, other applications may be simultaneously executing transactions touching the same data sets. So how are these transactions (and their corresponding subtransactions) executed in a way that ensures that they don't step all over each other? Although the complexity of transaction serialization, multiphased commits, and locking are beyond the scope of this book, a simple overview can be somewhat illuminating.

The basic principle is that the underlying transaction management system must ensure a number of assertions that will guarantee a consistent view:

The impact of the first assertion is the guarantee of a total ordering of the transactions performed by each distinct application. The second assertion enforces a consistent ordering of interleaved transactions from the different applications to enforce the view of data dependencies as described in Section 11.3.1. Simply put, database management systems employ processes to enforce these assertions by intermittently “locking” the underlying data asset being modified during any specific set of subtransactions until the system is able to guarantee the consistency of the entire transaction. Locking a data object means that no other transaction can modify that object, and any reads are likely to see the value before the start of the transaction sequence. When that guarantee is assured, the database system commits all the modifications and then releases the locks on each individual data object.

Because these locking and multiphase commit protocols are engineered into the database management system, their effect is completely transparent to any application developer or any application user. But because there are many different approaches to implementing master data management, the importance of incorporating these same protocols at the service layer is related to the degree to which the business requirements specify the need for the master data to be synchronized.

Clearly, many MDM environments are not specifically designated as tightly coupled transaction processing systems, which means that the requirements are going to be determined in the context of the degree to which there is simultaneous master data access, as well as how that is impacted by our dimensions of synchronization. Therefore, the process for determining the requirements for synchronization begins with asking a number of questions:

This is just a beginning; the answers to these questions will trigger additional exploration into the nature of coordinating access to master data. The degree of synchronization relates to the ways that applications are using master data. When there is a high overlap in use of the same assets, there will be a greater need for synchrony, and a low degree of overlap reduces the need for synchrony.

It is valuable to note that one of the biggest issues of synchrony in an MDM environment leveraging a services-oriented architecture (SOA) is the degree to which end point systems can consume updates. In this asynchronous paradigm, if some end point systems cannot consume updates, for example, or there's a huge backlog of updates, you may have both performance and consistency issues.

In the registry architectural style, the only shared components are the identifying attributes, whereas the existing data sets contain the data that compose the conceptual master asset. Although individual applications still control these component data sources, master data services can only impose synchronization control on the shared registry, in which case the identifying attributes become the focus of the synchronization requirements. The characterization of the levels of each synchronization dimension for the registry model is shown in Figure 11.1. Because data are available as master data as soon as they have been persisted to an application's data store, timeliness and currency of master data are both high. However, the existence of variant copies of records representing the same master entity means that consistency will be low, and, by definition, the absence of strict coordination between these copies means that coherence will be low. Lastly, because the consolidated view is materialized on demand (through federation), intervening changes between any two access requests may lead to differences in the result from two equivalent queries, which means that the level of determinism is low.

In the repository style, more master attributes are managed within a single conceptual data asset. If the data asset is truly a single data system comprising the entire set of data attributes, the focus of synchronization is the master data service layer's ability to flow data access transactions directly through to the underlying database platform to rely on its means for transaction serialization and so forth.

At the end of the spectrum representing the full-scale single repository, any updates to the master are immediately available to any of the client applications. Correspondingly, the levels of timeliness, currency, and consistency are high, as seen in Figure 11.2. In the case where there is only one underlying data system, there will be no issues with coherence between copies, and therefore the level of coherence is also high. Lastly, all changes will have been serialized according to the underlying data system's protocols, and subsequent queries are extremely likely to return largely similar, if not exactly the same, results; therefore, this approach will have a high degree of determinism.

Both ends of the spectrum seem to be covered; it is with the spectrum in between where we start to see issues, because the hybrid model (where some attributes are managed within a master repository and others are managed by the application) and the federated model (in which the conceptually monolithic data master is implemented via data replication and data distribution) are where we see multiple copies of what we believe to be unique instances of data.

In the hybrid approach, the applications are still driving the sequences of transactions, and with data writes needing to traverse multiple underlying database platforms, there is a new level of complexity for the managing record locking and multiple phases for committing transactions. Because the transactions may be broken up into chunks applied to different underlying databases, the transaction semantics may need to be engineered into the master data service layer.

A different issue arises in aspects of a replicated model (shown in Figure 11.3), in which copies of the master data asset are made available to different applications, allowing each to operate on its copy as if it were completely under application control. Modifications to a copy of a master object must be propagated back to any other replicated copies that exist across the enterprise.

The different aspects of the underlying implementations mean that without details of how the master data asset is managed, it is difficult to project the determination of levels of our synchronization dimensions. For example, in the replicated model, maintaining coherence across the replicas becomes the critical activity, with all the other characteristics directly dependent on the degree of coherence. On the other hand, if there is not an expectation of a high degree of overlapping master data object accesses, engineering delayed consistency into the framework is an acceptable solution.

Abstractly, the integrated view of master data begins to mimic a memory hierarchy not unlike the physical cache and memory models employed across most computer hardware systems. The master data repository itself represents the persistent storage, but the data from that asset are accessed and copied into versions both closer to and more easily by the using applications. For example, a replicated model essentially carries three levels of a memory hierarchy: the actual master version, a replica in a local data management asset, and individual records being manipulated directly by the application before commitment back through the local copy into the ultimate master copy.

Luckily, many of the issues regarding consistency and coherence across a cache memory environment have been addressed successfully to ensure that hardware caching and shared memory management will work at the hardware level. Because these are essentially solved problems, we can see the same approaches applied at the conceptual master data service level. In other words, we can apply a cache model to MDM in order to provide a framework for managing coherence across multiple copies. In this approach, there is an assumption that the applications store data to a local copy of the master data (whether that is a full replica or just a selection of attributes), which corresponds to the persistent version managed in the master repository (Figure 11.4).

Whenever a copy of a master record is updated in one of the application replicas, some protocol must be followed to ensure that the update is made consistent with the persistent master as well as the other application replicas. Examples of protocols include the ones shown in the sidebar.

The processes will be handled ultimately within the master data service layer at the component services level. The determination of when local copies are updated becomes a policy management issue, especially with respect to performance. For example, immediately refreshing copies marked as stale may lead to an increase in network interactions, with data moving from a single resource (the persistent master) to the applications' copies, even if the data are not needed at that point, creating a performance bottleneck and increase in bandwidth requirements. On the other hand, delaying updates until the data are read may cause performance issues when generating reports against the local copies if many records are invalid and need to be accessed before the report can be run. The implication is that managing synchronization and coherence within the replicated/distributed/federated approaches to deploying a master repository impacts the performance aspects of the client applications, and, yet again, we see that the decisions regarding implementation of the underlying master data architecture depend on business application requirements for consistency and synchronization.

In Chapter 10 we looked at the challenges of master data consolidation—the process of creating the initial master data repository. Because there is no existing master data asset at the beginning of the program, the initial consolidation does not incur any additional demands for data synchronization. However, as soon as the initial version is moved into production, any subsequent data source incorporated into the master data asset is likely to not be aligned with that master data set.

Incorporating new data sources without intending to “sunset” them will always create the situation of one data silo lagging behind the other in terms of maintaining consistency. Therefore, think twice before considering absorbing data sources that will probably be retained as data stores for applications (such as specialized business tools) that will not be migrating to integrating with the master environment. That being said, there will always be situations in which there are data sets that will be absorbed into the master repository even though they will not be eliminated. Some of the issues that need to be addressed basically focus on coordination between the two data sets (and their client applications).

When there are applications that still use the data source but not the master data asset, that data source must remain in production. In this case, when is the data set synchronized with the master? The best scenario would impose the same level of coherence on that data as the replicated/distributed/federated approach, yet this is unlikely for a few reasons. First, the level of effort necessary for full coherence is probably close to the level of effort necessary for full integration, but maintaining segregation implies constraint on some assets (resource, time, staff), preventing the full integration. Second, isolation of the data and its services may create a barrier that limits the ability to invoke master data services from within the client applications. The alternate suggestion is periodic, one-way synchronization. Similar to the consolidation process, we can apply the integration process in a batch mode applied via the consolidation services within the master data service layer.

The other issue, which may be of lesser concern, is synchronizing the master data back with the segregated data source. The decision to do so should be left to the discretion of the manager/owner of that data source's client applications.

When application managers prepare to adopt the use of the master repository, their production applications may need to run in a dual mode, with modifications being made to the existing source and the master data asset. This introduces yet another synchronization challenge—maintaining consistency between the data set to be retired and the master data asset. However, as we will explore in Chapter 12, the dual operation model should be handled transparently as part of the master data services layer.