One of the motivating factors for instituting an MDM program is the desire for consistency and accuracy of enterprise data. In fact, many MDM activities have evolved out of data cleansing processes needed for data warehousing or data migrations. The ability to use data parsing, standardization, and matching enabled the development of an “entity index” for people or products, which then was used for ongoing identity resolution and the elimination of duplicate entries. The realization that the entity index itself represented a valuable information resource was a precursor to the development of the master data environment altogether and the corresponding services supporting that master data environment.

When evaluating both the primary business objectives and the operational requirements for evolving toward an environment that relies on master data, there is going to be an underlying expectation that the instantiation of an MDM program necessarily implies improved data quality across the board. The reality is not so straightforward. By necessity, there will be some fundamental questions about the following:

At a conceptual layer, these questions center on the trustworthiness of the data as it is extracted from the numerous sources and consolidated into the master repository. However, at the operational level, these questions begin to delve much deeper into the core definitions, perceptions, formats, and representations of the data elements that comprise the model for each master data object. The challenges of monitoring and ensuring data quality within the MDM environment become associated with identifying critical data elements, determining which data elements constitute master data, locating and isolating master data objects that exist within the enterprise, and reviewing and resolving the variances between the different representations in order to consolidate instances into a single view. Even after the initial integration of data into a master data environment, there will still be a need to instantiate data inspection, monitoring, and controls to identify any potential data quality issues and prevent any material business impacts from occurring.

Data assessment, parsing, standardization, identity resolution, enterprise integration—all of these aspects of consolidation rely on data quality tools and techniques successfully employed over time for both operational and analytical (read: data warehousing) purposes. In this chapter, we will look at how the distribution of information leads to inconsistency, and then we explore the data quality techniques needed for MDM and how data quality tools meet those needs.

Because of the ways that diffused application architectures have evolved across different project teams and lines of business, it is likely that although only a relatively small number of core master objects (or more likely, object types) are used, there are going to be many different ways that these objects are named, modeled, represented, and stored. For example, any application that must manage contact information for individual customers will rely on a data model that maintains the customer's name, but one application may maintain the individual's full name, whereas others might break up the name into its first, middle, and last parts. Conceptually, these persistent models are storing the same content, but the slightest variance in representation prevents most computer systems from recognizing the similarity between record instances.

Even when different models are generally similar in structure, there might still be naming differences that would confuse most applications. For example, the unique identifier assigned to a customer is defined as a 10-character numeric string, padded with zeros out to the left, and that format is used consistently across different applications. Yet even when the format and rules are identical, different names, such as CUST_NUM versus CUSTOMER_NUMBER versus CUST_ID, may still confuse the ability to use the identifier as a foreign key among the different data sets. Alternatively, the same data element concept may be represented with slight variations, manifested in data types and lengths. What may be represented as a numeric field in one table may be alphanumeric in another, or similarly named attributes may be of slightly different lengths.

Ultimately, to consolidate data with a high degree of confidence in its quality, the processes must have an aligned view of what data objects are to be consolidated and what those objects look like in their individual instantiations. Therefore, an important process supporting the data quality objectives of an MDM program involves collecting the information to populate a metadata inventory. Not only is this metadata inventory a resource to be used for identification of key data entities and critical data elements, it also is used to standardize the definitions of data elements and connect those definitions to authoritative sources, harmonizing the variances between data element representations as well as identifying master data objects and sources. Collecting and managing the various kinds of master metadata is discussed in Chapter 6.

We must have some yardstick for measuring the quality of master data. Similar to the way that data quality expectations for operational or analytical data silos are specified, master data quality expectations are organized within defined data quality dimensions to simplify their specification and measurement/validation. This provides an underlying structure to support the expression of data quality expectations that can be reflected as rules employed within a system for validation and monitoring. By using data quality tools, data stewards can define minimum thresholds for meeting business expectations and use those thresholds to monitor data validity with respect to those expectations, which then feeds into the analysis and ultimate elimination of root causes of data issues whenever feasible.

Data quality dimensions are aligned with the business processes to be measured, such as measuring the quality of data associated with data element values or presentation of master data objects. The dimensions associated with data values and data presentation lend themselves well to system automation, making them suitable for employing data rules within the data quality tools used for data validation. These dimensions include (but are not limited to) the following:

Data quality and data integration tools have evolved from simple standardization and pattern matching into suites of tools for complex automation of data analysis, standardization, matching, and aggregation. For example, data profiling has matured from a simplistic distribution analysis into a suite of complex automated analysis techniques that can be used to identify, isolate, monitor, audit, and help address anomalies that degrade the value of an enterprise information asset. Early uses of data profiling for anomaly analysis have been superseded by more complex uses that are integrated into proactive information quality processes. When coupled with other data quality technologies, these processes provide a wide range of functional capabilities. In fact, there is a growing trend to employ data profiling for identification of master data objects in their various instantiations across the enterprise.

A core expectation of the MDM program is the ability to consolidate multiple data sets representing a master data object (such as “customer”) and to resolve variant representations into a conceptual “best representation” whose presentation is promoted as representing a master version for all participating applications. This capability relies on consulting metadata and data standards that have been discovered through the data profiling and discovery process to parse, standardize, match, and resolve the surviving data values from identified replicated records. More relevant is that the tools and techniques used to identify duplicate data and to identify data anomalies are exactly the same ones used to facilitate an effective strategy for resolving those anomalies within an MDM framework. The fact that these capabilities are available from traditional data cleansing vendors is indicated by the numerous consolidations, acquisitions, and partnerships between data integration vendors and data quality tools vendors, but this underscores the conventional wisdom that data quality tools are required for a successful MDM implementation.

Most important is the ability to transparently aggregate data in preparation for presenting a uniquely identifiable representation via a central authority and to provide access for applications to interact with the central authority. In the absence of a standardized integration strategy (and its accompanying tools), the attempt to transition to an MDM environment would be stymied by the need to modernize all existing production applications. Data integration products have evolved to the point where they can adapt to practically any data representation framework and can provide the means for transforming existing data into a form that can be materialized, presented, and manipulated via a master data system.

Data profiling originated as a set of algorithms for statistical analysis and assessment of the quality of data values within a data set, as well as for exploring relationships that exist between value collections within and across data sets. For each column in a table, a data profiling tool provides a frequency distribution of the different values, offering insight into the type and use of each column. Cross-column analysis can expose embedded value dependencies, whereas intertable analysis explores overlapping value sets that may represent foreign key relationships between entities. It is in this way that profiling can be used for anomaly analysis and assessment. However, the challenges of master data integration have presented new possibilities for the use of data profiling, not just for analyzing the quality of source data but especially with respect to the discovery, assessment, and registration of enterprise metadata as a prelude to determining the best sources for master objects, as well as managing the transition to MDM and its necessary data migration.

The original driver for data quality tools was correcting what was perceived to be “bad” data associated with database marketing, and so the early data quality tools focused on customer name and address cleansing. This typically consists of the following:

The cleansing process for a data value (usually a character string) typically follows this sequence:

Once a value has been transformed into a standard representation, existing master reference lists can be searched for the standardized entity name. If a match is found, the candidate record is compared with the existing master entry. Any discrepancies can also be called out into the stewardship process for resolution, with resulting updates communicated into either the master index or the record being evaluated.

Over time, cleansing has become more sophisticated; now we rely on the master repository for cleanliness, but the methods necessary to integrate stewardship roles in making corrections as well as learning from made decisions need to be introduced into the automation processes. For example, early cleansing processes were performed in batch, with out files provided to analysts for “postmortem” review and decision making. Modifications to actual records were performed manually, with all the associated challenges of synchronization and propagation of changes to dependent data sets downstream.

The automation process for MDM at the service layer must now be able to embed the functionality supporting the data stewardship part of the business process. Instead of batch processing for cleansing, services can now inline the identity resolution as part of data acquisition. If the identity can be resolved directly, no interaction with the business client is needed. However, if there are potential match discrepancies, or if no exact matches are found, the application itself can employ the underlying MDM service to prompt the business user for more information to help in the resolution process.

Enabling real-time decisions to be made helps in eliminating the introduction of duplicate or erroneous data at the earliest point of the work stream. At an even higher level of sophistication, there are techniques for learning from the decisions made by users to augment the rule sets for matching, thereby improving the precision of future matching and resolution. The use of parsing, standardization, and matching for master data consolidation is discussed in greater detail in Chapter 10.

Business processes are implemented within application services and components, which in turn are broken down into individual processing stages, with communication performed via data exchanges. Within the MDM environment, the business processing stages expect that the data being exchanged are of high quality, and the assumption of data appropriateness is carried over to application development as well.

However, no system is immune to the potential for introduction of flawed data into the system, especially when the acquired data are being repurposed across the enterprise. Errors characterized as violations of expectations for completeness, accuracy, timeliness, consistency, and other dimensions of data quality often impede the ability of an automated task to effectively complete its specific role in the business process. Data quality control initiatives are intended to assess the potential for the introduction of data flaws, determine the root causes, and eliminate the source of the introduction of flawed data if possible.

If it is not possible to eliminate the root cause, it may be necessary to use a data source that is known to have flaws. However, being aware of this possibility, notifying the downstream clients, and enabling the staff to mitigate any impacts associated with known flawed data helps to control any potential damage if that is the only source available for the needed data

But the reality is that even the most sophisticated data quality management activities do not prevent all data flaws. Consider the concept of data accuracy. Although we can implement automated processes for validating that values conform to format specifications, belong to defined data domains, or are consistent across columns within a single record, there is no way to automatically determine if a value is accurate. For example, salespeople are required to report their daily sales totals to the sales system, but if one salesperson inadvertently transposed two digits on one of the sales transaction amounts, the sales supervisor would not be able to determine the discrepancy without calling the sales team to verify their numbers (and even they might not remember the right numbers!).

The upshot is that despite your efforts to ensure quality of data, there are always going to be data issues that require attention and remediation. The goal is to determine the protocols that need to be in place to determine data errors as early as possible in the processing stream(s), whom to notify to address the issue, and whether the issue can be resolved appropriately within a “reasonable” amount of time. These protocols are composed of two aspects: controls, which are used to determine the issue, and service level agreements, which specify the reasonable expectations for response and remediation.

A data quality control framework bolsters the ability to establish data quality service level agreements by identifying the issues and initiating processes to evaluate and remediate them. Pushing the controls as far back as possible in each process stream increases trust, especially when the control is instantiated at the point of data acquisition or creation. In retrospect, the master data repository can be used to validate data quality, and, as we will explore in Chapter 12, the component service layer will embed the validation and control across the data life cycle.

A key component of establishing the control framework is a data quality service level agreement (SLA); the sidebar lists what should be in a data quality SLA.

In many master data management implementations, MDM team members and their internal customers have indicated that data quality improvement is both a driver and a by-product of their MDM or Customer Data Integration (CDI) initiatives, often citing data quality improvement as the program's major driver. Consider these examples:

Because the MDM program is intended to create a synchronized, consistent repository of quality master information, data quality integration incorporates data profiling, parsing, standardization, and resolution aspects to both inventory and identify candidate master object sets as well as to assess that data's quality. However, this must be done in a way that establishes resolution and the management of the semantics, hierarchies, taxonomies, and relationships of those master objects, and this process will be explored in Chapter 6 and Chapter 10. On the other hand, we have seen (in Chapter 4) that the benefits of imposing an enterprise data governance framework include the oversight of critical data elements, clear unambiguous definitions, and collaboration among multiple organizational divisions.

For MDM, these two ideas converge in the use of data profiling and data quality tools for assessment, semantic analysis, integration, data inspection and control, and monitoring—essentially across the board. Using profiling to assess and inventory enterprise metadata provides an automated approach to the fundamental aspects of building the master data model. Data profiling and data quality together are used to parse and monitor content within each data instance. This means that their use is not just a function of matching names, addresses, or products, but rather automating the conceptual understanding of the information embedded within the representative record. This knowledge is abstracted as part of a “metadata control” approach, with a metadata registry serving as the focus of meaning for shared information. Fully integrating data quality control and management into the organization is one of the single most important success factors for MDM.