Let's review our definition of master data from Chapter 1:

Aside from the preceding description, though, master data objects share certain characteristics:

For any conceptual master data type there will be some classifications or segmentations that naturally align along hierarchies, such as corporate ownership for organizations, family households for individuals, or product classification. However, the ways that even some of these hierarchical classification schemes (also referred to as “taxonomies”) are applied to the data instances themselves might cross multiple segments. For example, a “party” may represent either an individual or an organization; however, a party categorized as an individual may be further classified as both a “customer” and an “employee.”

As a valuable by-product of evaluating and establishing the hierarchies and taxonomies, we gain a degree of consistency that relates operations to reporting. The same master data categories and their related taxonomies would then be used for transactions, reporting, and analysis. For example, the headers in a monthly sales report may be derived from the master data categories and their qualifiers (e.g., sales by customer by region by time period). Enabling transactional systems to refer to the same data objects as the subsequent reporting systems ensures that the analysis reports are consistent with the transaction systems.

The top-down approach seeks to determine which business concepts are shared in practice across the different business processes. Because the operations of a business are guided by defined performance management goals, an efficient organization will define its business policies to be aligned with the organization's strategic imperatives and then implement business processes that support those policies. A business process, in this environment, is a coordinated set of activities intended to achieve a desired goal or produce a desired output product. Business process models are designed to capture both the high level and detail of the business process, coupled with the underlying business rules originating from the business objectives and subsequent auxiliary material that accompanies the corresponding definitions.

The traditional approach to designing business applications involves documenting the end clients' needs and then refining the ways that available technical solutions can be used to meet those needs. In essence, business objectives lead to formalizing the business policies. Those business policies then drive the definition of functional and informational requirements. The applications are then designed to implement these requirements as business logic operating on the underlying data objects.

Although as a matter of fact many application architectures emerge organically, the developed business applications are always intended to implement the stated business policies. Therefore, application designers should always strive to synchronize the way they implement policies with the ways that other applications in the enterprise implement business policies. To do this, the organization must maintain the relationships between business strategy and the components of the application's model:

Given an understanding of the way that a business application can be designed and implemented, we can apply the top-down approach to identifying the sources for master data by reviewing the business process models and identifying which data objects are critical within the processing streams of more than one application. What we will find is that individual business activities depend on the sharing or exchange of data objects. Those objects that would be shared are the ones that analysts can anticipate to be managed as master data.

Each shared data object has underlying metadata, a role within the business process, as well as its representation within the environment, suggesting a characterization as key data entities (as was discussed in Chapter 4). These key data entities reflect the business terms employed within the process, and those business terms are eventually rolled up into facts about the business process and its operations. Those key data entities that appear with relevant frequency as touch points within the business process models become the logical candidates for data mastering.

The bottom-up approach seeks to identify the master data objects that are already in use across the organization. This approach evaluates the enterprise data assets to locate applications using what are, in effect, replicas of master data objects and then enable their resolution into a proposed master data environment This is more of a widespread assessment of the data sets performed using data discovery tools and documenting the discoveries within a metadata management framework, as suggested in Chapter 6. This process incorporates these stages, which are examined in greater detail in this chapter:

In Figure 7.2, we look at the example of the two different representations for “customer” first displayed in Figure 7.1. Note that we are only looking at resolving the representations for data entities, not their actual values—that is discussed in Chapter 10 on data consolidation.

The first representation has four attributes (FirstName, MiddleName, LastName, and TelNum), whereas the second representation has eight (First, Middle, Last, Address1, Address2, City, State, and Zip). The data elements are reviewed at the three levels (syntactic, structural, and semantic). At the syntactic level, we can compare the data types and sizes directly and see that there are apparently minor differences—first and middle names are of length 14 in the top example but are of length 15 in the bottom; there is a similar difference for the corresponding last name fields.

At the structural level, though, we see some more pointed differences. Reviewing the data elements shows that each of these entities contains data related to two different aspects of the “person” master data type. One aspect is the identifying information, which incorporates the data values used to make a distinction between specific instances, such as name or assigned identifiers (social security numbers, customer account numbers, etc.). The other aspect is related entity data that is not necessarily required for disambiguation. In this example, there are data elements associated with contact information that could potentially have been isolated as its own conceptual master data object type.

Lastly, reviewing the semantics of the business terms associated with the data entities reveals another subtle difference between the two data models. In one application, the concept of customer refers to those who have “purchased one of our products,” whereas the other defines a customer as a person to whom we have “delivered one of our products.” The latter definition is a bit broader than the first in some ways (e.g., items such as marketing material sent free of charge to prospects may be categorized as products) and is limiting in others (e.g., service customers are not included).

The sidebar highlights the three stages in master data resolution that need to dovetail as a prelude to any kind of enterprise-wide integration, suggesting three corresponding challenges for MDM.

One approach involves analyzing and documenting the metadata associated with all data objects across the enterprise, in order to use that information to guide analysts seeking master data. The challenge with metadata is that to a large extent it remains sparsely documented. However, many of the data sets may actually have some of the necessary metadata documented. For example, relational database systems allow for querying table structure and data element types, and COBOL copybooks reveal some structure and potentially even some alias information about data elements that are components of a larger logical structure.

On the other hand, some of the data may have little or no documented metadata, such as fixed-format or character-separated files. If the objective is to collect comprehensive and consistent metadata, as well as ensure that the data appropriately correlates to its documented metadata, the analysts must be able to discover the data set's metadata, and for this, data profiling is the tool of choice. Data profiling tools apply both statistical and analytical algorithms to characterize data sets, and the result of this analysis can drive the empirical assessment of structure and format metadata while simultaneously exposing embedded data models and dependencies.

Discoveries made through the profiling process should be collected into a metadata repository, and this consolidated metadata repository will eventually enumerate the relevant characteristics associated with each data set in a standardized way, including the data set name, its type (such as different storage formats, including relational tables, indexed flat VSAM file, or comma-separated files), and the characteristics of each of its columns/attributes (e.g., length, data type, format pattern, among others).

At the end of this process, we will not just have a comprehensive catalog of all data sets, but we will also be able to review the frequency of metamodel characteristics, such as frequently used names, field sizes, and data types. Capturing these values with a standard representation allows the metadata characteristics themselves to be subjected to the kinds of statistical analysis that data profiling provides. For example, we can assess the dependencies between common attribute names (e.g., “CUSTOMER”) and their assigned data types (e.g., VARCHAR(20)) to identify (and potentially standardize against) commonly used types, sizes, and formats.

Despite the expectations that there are many variant forms and structures for your organization's master data, the different underlying models of each master data object are bound to share many commonalities. For example, the structure for practically any “residential” customer table will contain a name, an address, and a telephone number. Interestingly, almost any vendor or supplier data set will probably also contain a name, an address, and a telephone number. This similarity suggests the existence of a de facto underlying concept of a “party,” used as the basis for both customer and vendor. In turn, the analyst might review any model that contains those same identifying attributes as a structure type that can be derived from or is related to a party type.

Collecting the metadata for many different applications that rely on the same party concept allows the analyst to evaluate and document the different data elements that attribute a party in its different derivations. That palette of attributes helps the analyst to assess how each model instance maps to a growing catalog of data models used by each master data entity type.

There are two aspects to analyzing structure similarity for the purpose of identifying master data instances. The first is seeking out overlapping structures, in which the core attributes determined to carry identifying information for one data object are seen to overlap with a similar set of attributes in another data object. The second is identifying where one could infer “organic derivation” or inheritance of core attributes in some sets of data objects that are the same core data attributes that are completely embedded within other data objects, as in the case of the “first name,” “middle name,” and “last name” attributes that are embedded within both structures displayed in Figure 7.2. Both cases indicate a structural relationship. When related attributes carry identifying information, the analyst should review those objects to determine if they indeed represent master objects.

The many data types that abound, especially within the scope of the built-in types, complicate the population of the metadata repository. Consider that numbers can be used for quantity as easily as they can be used for codes, but a data profile does not necessarily indicate which way the values are being used within any specific column. One approach to this complexity is to simplify the characterization of the value set associated with each column in each table.

At the conceptual level, designating a value set using a simplified classification scheme reduces the level of complexity associated with data variance and allows for loosening the constraints when comparing multiple metadata instances. For example, we can limit ourselves to the six data value classes shown in the sidebar.

At this point, each data attribute can be summarized in terms of a small number of descriptive characteristics: its data type, length, data value domain, and so on. In turn, each data set can be described as a collection of its component attributes. Looking for similar data sets with similar structures, formats, and semantics enables the analyst to assess each data set's “identifying attribution,” try to find the collections of data sets that share similar characteristics, and determine if they represent the same objects.

Using data profiling to assess data element structure allows the analyst to collect structural metadata into a metadata repository. The analysis to evaluate the data attributes to find the ones that share similar characteristics is supplemented by data profiling and parsing/standardization tools, which also help the analyst to track those attributes with similar names. The analyst uses profiling to examine the data value sets and assign them into value classes and then uses the same tools again to detect similarities between representative data metamodels.

In essence, the techniques and tools we can use to determine the sources of master data objects are essentially the same types of tools that we will use later to consolidate the data into a master repository. Using data profiling, parsing, standardization, and matching, we can facilitate the process of identifying which data sets (tables, files, spreadsheets, etc.) represent which master data objects.

The analyst is now presented with a collection of master object representations. But as a prelude to continuing to develop the consolidation process, decisions must be made as part of the organization's governance process. To consolidate the variety of diverse master object representations into an environment in which a common master representation can be materialized, the relevant stakeholders need to agree on common representations for data exchange and for persistence, as well as the underlying semantics for those representations. This common model can be used for both data exchange and master data persistence itself; these issues are explored further in Chapter 8.

As discussed in Chapter 4, MDM is a solution that integrates tools with policies and procedures for data governance, so there should be a process for defining and agreeing to data standards. It is critical that a standard representation be defined and agreed to so that the participants expecting to benefit from master data can effectively share the data, and the techniques discussed in Chapter 10 will support the organization's ability to share and exchange data.