Chapter Eight. Plan for Change and Mass-Customize

Reality is merely an illusion, albeit a very persistent one.

Albert Einstein

Planning for change is a principle based on the underlying premise that it is impossible to precisely specify or control all requirements for an integrated enterprise information system. As this quote from Einstein suggests, when we think we have a clear specification for an application interface, it may not be reality. Factors that contribute to the hopelessness of realizing precise and complete specifications include

• Unpredictability of complex dependent software systems with millions of permutations and combinations of internal states

• Constantly evolving needs because of business, market, or technology changes

• Limits of human cognitive ability to translate qualitative needs and tacit knowledge into quantitative and codified requirements

In other words, while small, well-defined components can be precisely specified and engineered, the overall end-to-end integrated system must be viewed as an evolving system that is not entirely predictable and in a constant state of flux. In systems theory this is referred to as emergence, and it is the way complex systems arise out of many relatively simple interactions. In other words, the enterprise system-of-systems that emerges from years of individual integration efforts is considered a “complex” system that is continuously changing. It is for this reason that the principle of planning for change is critically important.

Most Lean practices refer to this principle as “defer commitment” or “decide as late as possible.” The general idea is to wait as long as possible until you have the maximum amount of information before making a decision that would be hard (or expensive) to reverse. Agile software development makes effective use of the plan-for-change principle in the practice of constantly refactoring components as new requirements emerge. A concept that is closely tied to this idea is mass customization, which includes practices such as designing products that are based on standard components and are customized, assembled, or configured for specific users; reducing setup times to enable small batch sizes (even batches of one); and leveraging the supply chain to achieve just-in-time inventory management. In the realm of integration, it is more relevant to think of this principle as planning for change, and there are several key techniques that can be leveraged to enable it.

This chapter describes several methods for enabling constant change and how to apply the concept of mass customization in an integration context. We then describe how these concepts can be applied in an Integration Factory scenario to deliver increased business value.

The chapter closes with a case study of a company in the health industry that used integration design patterns so that it could quickly create, or recreate, integration points to meet the needs of a dynamic and constantly changing business and technical environment.

Techniques for Enabling Constant Change

This section describes three practices that help to turn the principle of planning for change into concrete actionable activities:

1. Break dependencies so that one element of the system or process can change without affecting others.

2. Make decisions reversible in order to adjust quickly when new requirements or dependencies emerge.

3. Maintain “live” documentation to facilitate reuse and streamline planning.

Break Dependencies

The concept of breaking dependencies between components to enable each component (or system) to change without impacting others is not new or difficult to grasp; an example would be upgrading to a larger flat-screen monitor or an ergonomic keyboard for your desktop PC without changing other components. In practice it is difficult to implement at the enterprise system-of-systems level and requires three complementary techniques.

The first technique is an architectural approach to differentiating the whole. By this we mean that you need to create a holistic picture, or model, of the enterprise and its system-of-systems, and to divide it into logical components that minimize information exchanges between components. To relate this to a simple example in the non-IT world, a car consists of a number of systems including the power-train, electrical, cooling, steering, and braking systems, to name a few. Each system in turn consists of multiple components, many of which have interchangeable parts. One of the reasons this works is because car designers have clearly separated the functions of each system to minimize the number of interaction points with other systems and to optimize the overall economy and efficiency of the vehicle. There is nothing physically preventing the cooling system from generating electrical power for its own use, but if each system did this, it would create inefficiencies due to redundant electrical systems as well as limit the flexibility of interchangeable parts.

A number of architectural methodologies can accomplish modeling the enterprise to identify logical components that minimize information exchanges between components. A detailed discussion of them is beyond the scope of this book, but we do have some general advice. The best approaches for integrating information systems are those that use a top-down, service-based orientation for modeling the enterprise at the operations level. From a business perspective, the relationships between external entities, such as suppliers, channels, or customers, are described based on the services that flow between them. Services include provision of goods, selling of goods, transport, installation, and repair. A service-oriented approach is useful at this level for identifying cross-enterprise collaboration opportunities, evaluating in-sourcing and outsourcing scenarios, and assessing integration synergies with merger and acquisition candidates.

Just as companies provide services to each other in a supply chain, so too do internal functions within a corporation provide services to each other. Both the internal and external services can be modeled at a business level, independent of technology implementation details, including a description of the information exchanges associated with the flow of services. In other words, functional groups within an organization are linked by the services they provide to each other. This is a powerful way to view the internal capabilities of the enterprise, since the services focus on the purpose and results of the function. A service-oriented approach provides clean delineations for designing process interfaces, establishing meaningful performance metrics, and aligning functional accountability throughout the organization.

The end result of a good architectural approach is a model of the business that is ME&C—mutually exclusive and comprehensive. The model is comprehensive in that it describes the entire business entity, and it is mutually exclusive in that each of the elements of the model is maximally differentiated and nonoverlapping. Furthermore, the best models also map the functions to the data that is created and used by each function. This matrix of “service functions” mapped to “information subjects” provides a crisp and unambiguous definition of business services. This is essential for an architectural view that describes a hierarchy of systems that are maximally differentiated and minimally dependent.

While the model of the business may be ME&C, the underlying systems and data as implemented are not structured as nicely. Business processes and data are duplicated in different systems, and the lines of responsibility can be challenging to determine and relate across systems. While creating a model of the business is challenging, it is more challenging to take the next step of mapping that model to the underlying data as implemented.

It is important to use a logical abstraction layer for this mapping of data to the logical business model, because you do not want to have numerous groups independently reinventing the mapping on a continuous basis, which would be extremely wasteful and error-prone. Knowing the difficulty of the enterprise complexity of data and creating abstractions that can break traditional dependencies, and doing this in a governed, architected way that assumes continual change, means having an information or data service model layer that the business service layer uses for its operations.

The second technique for breaking dependencies is to implement loose coupling strategies between dependent components. Loose coupling is one of the most highly valued architectural properties—especially at the enterprise level—and is described more fully in Chapter 16, Modeling Management. Despite the earlier caution about applying middleware blindly, middleware technologies are indeed one of the most effective ways to achieve loose coupling and thereby help to break dependencies between applications.

One of best examples of loosely coupled systems can be seen in the typical supply chain. B2B interactions are made possible by industry standard data definitions, common protocols, and tools that handle process orchestration as a separate layer from the business applications. In other words, B2B interactions often include at least three types of middleware abstractions: canonical data models, common transport, and an orchestration layer. Loose coupling within an enterprise between vendor-purchased applications can be achieved with similar middleware infrastructures.

The third technique for breaking dependencies is to implement a modular integration process. Lean manufacturing uses the concept of “cells,” where each cell is a somewhat autonomous unit of work in an end-to-end manufacturing process. For example, some integration projects may require a data quality assessment while other projects may not. Furthermore, for some projects the data quality assessment might need to be performed during the requirements phase, whereas for other projects it might be needed during the design phase or the testing phase. To maintain this flexibility, the data quality assessment should be structured as an activity “cell” with similar inputs and outputs that can be applied at various points in a project process.

For this cell-like approach to work, it is critical that the information needs of the cell be well defined and that all the information be passed along when its work is initiated. If some information is missing, the cell will be blocked from completing its work or its output will be low quality. In traditional waterfall project methodologies this was referred to as “over the wall” and is often viewed as an antipattern because of the dysfunctions that are created if it is not done correctly, for example, when analysts pass their requirements over the wall to designers, who then pass designs over the wall to developers, who then pass coded software over the wall to testers.

A modular project process will fail if cells are forced to deliver their output on a fixed date or schedule even if they don’t have sufficient resources or information to complete the work properly. The resultant deliverable that is passed over the wall is therefore of low quality. After several handoffs in succession, the quality of the output will have deteriorated to the point of total write-off. The key, therefore, to successful modularization is to pass all the information that is needed from one step in the process to another. Clear and consistent documentation standards and a shared metadata repository are key enablers. We will talk about this topic in more detail in Chapter 12, Integration Methodology.

Make Decisions Reversible

The core concept behind making integration decisions reversible is to automate changes as much as possible. First, make it easy to rebuild integrations at the “push of a button” by using metadata-driven automated processes. The reality is that you never build an integration just once. You build it many times—and in fact it is common practice to rebuild an integration a number of times even before the first production deployment since it sometimes isn’t until system integration testing that the “real” data definitions and data quality issues appear. So if changing an integration is the norm, why not automate as much of the work as possible to the point where you can quickly re-create a production integration at the push of a button?

For example, let’s say you made a design decision to use an EII (enterprise information integration) approach to create composite views of heterogeneous customer account data in real time, only to discover through volume testing that the response time does not meet requirements. The alternative approach is to use a data integration solution with some of the customer account data replicated or cached in order to achieve the performance requirements. Normally this would be a significant design change that would impact the overall project timeline if made late in the implementation cycle. However, if the logical model is location-transparent and the definition of the model hasn’t changed, this technology change would be entirely transparent to consuming applications. Furthermore, if the systems of record and data-mapping rules are maintained in a metadata repository, and if the data transformation routines are reusable middleware components, it will be possible to quickly reverse the original design decision, if this is ever desired.

An additional capability is to use metadata-driven automated processes to migrate software changes between environments (development to test to production) and to reverse the changes at the “push of a button” if unintended consequences emerge. This technique provides the benefit of maintaining an audit trail of configuration changes for improved governance of the IT environment. It also enables the capability of implementation changes in production in small increments (small batch sizes).

Maintain “Live” Documentation

Our definition of live documentation in the integration domain is “a structured repository that contains current data about data and data processes.” In other words, maintain a metadata repository that always shows the current configuration of data sources, data targets, business transformation rules, and information interchange processes. Methods for metadata management are discussed in more detail in Chapter 13, but what we want to address here is why it is important.

An analogy may help to make the point. Do you need the blueprints of a high-rise building after it has been constructed if you never plan to change the building? The short answer is no. Blueprints serve as design documents to help the builders construct a complex structure. Once construction is complete, the design documents are of little use and in fact may be wrong, since it is common for builders to modify the construction to resolve implementation details that the designer didn’t think about or didn’t specify. The reality for most buildings, however, is that they do change. The changes may be relatively simple, such as upgrading the heating and air-conditioning system, installing a security system, or wiring the building with fiber-optic cables. Or they may be major changes, such as knocking down a wall to create a larger room, adding a swimming pool to the roof, or rebuilding the sewer system that runs under the building. If you don’t have blueprints that reflect an accurate model of the building, you have three options before making a significant change: invest the time, money, and expense to re-create the design documents “as built” so that engineers and designers can safely plan the change; make the change and see what happens; or demolish the building and construct a new one.

At the enterprise level, the system-of-systems is constantly changing. In most large enterprises, hardly a day goes by without one or more changes being made to production systems. The count of change tickets documenting production changes typically numbers in the thousands of changes per year for most Fortune 1000 companies. Demolishing and rebuilding are not options for large enterprises because of the massive cost and complexity, not to mention the resulting downtime. Therefore, without current and accurate metadata, organizations are faced with a trade-off: Either perform the expensive and time-consuming process of re-creating an adequate understanding of the current state each time a change needs to be made, or simply make the change and keep your fingers crossed.

There is a compelling business case to be made for maintaining a metadata repository in terms of reducing operational risk, reducing the cost of implementing changes, and increasing the speed with which changes can be made. It is not easy to build a business case for a metadata investment, and it is not easy to implement a metadata strategy. Nonetheless, both are achievable, as we will describe in Chapters 11 and 13, and organizations that have done so have realized tremendous benefits and sustained competitive advantages.

Mass Customization

Mass customization techniques facilitate planning for change in that a given integration becomes more of an assembly process than a custom development effort so it can be completed quickly in response to new or changed requirements.

The benefits of mass customization are all around us. You can order a PC online from Dell with a custom combination of CPU chip, memory capacity, graphics card, hard drive, DVD, and other components, and it shows up on your doorstep within a few days. You can lay out your Yahoo! Web page with whatever news, weather, stocks, and other information suits your needs. Or you can order customized food such as personalized M&M candies, cakes with your picture or logo on them, or custom cereal with just the right mix of ingredients to match your dietary needs.

At its core, mass customization enables a tremendous increase in variety and customization without a corresponding increase in costs. When mass customization is done well, it costs the consumer no more for a customized product than for a mass-produced standard product. Of course, the reason this trend is so strong is that everyone is different; if we can have something that meets our unique needs at a similar cost to a generic product, it becomes an easy decision.

To be clear, mass customization is different from mass production and from continuous improvement. Pine, Victor, and Boynton stated the differences well in their 1993 article:¹

1. B. Joseph Pine II, Bart Victor, and Andrew C. Boynton, “Making Mass Customization Work,” Harvard Business Review (September–October 1993).

Mass Production
The traditional mass-production company is bureaucratic and hierarchical. Under close supervision, workers repeat narrowly defined, repetitious tasks. Result: low-cost, standard goods and services.

Continuous Improvement
In continuous-improvement settings, empowered, cross-functional teams strive constantly to improve processes. Managers are coaches, cheering on communications and unceasing efforts to improve. Result: low-cost, high-quality, standard goods and services.

Mass Customization
Mass customization calls for flexibility and quick responsiveness. In an ever-changing environment, people, processes, units, and technology reconfigure to give customers exactly what they want. Managers coordinate independent, capable individuals, and an efficient linkage system is crucial. Result: low-cost, high-quality, customized goods and services.

Mass Customization of Data

Mass customization is a core concept behind SOA, but discipline is still required to execute it effectively. For example, most large IT shops have a common routine for doing customer address cleansing, but it is surprising how many teams within the enterprise don’t use it and reinvent their own, which defeats the purpose of a common routine. Part of the solution is to establish an ICC or a central team that is responsible for maintaining and enhancing the shared business objects and reusable components on an ongoing basis.

Mass customization can be applied to the integration domain in two contexts: mass customization of integration components, or mass customization of data. Mass customization of integration components has many parallels with traditional product manufacturing. The basic idea is to build integration components (standard routines for data extract, data load, transformation, security, encryption, etc.) so that they can quickly be configured and assembled for a specific set of requirements rather than developed from scratch.

The idea of mass-customizing data may not be so intuitive. This may sound radical, but we suggest that organizations should strive to create more, not less, customized data. Actually, it’s not that organizations need more customized data; what they need is the flexibility to adapt information to meet local needs (e.g., for different functions, divisions, or regions) while still maintaining consistency where it matters—in other words, to accept the reality of variation in data and, rather than fight it, adopt methods that effectively support it.

For example, at a high level of abstraction, the process and supporting data for authorizing a loan for a consumer are quite consistent. Whether you are applying for a credit card, car loan, mortgage, line of credit, or student loan, the types of data that is needed are similar. The lending organization needs information about the borrower, what the money is for, and the borrower’s ability to repay the loan, and then the lender can make a calculated decision about whether or not to extend credit. The devil, of course, is in the details. Is the credit request for a large or small amount; for a high-net-worth customer or a minimum-wage worker living from paycheck to paycheck? Is there a secured asset that needs to be appraised? Is the application being processed on the Internet, at a call center, or at a retail outlet? What regulatory rules apply, and should the documents be printed in English, Spanish, or French? These and many other factors drive the need for tremendous variation. Rather than trying to eliminate the variation, we should embrace the natural diversity that exists in a typical complex business setting.

This is where the concept of mass customization comes in. We need a way to facilitate a huge, even limitless, variety of data without losing the ability to consolidate it or make sense of it. Having the appropriate metadata and modeling management capabilities is crucial for understanding the relationship between the enterprise model for data and the “customized for local needs” variants necessary to run the business.

The industry has been fighting variation in data for decades with efforts such as enterprise data models, which attempt to impose consistent definitions on data for the entire enterprise, or ERP solutions, which mandate a “one size fits all” strategy on business processes. The industry has also tried to standardize data through industry organizations that have worked hard to homogenize subject domains as diverse as financial transactions, shipping orders, invoices, health insurance claims, and credit ratings. The prevailing practice that is embedded in most IT methodologies is the notion that all data should be consistent. It is our view that this is a flawed premise that fights the natural order of things, and that instead we should find ways to efficiently customize data and effectively manage the inherent variations.

To visualize this, the first thing you need to do is step out of your silo and look at the big picture: the information needs of the enterprise where “one version of the truth” is the product that is being produced. The information needs at this level are relatively modest when compared to the information needs of the front-line operations of all functions combined. Most enterprises have in the order of 10 to 20 subject areas, and each subject area has around 50 to 100 business objects. Simple objects might only have 10 attributes, and complex objects could have 100 or more attributes. While it is a nontrivial effort to create an enterprise-level subject area model, it is an order of magnitude (10 times or 100 times) simpler than the detailed operational-level data model that captures all the as-built variations.

From an enterprise information perspective, what you should see is a relatively stable and consistent picture of the business, and the information to support the enterprise should also be correspondingly stable. From an operational perspective, what you should see is a highly customized and dynamic collection of data and processes that allows each function to adapt quickly to meet day-to-day needs and to do so without having to rely on complex coordination with other functions.

To achieve this apparent paradox between the stable enterprise view and the flexible operational view, we need four elements: an enterprise-level subject area model, an architectural approach that separates business functions into logical groups, a middleware layer that decouples the enterprise view from the operational view, and an integration function that sustains the links between operational functions and the enterprise view. We will come back to each of the elements in specific sections of Part III of the book, but for now we will stay at a high level with the following definition:

To mass-customize data: Implement a consistent definition of information that can be applied across all business functions independent of the technology implementation and based on a sustainable integration infrastructure that separates the consolidated enterprise view from the many departmental and functional systems in a loosely coupled fashion. This is quite a mouthful, so we will elaborate on the highlighted terms.

For a consistent definition of the business and its information needs you must start from a top-down perspective. This is hard work but it needs to be done. There may be a tendency to take a bottom-up approach since it seems easier to abstract away details from an existing system or data model, but this presents several problems. First, the existing systems may not be ideal implementations in terms of their support of the business operations and won’t be consistent if they came from different vendors or were architected by different groups. A bottom-up approach tends to impose current-state implementation details on the business, which may limit future flexibility. As a result, you end up “paving the cow path” by institutionalizing an inappropriate implementation at the enterprise level. Furthermore, a bottom-up approach may be overwhelming and take on the characteristics of a “boiling the ocean” problem.

It is also critical to describe the business functions and data independent of implementation details, which also argues for a top-down approach. In other words, you need to describe what the business does and not how it does it. The how part of the picture is a black hole of implementation details; it is a never-ending job to try to model it all.

A sustainable integration infrastructure demands two key elements. First is the recognition that integration systems are distinct entities from business systems and should be viewed as applications in their own right and not simply as extensions or appendages of business applications. The second key point is that the infrastructure must be sustainable, which means it needs to be managed in a disciplined fashion with periodic investments to continuously optimize it.

The core purpose of the integration infrastructure is to maintain a consolidated enterprise view by collecting and synchronizing data across operational systems, and to do so in a loosely coupled fashion so that individual components can change with little or no impact on other components. This concept is not new, but it is not automatic or trivial. Mature ICC groups have demonstrated that it is both achievable and beneficial, but it does require an enterprise strategy and a dedicated organization to execute it.

Mass Customization of Integration Logic Using Assembly Lines

One of the most logical approaches to mass-customizing data is to use mass customization of integration logic to deliver that data. Within different types of process or data integration projects, there are different styles of data movement, transformation, and delivery. Assembly lines can create this integration logic, customized for the data in question, but following a relatively small number of identifiable patterns that are common to that type of project.

For instance, in a data warehousing project, many different pieces of integration (sometimes called “mappings” or “services”) are necessary to keep the data warehouse populated with clean, current, and historically correct data. Some integration logic is necessary to update master data or dimension tables, and some integration logic is necessary to update transaction data or fact tables. For a dimension table, whether it is a product dimension or a customer dimension table, the integration logic follows a common pattern. The specifics of the field names and the particular field transformations may be unique, but the way that the keys are generated, the way that date fields are updated, and the way that history is maintained are common from one dimension table to another. For a typical warehouse project, there may be hundreds of mappings or services, but there are typically a dozen or so different patterns into which those mappings or services fall.

In specific data warehousing terms, patterns that are readily known in the industry include these:

• Slowly changing dimensions (SCDs)

• Audit dimensions

• Incremental fact tables

• Accumulating fact tables

• Bridge tables

• Staging area tables

• Aggregate tables

• Incrementally updating aggregate tables

• Change-data-captured real-time fact tables

• Change-data-captured real-time dimension tables

Other types of integration projects would introduce different patterns. For data warehousing, while this list is not exhaustive, it provides a representative idea of different patterns of integration logic.

An optimal approach to implementing patterns (or mass customization) in a factory context is to have the integration architect capture the pattern in a tool that allows the generation of many mappings that follow this pattern. In other words, once the template or pattern is defined, a wizard is automatically generated from it. Individual analysts or developers fill out this wizard or set of forms, specifying the unique characteristics of each mapping or service.

This pattern-oriented approach to integration represents a purpose-built instantiation of model-driven architecture, as applied specifically to the field of integration services, mappings, or flows. We cleanly separate what a mapping is supposed to do from how the mapping gets executed, so that we can more specifically tune the how based upon the style of integration we are building.

The big win comes with the second project of that type. In the new project, most if not all of the patterns may be reused. The first project benefited from the use of patterns, but subsequent projects benefit even more.

Several organizations have incorporated this approach. One of us (Lyle) and his ICC team, responsible for building the Informatica Analytic Applications in 2000–2003, built a tool set with about a dozen data warehousing mapping patterns. Once the data warehouse target data model was designed, an Excel spreadsheet for the appropriate pattern was identified. The developers would fill out the spreadsheet, defining how SAP or Oracle application fields mapped to the target fields within the data warehouse, and a proprietary tool would read the spreadsheet and generate a robust, real-world mapping that corresponded to the mapping logic specified by the spreadsheet.

The result was elimination of waste not only in the implementation process, but more important in the QA and change management cycles. The details of the different patterns of data warehouse mappings were generalized and hidden from the analysts and developers, so they didn’t have to worry about concepts like how to handle SCDs, or how to resolve surrogate keys, or how to properly convert currencies to a global currency, or how to design mappings that took advantage of the best-known system performance approaches. The analysts and developers focused on the core business requirements, reducing wasted energy and variance in the output.

These same concepts are just as valuable when building other styles of integrations besides data warehousing. Master data management projects have their own integration patterns, as do process automation projects. Building and using these patterns so that developers are simply filling out forms or custom wizards is a way to use automation to effectively eliminate waste while still allowing customizations when necessary.

Using these patterns to generate integration logic (mappings, services, or what have you) allows for the automatic incorporation of the best-known approaches for that particular pattern. The person filling in the pattern does not have to be an expert in all of the intricacies of how to best build that integration artifact. He or she needs to focus only on the specific data and any unique business rules or cleansing that are unique to that data. Once the form is filled out, the generated object automatically incorporates the best approaches as defined by the continuously improved integration patterns produced by the team over time.

For many ICCs whose members have time to be responsible for only administration and operations, this assembly-line approach is the way to make the ICC, or any integration team, scale to create higher quality and more efficient integration. Quality will be built into the integration logic, reducing variance and defects. Deviation from those standards is both possible and allowed, but having some foundation to start from has tremendous benefits.