Introduction

Knowledge is power.

Sir Francis Bacon¹

1. Sir Francis Bacon, Relegious Meditations of Heresies, 1597.

Lean Integration is a management system that emphasizes continuous improvement and the elimination of waste in end-to-end data integration and process integration activities. Lean practices are well established in other disciplines such as manufacturing, supply chain management, and software development to name just a few, but the application of Lean to the integration discipline is new.

Lean manufacturing is a management system that emphasizes creating value for end customers and eliminating activities that are not value-added (waste). Its principles were derived from the Toyota Production System (TPS), which was developed over 50 years ago but since the 1990s has simply been referred to as Lean. While Lean is rooted in product manufacturing, it is now widely regarded as a management approach that can be applied effectively to a wide range of product and service industries. Lean is closely related to, and borrows from, other methodologies, including Value Network, Theory of Constraints, Six Sigma, and Statistical Process Control (including the work of W. Edwards Deming).

Lean software development is an agile approach that translates Lean manufacturing principles and practices for the software development domain. It was adapted from the TPS and introduced by Mary and Tom Poppendieck in their book Lean Software Development and expanded in Implementing Lean Software Development, followed by Leading Lean Software Development in 2009.²

2. Mary and Tom Poppendieck, Lean Software Development: An Agile Toolkit (Addison-Wesley, 2003); Implementing Lean Software Development: From Concept to Cash (Addison-Wesley, 2007); Leading Lean Software Development: Results Are Not the Point (Addison-Wesley, 2009).

Lean Integration builds on these prior works by applying their principles to the process of integration. The definition of integration used in this book is “the practice of making independent applications work together as a cohesive system on an ongoing basis.” While there are myriad integration technologies and information exchange patterns, broadly speaking, integration solutions typically fall into one of two styles:

1. Process integration: automation of processes that cut across functional or application boundaries where process state needs to be maintained independently of the underlying application systems or where multiple data consumers or data providers need to be orchestrated as part of a business transaction

2. Data integration: accessing data and functions from disparate systems to create a combined and consistent view of core information for use across the organization to improve business decisions and operations

Another integration style that some practitioners call out as a separate category is service integration as part of a service-oriented architecture (SOA), where application functions are separated into distinct units, or services, that are directly accessible over a network in a loosely coupled manner and may be orchestrated with stateless interactions. The contrary argument is that service integration and the associated infrastructure of an enterprise service bus (ESB) are simply characteristics of a given process integration or data integration design.

Regardless of how many integration categories there are, Lean Integration applies to all of them. Software systems are, by their very nature, flexible and will change over time (legacy systems are often an exception). Interfaces and information exchanges between systems are never built just once. Integration therefore is not a one-time activity; it is ongoing. In summary, therefore, Lean Integration is the application of Lean principles and methods to the challenges of process and data integration on a sustainable basis.

This book builds on the first book that we wrote in 2005, Integration Competency Center: An Implementation Methodology.³ Integration Competency Centers (ICCs) continue to be a key ingredient of efficient and sustainable integration, and the core concepts in the 2005 book remain applicable to a Lean Integration practice. It is not necessary to read the ICC book first, but it still serves as an effective primer. This book adds depth to ICC principles and methods and extends the concepts to a broader view of the value chain.

3. John G. Schmidt and David Lyle, Integration Competency Center: An Implementation Methodology (Informatica Corporation, 2005).

One of the things we have learned since writing the first book is that an ICC by any other name is still an ICC. For example, some groups have names such as

• Integration Solutions Group (ISG)

• Center of Excellence or Center of Expertise (COE)

• Business Intelligence Competency Center (BICC)

• Data Quality COE

• Enterprise Data Warehouse (EDW)

• SOA COE

• Center of Competency (CoC)

And the list goes on. The point is that the principles and methods presented here are management practices that apply to all of them. Of course, the subject area expertise is different for each one, as are the scope, the technical skills needed by the staff, the specific tools and technologies that are used, and the internal and external customers that are served by it. In any event, we need to have a name for the organizational unit and need to call it something in this book, so we will continue to use Integration Competency Center, or ICC for short, as the umbrella term for all varieties. For the record, our definition of Integration Competency Center is as follows:

A permanent cross-functional team operating as a shared-services function supporting multiple organizational units and sustaining integration solutions in a coordinated manner

This book is divided into three parts. Part I serves as a summary for executives of Lean Integration and therefore provides an overview for a broad audience from senior IT executives to front-line operations staff. It provides an overview of and justification for Lean by answering questions like these:

• “Why Lean?” and “So what?”

• “As a business executive, what problems will it help me solve?”

• “As an IT leader or line-of-business owner, why am I going to make a considerable investment in Lean Integration?”

• “How is this different from other methods, approaches, and frameworks?”

• “Why am I as an IT professional going to embrace and sell Lean Integration internally?”

This first part also provides an overview of Lean practices, where they come from, and how they have evolved. It includes insightful research and current trends in how Lean is being adapted to different industries and management disciplines.

Part I concludes with an overview of the Integration Factory—the next-generation integration technology that adds a high degree of automation to the flow of materials and information in the process of building and sustaining integration points. Examples of automation include requirements definition, code generation, testing, and migration of code objects from development to test to production environments. The Integration Factory, we believe, will be the dominant new “wave” of middleware for the next decade (2010s). It views the thousands of information exchanges between applications in an enterprise as mass customizations of a relatively small number of patterns.

The management practice that optimizes the benefits of the Integration Factory is Lean Integration—the use of Lean principles and tools in the process of making independent applications work together as a cohesive system. The combination of factory technologies and Lean practices results in significant and sustainable business benefits.

Part II introduces the seven Lean Integration principles that optimize the Integration Factory and shows how they can be applied to the challenges of system, data, and application integration in a sustainable fashion. This section of the book is targeted at business and IT leaders who are implementing, or considering implementing, a Lean Integration program. Each chapter in this part focuses on one of the seven core principles:

1. Focus on the customer and eliminate waste: Maintain a spotlight on customer value and use customer input as the primary driver for the development of services and integration solutions. Waste elimination is related to this principle, since waste consists of activities that don’t add value from the customer’s perspective rather than from the supplier’s perspective. Related concepts include optimizing the entire value stream in the interest of things that customers care about and just-in-time delivery to meet customer demands.

2. Continuously improve: Use a data-driven cycle of hypothesis-validation-implementation to drive innovation and continuously improve the end-to-end process. Related concepts include how to amplify learning, institutionalizing lessons learned, and sustaining integration knowledge.

3. Empower the team: Share commitments across individuals and multi-functional teams and provide the support they need to innovate and try new ideas without fear of failure. Empowered teams and individuals have a clear picture of their role in the value chain, know exactly who their customers and suppliers are, and have the information necessary to make day-by-day and even minute-by-minute adjustments.

4. Optimize the whole: Make trade-offs on individual steps or activities in the interest of maximizing customer value and bottom-line results for the enterprise. Optimizing the whole requires a big-picture perspective of the end-to-end process and how the customer and enterprise value can be maximized even if it requires sub-optimizing individual steps or activities.

5. Plan for change: Apply mass customization techniques to reduce the cost and time in both the build and run stages of the integration life cycle. The development stage is optimized by focusing on reusable and parameter-driven integration elements to rapidly build new solutions. The operations stage is optimized by leveraging automated tools and structured processes to efficiently monitor, control, tune, upgrade, and fix the operational integration systems.

6. Automate processes: Judiciously use investments to automate common manual tasks and provide an integrated service delivery experience to customers. In mature environments, this leads to the elimination of scale factors, the ability to respond to large integration projects as rapidly and cost-effectively as small changes, and the removal of integration dependencies from the critical implementation path.

7. Build quality in: Emphasize process excellence and building quality in rather than trying to inspect it in. Related concepts include error-proofing the process and reducing recovery time and costs.

Part III is intended for those who have direct responsibility for implementing an integration strategy or are members of an integration team and are interested in improving their ICC or Lean skills. It provides detailed best practices, grouped into seven competency areas. These competencies are ongoing capabilities that an organization needs in order to provide a sustainable approach to integration.

1. Financial management: Financial management is a vital competency of a Lean practice that is operating a shared-services group; it takes more than technical activities to create and sustain an effective shared-services group. By showing how to articulate the business value of technology, we are seeking to ensure that the team does not become isolated from the business environment that sustains it. Credibility is established not only by operating a successful competency team, but also by being perceived by business leaders as doing so.

2. Integration methodology: An integration methodology defines the life cycle of dependencies that are involved in building a Lean Integration team to the point that it becomes an ongoing governing body, able to sustain the integration achieved from specific projects. The integration methodology is concerned with not just building quality solutions, but more important with setting up the processes to sustain solutions indefinitely in a production environment.

3. Metadata management: The ability to manage metadata is essential for managing data as an asset, and it is an enabler for a broad range of process and data integration programs such as straight-through processing, master data management, and data governance. Metadata is conceptually simple but challenging to implement in large, complex organizations. Essentially it is documentation and business rules about data in terms of what it means; where it is located; how it is accessed, moved, and secured; who is responsible for it; and who is allowed to use it.

4. Information architecture: Information architecture is an element of a broader enterprise architecture capability. Architecture and integration are complementary enterprise practices that intersect most significantly in the information domain. Architecture is about differentiating the whole and transforming the business, whereas integration is about assembling the parts into a cohesive, holistic system. One discipline takes a top-down approach while the other is bottom-up; both are essential.

5. Business process management: Business process management (BPM) is a method of efficiently aligning an enterprise with the needs of its customers. It is a holistic management approach that promotes business effectiveness and efficiency while striving for innovation, flexibility, and integration with technology. Business processes are a prime integration point since this is often where disparate functions come together and interact to share data in the interests of maximizing customer satisfaction and enterprise effectiveness.

6. Modeling management: Modeling management is the discipline for defining, using, and maintaining a variety of views (simplified abstractions) of the enterprise and its data in support of integration strategies. One of the critical needs within all large corporations is to achieve efficient information exchanges in a heterogeneous environment. The typical enterprise has hundreds of applications that serve as systems of record for information that, although developed independently and based on incompatible data models, must share information efficiently and accurately in order to effectively support the business and create positive customer experiences.

7. Integration systems: This competency addresses the need for managing the life cycle of integration technology and systems as a distinct class of application that provides a sustainable operating infrastructure to support the free flow of information in an organization in an integrated manner.