Chapter Three. The Integration Factory
Very simply, a flexible factory enjoys more variety with lower total costs than traditional factories, which are still forced to make the trade-off between scale and variety.
George Stalk1
1. George Stalk, “Time—The Next Source of Competitive Advantage,” Harvard Business Review, no. 4 (July–August 1988).
Lean is a management system that originated in a factory setting. It has since been applied to many other types of production processes and also more recently to a wide range of nonmanufacturing business processes such as supply chain management, health care delivery, order management, and others. Perhaps one of the most interesting and innovative applications of Lean is the use of factory concepts for common integration activities. How can one apply Lean concepts such as mass customization, just-in-time delivery, mistake-proofing (poka-yoke), pace (Takt time), production leveling (heijunka), and work cells to the virtual world of process and data integration?
In this chapter we will explore the use of the factory model to answer these questions and explain how they can help you achieve improved customer satisfaction, greater efficiencies, and business agility.
What Is an Integration Factory?
An Integration Factory is a cohesive integration technology platform that automates the flow of materials and information in the process of building and sustaining integration points. The reason this approach has proven to be highly successful is the incredibly high degree of similarity among integration points. In a manner similar to automobiles, which can be classed as sedans, pickup trucks, coupes, and so on, integration “code” also falls into different classes. Whether the class is a “slowly changing dimension” class, a “change-data-captured event stream” class, or a “publish-and-subscribe” class, the exact makeup of the data being integrated may be different for each instance, but the general processing pattern and structure are the same for each class.
Not only do integration points fit into a relatively small number of classes similar to cars, but the life cycle of discovery, development, and deployment of these integration points also follows a similar pattern, and factory-oriented paradigms assist in the life-cycle process as well. Examples of automation include automating requirements definition by leveraging a business glossary, canonical models, and metadata repositories for source and target data models; automating code generation using pattern-based wizards; automating testing using script-driven testing frameworks; and automating migration of code objects from development to test to production environments using metadata-driven scripts and workflows.
The Integration Factory, we believe, will be the dominant new “wave” of middleware for the next decade (2010s). It views the thousands of information exchanges between applications in an enterprise as mass customizations of a relatively small number of patterns. The integrations are managed collectively as a system, independently of the applications they serve. The management practice that optimizes the benefits of the Integration Factory is Lean Integration—the use of Lean principles and tools in the process of making independent applications work together as a cohesive system. The combination of factory technologies and Lean practices results in significant and sustainable business benefits.
How does an Integration Factory compare with a traditional factory? The Merriam-Webster Dictionary defines factory as a “facility for manufacturing” and defines manufacturing as “the process of making wares by hand or machinery, especially when carried on systematically with division of labor.” The term factory may, however, conjure up negative images of pollution or low-cost labor and poor working conditions. These older factories did indeed offer efficiencies in comparison to prior methods, but they also introduced downsides. Standardized work can be very mundane (boring), and working conditions may be less than ideal. The term sweatshop comes from a working environment where conditions are considered to be difficult and where the workers have few opportunities to address their situation.
Modern factories with a high degree of automation (jidoka) and emphasis on cleanliness look very different. There are still workers in these factories, but their roles are more related to maintaining the machines that do most of the work, supervising workflow, performing the manual activities that are difficult (or expensive) to automate, and dealing with exceptions.
The traditional view of a factory, and the images that come to mind for most people, relates to the flow of materials and the process that results in an end product—in other words, the assembly line. There is, however, another flow that is just as critical for an effective and efficient factory: the flow of information both among factory workers and to and from external customers, suppliers, and stakeholders.
Prior to the computer age, the information flow was largely verbal or paper-based. Lean organizations improve the information flow among workers with andon lights, kanban cards, and other visual signaling mechanisms. Computers, industry standards (such as EDI—Electronic Document Interchange), and supply chain integration methods have enabled suppliers to be more directly connected to the factory information flow. The age of the Internet has enabled customers, regulators, and other stakeholders to interact directly with the factory. Dell is a good example of a personal-computer factory that has largely automated the flow of information. The Dell Web site allows users to customize the configuration of a personal computer and return to the Web site later to change the configuration or track progress. These self-service portals coupled with mass customization techniques allow individuals to customize and purchase products and even monitor production status in some cases.
How Is the Integration Factory Different from Traditional Integration Approaches?
The Integration Factory is a unique innovation in the integration industry. It views the thousands or tens of thousands of information exchanges between application components in an enterprise or supply chain as a relatively small number of patterns that can be mass-customized. It combines the concepts of reusable component frameworks, a highly optimized process, the ability to perform specific integration activities anywhere in the world, a shared meta-data repository, and industry standards to produce low-cost, high-quality interfaces that can connect anything to anything.
Another way to think of the Integration Factory is that it produces products of one, delivered as a service, and it does so inexpensively and fast. It is based on the premise that the utopian vision of a universal interface doesn’t exist and won’t exist in the foreseeable future. Rapidly advancing technologies, a proliferation of standards, and vendor proprietary interests ensure that the software industry will continue to create new waves of interface protocols for years to come. The conclusion? Interfaces are not built once; they are built dozens or even hundreds of times over the life of the application as the various dependent platforms, tools, technologies, or business processes change.
The Integration Factory breaks the traditional paradigm of custom point-to-point interfaces with a new paradigm that challenges the traditional product model. The factory output is indeed a “product”: a fully deployable and executable program (or set of components) that connects applications or consolidates data from two or more applications. Yet each “product” is unique in that it is constructed to meet a specific business need and to interface with a specific application, handle specific data, on a specific operating system platform, with a specific protocol. But the factory is also a “service.” Interfaces are built on demand and to specification, they are maintained and supported in operation, and they are quickly regenerated whenever any of the dependent components or technology is changed.
A key difference between the Integration Factory and the traditional view of a widget assembly line is the operational aspect of the deployed integration components. In other words, we have both a development factory that builds or modifies integration components and an integration hub or bus that operationalizes the components to perform the day-to-day or second-to-second exchanges and transformations of data between applications. To put yet another spin on this, the development factory produces the machines (software components) for the operational factory.
Furthermore, both of these factories, the development factory and the operations factory, contain both types of flows: material flow and information flow. The net result is an Integration Factory that may provide both development and operational services, as summarized graphically in Figure 3.1.
Figure 3.1 Four flow types in an Integration Factory
As we discussed in Chapter 1, the Integration Factory is a logical next step in the evolution of integration technology. It takes the integration platform to the next stage of maturity by adding automation to the flow of materials and the flow of information. In this regard it is difficult to separate the technology elements from the people and process elements because of the interplay between them.
Can an Integration Competency Center (ICC) run a factory without applying Lean practices? The answer is yes—at least to some degree. The ICC provides the necessary organizational focus, which is one of the prerequisites for effectively leveraging a factory. An ICC also brings to bear a number of competencies that are essential for sustaining factory operations. That said, it is the Lean practices that really begin to leverage the power of the factory infrastructure. The disciplines associated with continuous improvement, value stream mapping, pull, flow, and waste elimination (to name a few) will result in even greater benefits than ICC competencies alone.
What Problem Is the Integration Factory Trying to Solve?
The core objective of the factory is to provide an efficient and effective way to build and sustain information dependencies in a highly complex application environment that is constantly changing. The Integration Factory improves the quality and performance of the integration solutions by using reusable components, standard processes, and continuous improvement processes to build quality in. The factory also enables organizational agility by delivering integration solutions rapidly as needed (just in time). In short, the purpose of an Integration Factory is to design, build, and support integration solutions better, faster, and more cheaply than traditional custom or ad hoc integrations.
To put the Integration Factory approach into perspective, it is worth restating the fundamental reason why integration is needed and what approaches have been attempted in the past. In order to meet the computing needs of a typical enterprise, it is necessary to operate numerous distinct computing platforms simultaneously. On each platform, separate software applications work together to handle the data-processing needs of the enterprise. These computer applications are designed first and foremost to operate as stand-alone autonomous units and second to interoperate with other applications.
In the 1970s and 1980s, individual project teams handled intersystem communication on a case-by-case basis; interfaces were seen as simple “appendages” to applications. In the 1990s, in response to the rapidly growing number of interfaces, a new class of software emerged called middleware. Its express purpose was to provide a common reusable integration layer between independent applications with the goals of reducing the number of point-to-point interfaces, increasing reuse, and reducing costs. The vision was to provide a service to applications that would permit them to be tightly integrated while remaining loosely coupled; although dependent on each other, applications should be allowed to evolve independently. The middle-ware strategy worked extremely well—so much so, that the number of interfaces exploded in response to increasing business demands to link systems and automate business processes.
Despite all the progress, however, a major challenge remained. The industry failed to achieve a universal interface standard that would allow any given application to communicate with any given middleware tool. Several industry standards have been proposed and some are promising, but none is dominant, and any given large enterprise likely has many flavors. Furthermore, many middleware vendors and application vendors promote various proprietary protocols in the interest of promoting their product strengths or other marketing-related motivations. The interface variations are further complicated by the constant evolution of new proposed standards and new technology waves. Since it is rarely justifiable to rewrite all the legacy interfaces for every new technology wave, most organizations have a variety of legacy and new interfaces that all must be supported. The end result is that a typical large enterprise has a hodgepodge collection of interfaces and high maintenance costs. Estimates for building and maintaining interfaces between applications range from 30 percent to 70 percent for typical large, complex IT environments.
When Not to Use a Factory Approach
There are indeed some circumstances when an Integration Factory is not the best approach for data and process integration. If we go back to the car manufacturing comparison, the first Model T rolled off the Ford assembly line in 1908 and ushered in an era of affordable automobiles. Yet even now, over 100 years later, while the vast majority of cars are built in highly efficient factories, there are still times when it is appropriate to build a car by hand. The most common reasons for custom-built cars are to meet special high-performance requirements such as for racing cars, prototypes to test new ideas, or simply for aesthetic and artistic motivations.
There are similar justifications for custom-built integrations in the software world. The first time an enterprise has requirements for extremely high-performance response, massive scalable throughput, special security controls, or other innovations, a custom solution is often the only way to achieve the desired results. If the solution requires a truly innovative approach, it may indeed warrant building a prototype or two to test the ideas and find out which way works best before building the production implementation.
The point is that factories are most effective when there is a clear pattern. Architecture and integration teams should be prepared to build custom solutions when required, but they should also always be on the lookout for new patterns that can be standardized and addressed with reusable components and efficient processes.
Is there a place in the integration world for creative approaches strictly for aesthetic reasons? It’s hard to imagine a business sponsor funding such an initiative, but in any event, we will leave it to you to answer this question.
The Integration Factory as an Element of an ICC
In Chapter 1 we wrote about how the Integration Competency Center needs to be viewed as a business serving an internal market. At the highest level, this unit governs itself by measuring the value it provides to the enterprise from the perspective of the internal customers. There are several practices that can be established for how the ICC initiates engagements so that customer pull triggers factory flow. Or, as stated in Lean Thinking, “It is because the ability to design, schedule, and make exactly what the customer wants just when the customer wants it means you can throw away the sales forecast and simply make what the customers actually tell you they need. That is, you can let the customer pull the product from you as needed rather than pushing products, often unwanted, onto the customer.”2
2. James P. Womack and Daniel T. Jones, Lean Thinking: Banish Waste and Create Wealth in Your Corporation (Simon & Schuster, 1996), p. 24.
The ICC must have a way to initiate integration projects in a repeatable, structured way. This can involve creating a structured project document that outlines who the primary contact is, who the business sponsor is, what the primary objective of the initiative is, and a preliminary idea of what the scope might look like at a very high level. The document should specify only what the business is looking for, who the key people are on the business side, what the important systems are, when the business needs this, what some of the basic business requirements are for integration (real-time synchronization or not), and a rough estimate of whether the effort required is easy, medium, or difficult. The speed and effectiveness with which the ICC responds to the business requirements will be measured from the perspective of the customer. For that reason, it is useful to understand why the customer needs the problem solved. From that, it is usually possible to find a customer-driven metric that can galvanize the ICC effort and allow measurement of the end-to-end value chain from the end-customer perspective rather than the internal factory/ICC perspective.
The resulting project initiation document is stored in a structured document (either an online form or a structured document that can be loaded into the metadata repository). Ultimately, all of the documents that describe the activities within the factory will be tied together, from the requirements documents to the design documents to the implementation artifacts to the testing plans and test code to the deployment scripts, history, and change management history. This requires a metadata repository to house and relate this information, which serves as the live documentation describing all information sources and exchanges.
Automating processes is not just the use of computers to do things automatically for people. Ultimately, it is the ability to tie together related information in a way that allows humans to quickly and efficiently grasp tremendous complexity. Our goal is to transform the integration hairball into a flexible integration system that quickly adapts to the needs of the business and technical dependencies as they change.
At this point, no decisions have been made about how the project should get done, which team is appropriate, or what the priority is. Effectively, the customer has stated what type of vehicle is wanted: a “small blue truck” or a “red four-door sedan.” This describes a car style and the basics that can get passed to the appropriate “factory” within the ICC business for manufacturing.
Integration Factory Variants
The ICC and its factory processes can be designed to execute one or more styles of integration projects. Examples of these projects include
• Data warehousing
• Master data management
• Straight-through process integration
• Application retirement
• Data migration
• Business-to-business (B2B) or supply chain integration
Each of these project styles (this is not an exhaustive list and your terminology may be different) has a unique methodology. Once the ICC has identified a particular project and the people are mobilized to execute on this project, the factory for that project kicks into gear.
For each project style, particular life-cycle stages, roles, and artifacts are mobilized. While at the highest level traditional projects go through requirements, design, implementation, testing, deployment, and change control processes, the details of the roles, the way they interact, and the detailed artifacts that get produced are specific to the style of “vehicle” being produced. Furthermore, some projects may use agile methods, which involve less formal documentation and multiple iterations before the final solution emerges. Therefore, each factory, while having a similar shape and construction, must be uniquely styled to the type of products it is producing and the process for handling material and information flows.
Let’s take one factory as an example. A leading mortgage company, prior to the subprime mortgage meltdown, had a data migration factory that had eliminated waste in the process of merging acquired companies’ data into its systems. The company was purchasing small mortgage companies at a rate of approximately two per month between 2005 and 2007. When a company got purchased, the data migration factory kicked into gear. The factory had a detailed methodology and trained personnel who followed well-documented steps to achieve virtually waste-free integration of the purchased company’s data with that of the parent company.
The mortgage company defined specific target models for each of the different critical data subject areas it required. It mapped the purchased company’s source data to these target models, kept the target model definitions constant, and transformed and cleansed the source data where necessary. Business people were directly involved in the resolution of data issues in the process. And because the processes were so explicitly defined, multiple migrations could be in motion simultaneously, all working efficiently toward completion.
While this example leaves room for further automation, the company was indeed taking advantage of specialization and was using some factory concepts to increase the speed and efficiency with which acquired businesses could be integrated. It had purchased specific tools and developed people for the purpose of eliminating waste from the data migration process. Migrations were not seen as one-off projects that appeared and disappeared as if the current merger were the last one the company would ever go through. The company recognized that it had a “growth through acquisition” strategy and adjusted and invested in its IT model to meet that business strategy.
To generalize this further, it may be easier to think about these Integration Factories as operating in a hierarchy. The top level in this example is the data migration factory that applies Lean concepts to the repeatable process of rationalizing applications in acquired companies. This factory in turn takes advantage of lower-level factories that apply Lean concepts to repeatable processes such as data mapping, data cleansing, data replication, and application retirement.
Taking this approach provides several benefits. Quality is built into the integration logic because the logic follows patterns that have been proven to work best over time. Change management is easier because the integration logic looks familiar to developers. Productivity is improved because a great deal of manual labor is eliminated. An Integration Factory that takes automation down to the level of maintaining assembly lines for mass-customizing common integration patterns is taking one of the most effective Lean approaches to reducing waste and delivering better, faster, and cheaper to the customer.
How Does the Integration Factory Work?
There are several perspectives from which the factory workings could be described. In this section we will describe the factory in three ways: a project workflow scenario, work-group interactions, and a factory tools perspective.
Project Workflow Scenario
Figure 3.2 shows an overview of a process for creating middleware interfaces in support of an IT project. Note that the workflow steps and activities will change for different styles of integration, but nonetheless this should be a helpful representation of a typical process.
Figure 3.2 Example factory process
In step 1, a customer (which may be a project team that requires services from the factory) requests one or more integrations during the initiation stage of a project, which generally includes the scoping and business justification activities—in short, before the full project funding is approved. At this stage in the project life cycle, the requirements are high-level functional in nature; the low-level detailed requirements and technical specifications are determined in subsequent stages. Note that even in this very early, often exploratory, stage of a project, the team interacts with the Integration Factory using standardized tools and templates. This facilitates clear communication and sets the stage to initiate the flow of information between teams.
Step 2 formalizes the integration scope in the form of a statement of work (SOW) or equivalent. In short, this document represents the promise the Integration Factory makes to the customer. It is hard to overestimate the importance of this step since much of the credibility, and the ultimate success, of the factory arises from effectively making and keeping promises to customers. Chapter 11, Financial Management, discusses techniques for scoping the effort and cost of integrations based only on a high-level understanding of requirements.
Step 3 involves developing detailed (low-level) functional requirements for the integration and also determining nonfunctional requirements related to specific technology platforms and physical constraints.
Step 4 is where the integration effort transitions from problem-solving activities to assembly and construction activities. For routine integrations, this process may be very quick since it involves simply configuring existing reusable objects or infrastructure elements. However, if the requirements cannot be met by the current portfolio of reusable parts, the integration team will need to either construct the new elements or purchase them from a supplier. The components shown in the figure are representative of the types that may be involved in assembling a real-time integration. A mature Integration Factory may have hundreds of reusable components.
Step 5 comprises the final stages of assembly, quality assurance, and deployment activities. Once the integration elements become operational in a production environment, the factory teams may be called upon to address defects or to create a new variant of the integration components in response to changes.
Large-scale factories may have separate assembly lines for new integrations and modifications to existing interfaces. If the requested interface is new, the interface is designed, developed, and tested. Design and development typically occur at the factory site, which may be remote from the customer site. Testing can occur either at the factory or at the customer’s location, but in any event quality control is governed by the factory processes. If the requested integration is a modification to an existing interface, the related code is extracted from a version control tool, changed, and regression-tested to compare the results to those of the original tests.
Work-Group Integrations
The factory (people, tools, and materials) for a given enterprise can be at one physical location, or it can be virtual with different activities and process steps performed anywhere in the world. This high degree of flexibility is enabled by modern network technology, structured repositories, and integrated workflow tools.
Figure 3.3 shows the primary flow of materials (large block arrows), beginning with requirements and the flow of information (thin lines) between teams inside and outside the Integration Factory. Note that this is a generalization and abstraction of a real-world factory for the purpose of making several key points related to how Integration Factories work. An actual factory may have more or fewer teams, subject to its scope of integration services and volume of work. Furthermore, a “team” may be represented by a group of people, one person, or, in small factories, even a fraction of a person who fulfills several roles. Nonetheless, the key roles and their interactions are instructional.
Figure 3.3 Integration team interactions
Reusable Components Team
The flow of material in an Integration Factory begins with the concept of mass customization, that is, standardized components that can be configured or assembled with other standard components to meet the unique needs of a given set of integration requirements. It is critical that this team be separate from the project or deployment teams so that it can retain a long-term rather than point-in-time perspective on requirements. It is also critical that this team synthesize and integrate the needs of different projects to create components that are standard where appropriate and configurable as needed.
Reusable objects are made available to the factory assembly line through a formal release process. Versioning of objects is essential, since reusable objects are often refactored to accommodate new functions. Releases may come in large batches in situations where a collection of objects that are dependent on each other all need to change at the same time. In general, however, the recommendation is to apply the Lean concept of small incremental batches to the release process, ideally, batches of one.
Assembly Cells
The distinction between “assembly” and “development” is fuzzy at best. As software development tools have continued to mature and morph over the years, the line where development stops and configuration begins has continued to move. In any event, the intent with the factory is to truly achieve a metadata-driven assembly process rather than a custom development process. The output of this group in terms of the flow of materials is the collection of objects and code elements that represent the integration points as required by a given project effort.
A factory assembly process may have only one cell, but as the scope and scale of a factory increase, the Lean concept of work cells all operating according to a defined pace (Takt time) begins to pay off. Some of the major sub-activities associated with the core factory line are given below. Depending on the scale and workload associated with each activity, these activities should be grouped to optimize staff skills, workload, and cycle time for each cell.
• Canonical mapping
• Message schema definition
• Integration design
• Component assembly and configuration
• Unit testing
• Performance testing
• As-built metadata documentation
Deployment Teams
Figure 3.3 shows multiple deployment teams—typically in a one-to-one alignment with the external project teams. In effect, the deployment teams are part of the project teams but are shown as separate entities since they operate within the factory context. One of the key responsibilities of these teams in the information flows is to supplement the functional requirements (what) from the project teams with technical requirements (how) related to physical implementation details.
The level of effort for managing the integration elements of a project is generally less demanding than the management effort for the overall project. For this reason we often find that one integration project manager can manage several deployment teams at the same time.
To continue the flow of materials, the deployment teams accept assembled integration elements from the assembly team and deliver high-quality tested integration elements to the project teams.
Project Teams
The project teams in this model are exactly what you would expect—cross-functional teams led by a project manager who is focused on business process changes and application system implementations. These project teams are in effect the key customers of the development Integration Factory. The integration requirements are the pull signal from the customer that signals demand for integration elements and begins the factory flow.
Production Operations
The production operations teams are the final step in the flow of materials. The operations function may be performed by one team or multiple teams. Common scenarios for multiple operational groups include enterprises with separate operations for different countries, supply chain integration, or cloud integration, which includes externally hosted applications.
Furthermore, depending on how the ICC is organized, the factory itself may have operational responsibility for selected portions of the IT infrastructure—typically the middleware systems such as data integration hubs or enterprise service buses (ESBs). In any event, the factory staff may be called upon to support production operations for incident management and problem resolution.
Finally, the production groups provide operational metrics, which are captured in a metadata repository.
Metadata Management
The metadata management team’s role is crucial in the information flows between all teams; hence this team is an enabler for the efficient operation of the overall factory processes. If we accept the old adage that “knowledge is power,” the metadata team has the most power and influence in a factory operation.
The metadata team responsibilities generally include defining the information metamodel, developing interfaces/integrations between various metadata sources and repositories, optimizing the processes to maintain a high level of data quality in the repositories, enabling access to the repositories by different users through a user interface, and developing standard or custom reports.
Finally, this team may go by different names. The term metadata is somewhat techie and may not resonate with customers of the factory or with business managers. Alternative names for this could be the “Data Exchange Administration Team” or “Knowledge Base Management.”
Governance Committees
Governance committees play an essential role in the processes, policies, standards, organization, and technologies required for a sustainable and Lean factory. Figure 3.3 lists four distinct domains that typically require a formal level of management control:
1. Data governance, to ensure the availability, accessibility, quality, consistency, auditability, and security of data in a company or institution
2. Project governance, to evaluate, prioritize, approve, enable, control, monitor, and steer business and IT initiatives
3. Architecture governance, to provide a comprehensive picture of the business (functions, processes, systems, technology, etc.) and its transformation from the current state to a defined future state
4. Operations governance, to control the pace of changes to the production environment and monitor the quality of the changes
Whether an enterprise has one or many committees, they are an essential ingredient of a sustainable integration strategy, and the ICC or Integration Factory must be an active participant, or even a leader, in them.
Factory Tools
We all have images of what tools and machines in a traditional widget factory look like: looms for weaving cloth, boilers for brewing beer, robots for assembling cars, saws for milling wood, and so on. An Integration Factory also needs tools, although they may be harder to visualize since we are talking about machines that exist in the virtual world of software.
Factory tools may be purchased from a number of suppliers or custom-developed in-house. As we have stated previously, developing and maintaining tools is a big part of the job that factory staff need to perform. Even when one buys integration tools, there is almost always an opportunity to improve efficiency by enhancing them. Factories that produce physical products do this all the time. For example, they might buy one machine that molds pieces of wood to a particular shape, buy another machine that cuts them to specified lengths, and then custom-build a conveyer system to automatically feed the materials from one to the other. By the same token, Integration Factory staff should be prepared to build custom harnesses or interfaces between factory tools, for example, to move metadata from a data quality assessment tool to a data-cleansing or workflow tool, or to automatically move code objects from development to test to production.
Figure 3.4 is a block diagram showing a factory knowledge base. The knowledge base includes a repository and a collection of tools to facilitate the information flows in the development factory. The knowledge base functions as a repository of information gathered throughout the development life cycle. For example, information gathered in the engagement tool is provided to and stored in the repository. This information is typically the high-level requirements of the interface provided to the factory by the customer. It can include such high-level requests as when the interface is needed, how many interfaces are required, and the like. Information from the engagement tool is provided to the designer, as is further information provided via the requirements tool. The requirements tool asks for more detailed and technical information about the interface requirements than the engagement tool and is also captured and stored in the repository.
Figure 3.4 Development factory information flow tools and repository
The repository can also generate a set of reports for insight into the factory process in connection with a tool having information related to the factory process. The reports in the set are customized to the intended audience and can include a support view, a test team view, a build team view, a design team view, and a customer view. These reports provide audience-specific information to aid in the creation of the integration.
Note that there are two central elements to the factory tools: the repository (or metadata repository), which is the permanent knowledge base for the factory information and material flows; and the scheduler tool, which controls the workflow according to customer pull demands and factory Takt time. Not only does the scheduler tool provide factory management with a picture of what is happening in the factory at any given moment (which, if you recall, may be distributed to all corners of the world), but it also provides information about upcoming activities. For example, the scheduler tool can provide a projection of how many data-mapping analysts will be needed next week or how many integration testers will be required next month on a given day. This forward-looking perspective is essential in order to scale factory cells (teams) up or down in support of production leveling and to maintain factory flow.
Integration Factories as Self-Service ICCs
When we wrote our first book on Integration Competency Centers, we created a graphic that showed a progression of models or approaches into which ICCs generally fell, shown in Figure 3.5.
Figure 3.5 ICC organizational models
At the far right of this figure we describe a “self-service” ICC model, where highly mature teams automate many of their processes and dynamically generate (or mass-customize) a great deal of their integration logic, and the customers of the ICC are able to self-serve many of their integration needs without concern for the detailed behind-the-scenes operations. This effectively describes what we now call an Integration Factory.
Several examples of Integration Factories, or self-service ICCs, are well established. Some vendors have created integration-as-a-service offerings in the cloud that provide perfect examples of Integration Factories. These services allow a customer to directly select an assembly line (data replication of Salesforce.com data, for example, or data synchronization between heterogeneous relational databases) and walk through the process of selecting the data to be integrated, defining any special transformation logic that the data requires, scheduling when the integration should happen, and so on. The patterns behind the assembly line take the high-level definition of what the customer wants and translate that into how the integration gets executed. These services allow the self-service of standard integrations directly by business users.
Some ICCs today are using the “data synchronization between relational databases” service for two different use cases. The first use case is where the ICC “publishes” data as sources for the business, allowing the business to define its own extracts or pulls from this data. At many organizations, the requests for data from the ICC exceed the ICC’s capacity to deliver to the business fast enough, thereby creating large (and wasteful) backlogs of work. The self-service capability for the business provides an elegant resolution to the problem.
Sometimes this self-service capability raises concerns among IT leaders. “The users are coming in and doing integration by themselves? What about governance? What about best practices?” Ironically, self-service capabilities provide better governance and adherence to best practices since in the absence of an easy-to-use tool that provides users with what they need, the users will generally resort to creative desktop-based solutions that have few controls. The integration-as-a-service approach, on the other hand, audits and logs absolutely all aspects of what the customers did, when they did it, where they moved the data, when the data was moved, and so on. And all generated integrations followed patterns optimally generated based on best practices.
The second use case allows data analysts to prototype the source-to-target mapping and transformation specifications, as well as execute those specifications to test whether they work as expected. At many organizations, analysts specify an integration using an Excel spreadsheet, which is then handed off to developers as a document. This use case allows a similar capability but with the added ability to test the results. Once the analyst is satisfied, he or she passes the specification to the ICC developers to augment and prepare the integration for production, collaborating with the original data analyst. In both use cases, significant waste is eliminated from existing processes.
In summary, Integration Factories are emerging faster than most people realize. While these integration services in the cloud today (at the time of this writing) may handle rudimentary integration problems, they add value and provide a model for what is possible in the future. Real-world complexity that deals with the unique realities of complex enterprises requires a more challenging journey toward on-premises Integration Factories. The next part of this book is devoted to helping organizations learn about Lean principles in taking this journey.