Chapter Five. Continuously Improve
Lean companies work to develop and establish value with continuous improvement of their value stream flow, not by cost reduction practices.
Joe Stenzel1
1. Joe Stenzel, Lean Accounting: Best Practices for Sustainable Integration (John Wiley & Sons, 2007), Kindle loc. 981–82.
A core principle of Lean is continuous improvement through experimentation and learning. The general notion is that there isn’t a “perfect” way to do something. Rather, seeking perfection is an ongoing process, since there are always opportunities for improvement that can be uncovered using scientific disciplines. As described by Mary and Tom Poppendieck, “. . . any development process that deals with a changing environment should be an empirical process, because it provides the best known approach for adapting to change.”2 The scientific method in general follows these steps:
2. Mary and Tom Poppendieck, Implementing Lean Software Development: From Concept to Cash (Addison-Wesley, 2006), p. 21.
1. Observe and describe a phenomenon or group of phenomena.
2. Formulate a hypothesis to explain the phenomena.
3. Use the hypothesis to predict something—the existence of other phenomena or the results of new observations.
4. Perform experiments to see if the predictions hold up.
5. If the experiments bear out the hypothesis, it may be regarded as a theory or rule.
6. If the experiments do not bear out the hypothesis, it must be rejected or modified.
This breaks down to the more simplified four-step method prescribed by Deming: Plan, Do, Check, and Act.
Continuous Learning and Knowledge Management
Most Lean practices focus on building knowledge, but we prefer to put the emphasis on sustaining knowledge, which incorporates the idea of ongoing maintenance and institutionalization of insights that drive continuous improvement. There a number of ways the principle of continuous learning can be applied to integration activities.
First, standards should be challenged and improved. A corollary to this principle is Integration Law #3: “There are no universal standards.”3 Standards are in fact essential for Lean Integration, but it is a mistake to consider them to be fixed, static, unchangeable, and applicable in all situations. Industry standards such as COBOL, TCP/IP, and HTTP (to name just a few) have evolved significantly over the years. Enterprise integration standards also need to be sustained and to evolve over time. This principle is reinforced by Integration Law #2: “There is no end state.”
3. John Schmidt, “The EAI Laws,” EAI Journal (July 2002). Also see Appendix B.
Second, Lean practices put a strong emphasis on scientific methods. One of the reasons this is so important in the integration arena is that we are often faced with the challenge of achieving alignment across multiple independent teams or organizational functions. Gaining agreement across groups that don’t usually work together is hard. In the absence of data, all you have are opinions, and gaining agreement between people with different opinions that are filtered by paradigm-colored glasses is virtually impossible without objective data. A particularly useful, and practical, technique for data-driven cross-team problem solving is management by fact (MBF).
Third, integration dependencies between components should be maintained in a structured, and searchable, repository rather than in static, unstructured formats. Your first reaction might be that this is unnecessary since your organization already has a mandate that each project develop detailed documentation about all information exchanges between components. In that case ask yourself these questions:
1. Does the documentation reflect what was deployed to production, including changes made after the design was completed?
2. Would a typical business analyst, designer, or developer be able to find the documentation two years later? Five years later? What if the original author of the documents is no longer with the company?
3. If the documentation can be found, would it be understandable? In other words, are the graphical notation conventions the same for all documentation, and are they based on a common glossary and taxonomy for data objects?
4. If maintenance changes were made to production after the project was deployed, was the documentation updated to reflect the changes?
If you answered yes to all four questions, you already have the necessary discipline for this particular aspect of sustaining integration knowledge. If you answered no to one or more questions, you are wasting time every time you need to change something and need to re-create yet another version of a custom integration document.
The remainder of this chapter is a rather detailed case study. We felt the best way to reinforce the principle of continuous improvement would be through a demonstrable real-life example. The case study actually touches on many Lean principles, including focus on the customer, team empowerment, mass customization, and automation. As you read it, take note of how all these elements were brought together with the financial management competency and fact-based metrics to align the organization and establish a sustainable integration strategy.
Case Study: Continuous Improvement at Clicks-and-Bricks
In early 2005, the chief technology officer at Clicks-and-Bricks discussed one of his major concerns. Maintenance costs were high and increasing steadily, and the complexity of the information exchanges between applications was becoming increasingly worrisome because of the quantity of integration points and the high degree of variation in tools, protocols, and standards. In the words of the CTO, “There has to be a better way!”
The complexity of the Clicks-and-Bricks environment had started to come into focus a few years earlier. The company had undertaken a major effort to inventory all the application systems and integration points as part of a major business transformation. At the time there were no plans to sustain the inventory in a structured repository, but nonetheless the team that did the work produced a very comprehensive picture of the application landscape. Every organization has its own version of such a picture, which illustrates the “integration hairball”—the unmanaged complexity that results from years of independent efforts to implement point-to-point integrations.
In any event, the comments from the CTO launched the ICC on a path to develop a formal business case and gain broad-based management agreement to move forward with a solution. The integration challenges were not new, and efforts had started one year earlier to establish the ICC and build a reusable framework that would serve as a standard for real-time integrations between applications. Management’s perception was that progress was slow, and there was great resistance from application groups and project teams to adopting the ICC standards, so the company brought in a new director to head up the group.
The first activity by the new director was to assess the capabilities of the ICC staff, their role in the organization, and how the group was perceived by internal stakeholders. The ten staff members on the ICC team at the time were very knowledgeable, competent, and committed individuals who worked well together. Furthermore, they had a deep understanding of the technical challenges that the company was struggling with, and they knew exactly how to solve them. In fact, the team had already architected and designed a reusable software framework as the basis of a standardized real-time integration infrastructure that would reduce costs, improve operations, and simplify maintenance.
While the ICC team had the right staff, an accurate analysis of the problem, and a well-designed solution, the group was running into a great deal of resistance from project teams that were accustomed to doing things their own way. The project architects and lead developers already had a set of tools and methods that they were comfortable with, and they were skeptical of a new approach. Project managers and project sponsors were afraid to rely on another team to perform some of the project deliverables; they felt the central team would slow down the project, add unnecessary bureaucracy, increase costs, and reduce quality. And across the IT organization, senior management support of the ICC concept was spotty; some executives strongly supported the approach while others were doubtful.
So while the ICC had the right staff and the right solution, it was missing a few things: It did not have strong leadership and management support for the ICC strategy; the group was using the “stick” rather than the “carrot” to gain adoption of standards; it did not have an effective approach to engaging project teams; and when the team did get involved in a project, the result was generally higher costs and slower implementation since the new integration framework was not fully developed. In short, the ICC was inwardly focused and trying to force a solution on others that was not yet better, faster, or cheaper than the alternatives.
The first step that the team took to turn things around was to build a business case to gain broad-based senior management support for a formal Integration Factory. An analysis of existing projects at Clicks-and-Bricks showed that the average cost of designing, building, testing, and deploying a new integration between two applications was $30,000 and took 30 to 40 days. This was the first time anyone had done the analysis, and management was shocked at the high cost. The factors contributing to the high cost included
• Poor or nonexistent documentation of existing interfaces; each new integration required a bottom-up analysis, discovery, and problem definition
• No coding or software standards, which prevented reuse of prior work and often led to disagreements among newly formed teams about what tools or standards to adopt
• Highly complex data and process interactions that could not be fully understood or specified in the analysis phase; these were left to the software engineers to figure out during the development phase, which usually resulted in a high degree of rework during integration testing.
In short, each integration point was a “work of art” that required lots of meetings, debate, and rework.
The business case showed that the variable cost per interface could be reduced from $30,000 to $10,000 if an investment was made to finish constructing the integration framework, install a metadata repository, and implement an integrated workflow management capability. The cost of the investment was $3 million, and at the time Clicks-and-Bricks was building 200 to 300 new integrations per year. Table 5.1 shows a five-year projection of low-end and high-end annual costs using the baseline analysis. Note that this is a very simple model that assumes no cost improvements due to efficiencies or any cost increases due to inflation or increasing complexity. There were valid arguments for both cases, so the team decided to simply project the status quo.
Table 5.1 Integration Development Cost Projections Based on Custom Project Silo Approach
In a nutshell, the business case was to invest $3 million in order to save $4 to $6 million per year in development costs as well as reduce maintenance costs, improve quality, and accelerate implementation of new integrations. Table 5.2 shows the investment costs in year 1 and the annual low-end and high-end savings from the resultant reduced variable cost. The worst-case net present value based on a 10 percent discount rate was $12.4 million with a return on investment of 415 percent.
Table 5.2 Integration Development Cost Projections Based on Integration Factory
This business case is a very simple model that does not include any maintenance savings or quantified benefits from cycle-time reductions. These other areas were treated as “soft benefits” from the perspective of the business case since the business case was already strong based solely on development cost savings. If the ROI had been marginal, the team could have performed additional analysis to quantify the maintenance and cycle-time benefits.
The point of all this is that the business case provided a simple, clear, and unambiguous value proposition for centralizing and standardizing the integration development process. More important, it established the business metrics, including baseline measures and target goals, that served as the foundation for continuous improvement. In any event, the investment funding was secured, and the CIO, CTO, and other senior IT executives provided strong support to proceed with the plan.
The second step that the ICC team took was to improve the customer engagement process—specifically the information management task as described by Womack and Jones in Lean Thinking.4 The “customers” in this case were the internal project teams that had specific requirements for new or modified interfaces between applications or databases. The project roles that the ICC focused on included project sponsor, project manager, project architect, lead developer, and tester. For the ICC to be successful, it would need a customer engagement process whereby the staff in these roles felt they were receiving value from the ICC and that they were in control of the process. The solution to achieve this was twofold.
4. James P. Womack and Daniel T. Jones, Lean Thinking: Banish Waste and Create Wealth in Your Corporation (Simon & Schuster, 1996).
First, the team developed a role-based Web application that project teams could use to initiate an integration project, specify requirements, and track progress. This represented a paradigm shift in how project teams interacted with central support groups. The process previously had been either informal (hallway discussions, a visit to a coworker’s cubicle to request support, etc.) or formal (queue-based requests sent by email or a request form to the support group’s generic mailbox). The traditional methods meant problems and frustrations for the customer—misunderstandings from informal communications, several days’ delay waiting for a response to formal email requests, and no ability to track the progress or status of specific requests. This last point was often the most frustrating. For example, the project manager might call the support administrator after a two-day wait with no response and ask about the status of the request. The administrator would say, “Oh, yes, I got your request and forwarded it to Jane for review.” The project manager would then call Jane and possibly after a rally of telephone tag get the response “Oh, yes, the request is fine. We can handle it but Frank needs to assign the staff resources so I forwarded your email to him.” The project manager would track down Frank and might get a response like “What request? I don’t recall seeing anything from you,” which would send the project manager back to Jane. And the merry-go-round kept on spinning.
The Integration Factory approach, however, solved the problem by providing project team members with a customized user interface that allowed them to provide the necessary information to the ICC and to track the progress of their requests. For example, if a project team was relying on the ICC to design and build ten integrations for a given project, the project manager could log in to the Web application and review a dashboard that showed the status of each integration point, including current status and projected dates for subsequent phases. If there was an issue with any item, the project manager could quickly identify the right contact person and thereby clarify the issue with one phone call. In short, the customers of the ICC felt as if they had a direct and efficient communication mechanism by which to interact with the group.
The second element of the solution for an effective engagement process was to provide a simple pricing model that could be applied very early in the life of a project, before the high-level design was done and even before the budget was finalized. The solution the team came up with was to define five categories of interfaces (simple, standard, medium, complex, and custom) and to determine average costs for the first four. Custom interfaces, on the other hand, had unique characteristics and high variation in cost, and they represented less than 5 percent of the total new integration development; hence they needed to be sized and estimated on a case-by-case basis.
For example, a simple interface might cost $5,000 (costs changed over time, as we shall see in this case study) and was defined as one where the adapters for extracting and loading the data were similar to ones that had already been implemented, which meant they could easily be reused; data transformations were not required or the messages were already in an appropriate XML format; and the interchange paradigm was either a straightforward point-to-point or file transfer protocol. A standard interface, on the other hand, might cost $10,000 and include simple business rules for extracting and loading the data (such as filters, default values, etc.), routine data transformations (such as translation into standard formats, date validations, etc.), and an interchange paradigm that was either point-to-point or publish/subscribe with or without acknowledgment.
The value that this pricing model provided is that project teams could determine the cost of integration by themselves. There was no need to send a request to a central group, wait for several days, have extensive meetings, play telephone tag, and all the other muda activities that are so common in many centralized shared-services groups. Furthermore, the commitment from the ICC was that the prices were a “firm fixed cost”; in other words, regardless of whether or not the actual effort was greater or smaller than the estimated effort, the cost that was charged to the project was fixed. The business sponsors and project managers loved this aspect. In other words, a fixed-price commitment was a huge part of the value that the ICC provided, since it eliminated one of the major risks in project cost overruns. From the ICC perspective, some interfaces did indeed end up costing more than the project was charged and others less, but on average all of the integration costs were recovered from project chargebacks. This is yet another example of how value can be provided to customers by taking a holistic perspective.
As mentioned earlier, the ICC first tried to use the “stick” approach to gain adoption of standards. It didn’t work. Clicks-and-Bricks is an organization with a dynamic, entrepreneurial culture where project teams and individuals are fiercely proud of their independence and relate the success of the enterprise to their personal innovations. When the ICC took the position that “you must do it our way because we have the mandate and this standard approach is in the best interests of the enterprise,” it simply ran into brick walls of resistance. It took some time (over one year) to switch the team to a “carrot” approach, but in the end it was much more effective.
The carrot approach is based on the idea that you can get people to change and adopt a standard by making it better, faster, and cheaper rather than by mandate. Several items that have already been discussed in this case were key ingredients in enabling the change. The streamlined engagement process and the investment in a reusable framework were both essential elements, but they were not sufficient. The ICC team also needed to be pragmatic and not get locked into idealistic “my way or the highway” positions. Once again, the ICC team needed to keep in mind that it was serving the customer, and it needed to understand the customer’s perspective when solving problems.
One of the best tools the ICC team used to avoid battles with architects and technical leads on project teams was to take a long-term perspective. For example, the ICC would always try to convince the project team to use the standard framework and integration protocols, but if the project architects dug in their heels because it would mean displacing their favorite technology or because they had a technical or risk concern, the ICC would look at it this way: “Fine. We’ll do it your way on this project. And it’s OK because we know this interface will need to be rebuilt again in a few years (because things are constantly changing), so we will apply the ICC standard next time around.” It is amazing how many of the technology-war battles (muda) between technical staff simply disappear when you take a five- to ten-year perspective.
The final ingredient in transforming the Clicks-and-Bricks integration hairball into an efficient integration system was the Lean principle of continuous improvement. As the Integration Factory business case demonstrates, there was a compelling case for the investment to drive the variable cost per interface to $10,000. But if the team had stopped making improvements once this goal was reached, it would not take long for expectations to be reset around this new standard, and project teams (and management) would no longer see value in the ICC. After a year or two of steady-state costs of $10,000 per interface, it would be easy for management to rationalize that the ICC was no longer needed and the organization might be better off without the central team. Indeed, if the group stopped improving, disbanding it probably would be the best decision. In other words, a one-time improvement in productivity is not sustainable; you need to continuously improve.
In order to institutionalize the continuous improvement mind-set in the team, the ICC managers were given a challenge: Reduce the cost per interface by 50 percent each year for the next three years. Since the initial investment would result in a target cost per interface of $10,000, this meant that one year later the average should be $5,000, two years later it should be $2,500, and three years later it should be $1,250. This goal was radical, and everyone initially considered it to be impossible. But the team actually achieved the goal! How did they do it?
The key factors were to have a well-defined end-to-end process, to measure each step in the process in excruciating detail, and to regularly review the process (kaizen) to look for opportunities for efficiency gains. In particular, every manual activity was viewed as potential muda and therefore was a candidate for elimination through automation.
A key aspect of the detailed metrics tracking is that it was not used for individual incentives or rewards. For example, the team tracked the number of defects found in integration testing and the amount of resultant rework effort, but developers were not rewarded or punished if they missed or exceeded the targets. The metrics were used to bring management attention to problem areas or potential improvements so that the entire team could help solve them. The team was, however, rewarded for achieving the overall end-to-end goals for cost per interface and lead-time reductions.
The ICC team made a point of communicating the cost per interface to all project teams and senior IT management on a quarterly basis. These, and other key metrics, of the ICC were one of the ways the team was able to demonstrate very clearly the continuous improvement results and therefore create a sustainable business model. Table 5.3 shows the actual results of the business case three years after it was approved. Note that the ROI was now up to 934 percent, which was double that in the original plan!
Table 5.3 Business Case Results after Three Years of Integration Factory Operation
In terms of overall outcomes, the financial results from applying customer focus, mapping the end-to-end value chain, eliminating waste, and continuously improving the process are impressive. The net present value of positive cash flows over five years of the original $3 million investment was up to $28 million after three years. The average cost per interface went from $30,000 to $2,500 after three years, and over 50 percent of the interfaces fell into the “simple” category for which the cost was only $1,000 (which included costs for design, development, assembly, testing, deployment, and one month of warranty support in production). In addition, the average time to build an interface fell from 30 days to fewer than 5 days, and many of the simple interfaces were delivered within 1 day.
Furthermore, the size of the ICC team grew from the initial team of 10 to 180 after three years (approximately two-thirds of the staff were contractors and 60 percent of those were offshore). Despite the tremendous labor productivity improvements, no employees were laid off. Instead, the scope and responsibilities of the ICC group expanded in order to leverage the effective model in other integration areas. The original scope with 10 staff included only the real-time message-based integration for a subset of applications. After three years the ICC team was responsible for 100 percent of the real-time integration, all the database-to-database integration, most of the external business-to-business integration, all the database administrators, the LDAP security authentication system, and the metadata repository.