The Standish Group survey included large, medium, and small companies across major industry segments: banking, securities, manufacturing, retail, wholesale, health care, insurance, services, and local, state, and federal organizations (365 companies in total). Over a 10-year period, it studied over 40,000 projects. In addition, it used focus groups and personal interviews to provide a qualitative context for the survey results.

The CHAOS study classified projects into three categories:

Figure 1-1 shows the percent of projects in each of these categories over a 10-year period.

Figure 1-1. CHAOS study data

As you can see, the success rate improved over time, but at a glacial pace: 1.3% a year. At this rate, successful projects won’t even break the 50% mark until the year 2020!

In fairness, some experts disagree with the definition of success used by the CHAOS study. They point out that many projects that have been classified as challenged have gone on to deliver very successful products. However, even if you take this into account, the success rate still leaves a lot of room for improvement.

So the question is: why is the failure rate so high? A large part of the blame can be traced back to the widespread adoption of the Waterfall method.

The Waterfall model of software development divides development into a set of distinct phases that are performed sequentially: requirements, design, implementation, testing, deployment, and maintenance (see Figure 1-2). Development is seen as flowing steadily downward (like a waterfall) through these phases. Each phase has a beginning and an end, and once you move onto the next phase, you never go back (just as water does not flow uphill).

Figure 1-2. Waterfall model

The Waterfall method has an appealing air of simplicity. Managers like the series of fixed milestones that seem to make it easy to track a project’s progress. However, this manageability is an illusion because this model does not allow for change.

Software development is a highly dynamic process and change is inevitable. During implementation you’ll find problems with the design. Customers don’t know precisely what they want ahead of time, and that will change the requirements.

Studies show that the Waterfall method is positively correlated with reduced productivity, increased defect rates, and project failure. Why, then, was the Waterfall method so widely promoted, and how did it become so entrenched in the face of so much evidence that it did not work?

The Agile Manifesto (formally called the Manifesto for Agile Software Development) is a model of simplicity. Written by Kent Beck et al., it eloquently states the four core values that form the philosophical bedrock of all Agile methodologies:

The wording of the Agile Manifesto was carefully crafted to communicate the intended message. By saying, “We are uncovering better ways of developing software,” they showed that they didn’t have all the answers and were still on a learning journey of discovery. Just as important was the next part, “by doing it and helping others do it,” which meant that they were actively engaged in using these “better ways” (no ivory-tower theoretical advice here) and were sharing what they learned with others (not telling people what to do).

Each of the four value statements in the Agile Manifesto lists two things and states that both items are important, but the first item is higher priority. Martin Fowler and Jim Highsmith described this very succinctly in their August 2001 article in Dr. Dobb’s Journal titled “The Agile Manifesto”:

Plans are important, but they shouldn’t be rigid, unchanging plans. The ability to respond to changes is critical to the success of most software development projects.

In addition to enumerating the four Agile values, the authors of the Agile Manifesto further refined what they meant in those value statements by listing a number of principles that they follow as a direct result of those values:

There are a fair number of formal software development methodologies that fall within the Agile camp. While they vary in their specific activities and artifacts, they all use short time-boxed iterations that deliver working software at the end of each iteration (the length of the iterations varies between methodologies). Figure 1-3 shows the typical Agile process.

Figure 1-3. Typical Agile process

All Agile methodologies share a number of core practices (either formally or informally). These core practices also support Lean software development and are the subject of the second half of this book.

Some of the more commonly used Agile methodologies include:

These methods all began at different times in the 1990s as a response to the failure of the Waterfall method. There has been a great deal of cross-fertilization of ideas and techniques between these methods.

Scrum has successfully taken the Agile approach and stripped it down to the essentials. The result is one of the simplest and easiest-to-implement Agile methodologies that still provides the benefits of Agile software development.

XP has a rich set of interlocking practices that can feel overwhelming to those uninitiated in the Agile way, but XP gets credit for popularizing most of the core practices that have been adopted by the other methodologies. XP has probably done more to raise the awareness of Agile software development than any other methodology.

Crystal is actually a family of methodologies created by Alistair Cockburn. The actual processes and practices vary depending on a project’s size and criticality or complexity.

FDD is unique among the Agile methodologies in that its perspective centers on creating domain models for the system being developed, which then organizes development around features that implement the model.

UP is generally considered to be one of the more “heavyweight” of the Agile processes, although it was intended to be tailored and not implemented in a heavy manner. Regardless, this has led to a number of variants, including the Rational Unified Process (RUP), the Agile Unified Process (AUP), and the Enterprise Unified Process (EUP).

DSDM is more formal than most of the other Agile methods, fully specifying many different roles, processes, and artifacts. A notable feature is the prioritizing of requirements according to the MoSCoW rules:

To really appreciate the emergence of Lean production and its derivatives, you have to understand what it was replacing (or competing with): mass production.

Henry Ford popularized mass production (which had itself replaced craft production) with the assembly-line manufacture of the Model T in 1913. Mass production is used to produce large quantities of goods at a low unit cost. It divides the manufacturing process into small steps that can be carried out by unskilled workers, and it relies on the use of high-precision machinery and standardized, interchangeable parts.

The drawback of mass production is its inflexibility. Since a mass production assembly line is so expensive to set up (and difficult to alter), it is only economical if it is going to produce large quantities of the same thing.

In 1945, in post-war Japan, the president of Toyota Motor Company, Kiichiro Toyoda, said that the Japanese automobile industry would not survive if it did not “catch up with America in three years.” This did not happen within three years (the Japanese automobile industry survived anyway), but it did lead to the creation of the Toyota Production System by Taiichi Ohno.

Taiichi Ohno realized that the American mass production system would not work in Japan. The domestic automobile market was too small and there was a significant demand for variety in automobiles, from small economical cars to larger, luxury cars—a poor fit for mass production. Out of necessity, Taiichi Ohno experimented with many ideas and techniques that eventually evolved into the Toyota Production System.

Taiichi Ohno described the Toyota Production System as “a system for the absolute elimination of waste.” He wasn’t kidding. By the early 1990s, Toyota was 60% more productive with 50% fewer defects than its non-Lean competitors. According to Ohno, this striking advantage rested on two pillars: Just-In-Time and autonomation.