37.5. The Sequential “Waterfall” Lifecycle
In contrast to the iterative lifecycle of the UP, an old alternative is the sequential, linear, or “waterfall” lifecycle [Royce70], associated with heavy and predictive processes. In common usage, a waterfall lifecycle defines steps similar to the following:
1. | Clarify, record, and commit to a set of final requirements. |
2. | Design a system based on these requirements. |
3. | Implement, based on the design. |
4. | Integrate disparate modules. |
5. | Evaluate and test for correctness and quality. |
A development process based on the waterfall lifecycle is associated with these behaviors or attitudes:
Carefully and fully define an artifact (for example, the requirements or design) before carrying on to the next step.
Commit to a frozen set of detailed requirements.
Deviation from the requirements or design during later steps indicates a failure in not having been sufficiently skillful or thorough. Next time, try harder to get it right.
A waterfall process is similar to the engineering approach by which buildings and bridges are constructed. Its adoption made software development appear more structured and similar to engineering in some other fields. For some time, a waterfall process was the approach most software developers, managers, authors, and teachers were taught when they were students (and then repeated), without critical research into its suitability for software development.
Some things should be built like buildings—such as, well...buildings—but not usually software.
A mentioned in the opening chapter on the UP, a two year study reported in the MIT Sloan Management Review of successful software projects identified four common factors for success; iterative development, rather than a waterfall lifecycle, was first on the list [MacCormack01].
Some Problems with the Waterfall Lifecycle
The building metaphor has outlived its usefulness. It is time to change again. If, as I believe, the conceptual structures we construct today are too complicated to be accurately specified in advance, and too complex to be built faultlessly, then we must take a radically different approach (iterative, incremental development).
—Frederick Brooks, “No Silver Bullet,” The Mythical Man-Month
Within a certain time scale, doing some requirements before design, and some design before implementation, is inevitable and sensible. For a short two month project, a sequential lifecycle is workable. And a single iteration in iterative development is like a short waterfall project.
However, difficulties start to arise as the time scale lengthens. The complexity becomes high, speculative decisions increase and compound, there is no feedback, and in general high risk issues are not being tackled early enough. By definition, one attempts to do all or most of the requirements for the entire system before moving on, and most of the design before moving on.
Large steps are taken in which many decisions are made without the benefit of concrete feedback from realistic implementation and testing. On the scale of a two-week mini-project (that is, an iteration), a linear requirements-design-implementation sequence is workable; the degree of speculative commitment to some requirements and design is not in the danger zone. However, as the scale expands, so do the speculation and risk.
Problems with a waterfall process at the scale of the entire project include:
delayed risk mitigation; tackling high risk or difficult problems late
requirements and design speculation and inflexibility
high complexity
low adaptability
Mitigation of Some Problems with the Waterfall Lifecycle
Iterative development is not a magic bullet for the challenges of software development. Yet, it offers support to reduce some problems exacerbated by a linear waterfall lifecycle.
Problem: Delayed Risk Mitigation
Risks come in many forms: the wrong design, the wrong set of requirements, a strange political environment, lack of skills or resource, usability, and so forth.
In a waterfall lifecycle, there is not an active attempt to identify and mitigate the riskiest issues first. As an example, the wrong architecture for a high-load high-availability website can cause costly delays, or worse. In a waterfall process, validation of the architecture's suitability happens long after all requirements and all design are specified (inevitably imperfectly), during the later major step of implementation. This could be many months or even years after inception of the project (see Figure 37.1). And there is no shortage of stories where separate teams have built subsystems over a long period, and then attempted to integrate these and start overall system testing near the end of the project—with predictably painful results.
Figure 37.1. Waterfall lifecycle and risks.
Mitigation
In contrast, in iterative development the goal is to identify and mitigate the riskiest issues early. The high risks might be in the core architectural design, the usability of the interface, disengaged stakeholders. Whatever, they are tackled first. As illustrated in Figure 37.2, early iterations focus on driving down the risk. Continuing with the prior high-load website example, in an iterative approach, before much investment in other requirements or design work, the team first designs, implements, and realistically tests enough of the core architecture to prove it is on the right track with respect to load and availability. If the tests prove them wrong, they adapt the core design in the early stages of the project, rather than near the end.
Figure 37.2. Iterative lifecycle and risks.
Problem: Requirements Speculation and Inflexibility
A fundamental assumption in a waterfall process is that requirements can be relatively fully specified and then frozen in the first phase of a project. On such projects, there is an effort to first do thorough requirements analysis, culminating in a set of requirements artifacts that are reviewed and “signed off.”
It turns out this is usually a flawed assumption. The effort to get all the requirements defined and signed-off before any design and implementation work is ironically likely to increase project difficulties rather than ameliorate them. It also makes it difficult to respond late in a project to a new business opportunity via a change in the software.
Granted, there are some projects where an effort to first fully and accurately specify the requirements is necessary. This is especially true when the software is coupled with the building of physical components. Examples include aircraft and medical devices. But note that even in this case, iterative development can still be profitably applied to the design and implementation process.
The most compelling research deconstructing the myth of being able to successfully first define all requirements comes from [Jones97]. As illustrated in Figure 37.3, in this large study of 6,700 projects, creeping requirements—those not anticipated near the start—are a very significant fact of software development life, ranging from around 25% on average projects, up to 50% on larger ones; Boehm and Papaccio present similar research-based conclusions in [BP88]. Waterfall attitudes, which struggle against (or simply deny) this fact by assuming requirements and designs can be specified and frozen, are incongruous with most project realities.
Figure 37.3. Changing requirements are an inevitable force in development.[2]
[2] Function points describe system complexity with a programming language-independent metric (see www.ifpug.org).
Thus, “the only constant is change,” usually because:
the stakeholders change their minds or cannot fully envision what they want until they see a concrete system[3]
[3] Barry Boehm has called this the “I'll know it when I see it” effect.
the market changes
correctly validated, detailed, and precise specification is a psychological and organizational challenge for most stakeholders [Kruchten00]
And so, there are predictable and often-seen problems that arise in waterfall projects. Since in reality significant change is inevitable, these include:
as described earlier, delayed discovery and mitigation of high risks
a negative feeling among team members of “living a fiction” or failure on the project, as the reality of changes does not correspond to the ideal
making a large (costly) investment in the wrong design and implementation (since it is based on incorrect requirements)
lack of responsiveness to changing user wishes or market opportunities
Mitigation
In iterative development, not all requirements are specified before design and implementation, and requirements are not stabilized until after at least several iterations. For example:
First, a subset of core requirements is defined, for example, within a two-day requirements workshop. Then, the team chooses a subset of those to design and implement (based usually on highest risk or business value). After a four-week iteration, stakeholders meet in a second one- or two-day requirements workshop, intensively review the partial system, and clarify and modify their requests. After a second (shorter) two-week iteration of incrementally implementing the system, stakeholders meet in a third requirements workshop, and refine again. At this point, the requirements start to stabilize and represent the true scope and clarified intentions of the stakeholders. At this point, a somewhat realistic plan and estimate of the remaining work is possible. These iterations may be characterized as part of the UP elaboration phase.
Later requirements changes are still acceptable. However, the interplay in early iteration of parallel implementation work and requirements analysis that obtains feedback from the partial implementation leads to better requirements definition in the elaboration phase.
Problem: Design Speculation and Inflexibility
Another central idea in the waterfall lifecycle is that the architecture and majority of the design can and should be relatively fully specified in the second major phase of a project, once the requirements are clarified. On such projects, there is an effort to thoroughly describe the complete architecture, object designs, user interface, database schema, and so forth, before implementation begins. Some problems associated with this assumption:
Since requirements will change, the original design may not be reliable.
Immature or misunderstood tools, components, and environments make speculative design decisions risky; they may be proven wrong upon implementation because “the application server wasn't supposed to do that, ...”
In general, lack of feedback to prove or disprove the design, until long after the design decisions were made.
Mitigation
These problems are mitigated in iterative development by quickly building part of the system and validating the design and third-party components through testing.