2.1. The Most Important UP Idea: Iterative Development

The UP promotes several best practices, but one stands above the others: iterative development. In this approach, development is organized into a series of short, fixed-length (for example, four week) mini-projects called iterations; the outcome of each is a tested, integrated, and executable system. Each iteration includes its own requirements analysis, design, implementation, and testing activities.

The iterative lifecycle is based on the successive enlargement and refinement of a system through multiple iterations, with cyclic feedback and adaptation as core drivers to converge upon a suitable system. The system grows incrementally over time, iteration by iteration, and thus this approach is also known as iterative and incremental development (see Figure 2.1).

Early iterative process ideas were known as spiral development and evolutionary development [Boehm88, Gilb88].

Example As an example (not a recipe), in a two-week iteration half-way through a project, perhaps Monday is spent primarily on distributing and clarifying the tasks and requirements of the iteration, while one person reverse-engineers the last iteration's code into UML diagrams (via a CASE tool), and prints and displays noteworthy diagrams. Tuesday is spent at whiteboards doing pair design work drawing rough UML diagrams captured on digital cameras, and writing some pseudocode and design notes. The remaining eight days are spent on implementation, testing (unit, acceptance, usability, ...), further design, integration, daily builds, system testing, and stabilization of the partial system. Other activities include demonstrations and evaluations with stakeholders, and planning for the next iteration.

Notice in this example that there is neither a rush to code, nor a long drawn-out design step that attempts to perfect all details of the design before programming. A “little” forethought regarding the design with visual modeling using rough and fast UML drawings is done; perhaps a half or full day by developers doing design work in pairs.

The result of each iteration is an executable but incomplete system; it is not ready to deliver into production. The system may not be eligible for production deployment until after many iterations; for example, 10 or 15 iterations.

The output of an iteration is not an experimental or throw-away prototype, and iterative development is not prototyping. Rather, the output is a production-grade subset of the final system.

Although, in general, each iteration tackles new requirements and incrementally extends the system, an iteration may occasionally revisit existing software and improve it; for example, one iteration may focus on improving the performance of a subsystem, rather than extending it with new features.

Embracing Change: Feedback and Adaptation

The subtitle of one book that discusses iterative development is Embrace Change [Beck00]. This phrase is evocative of a key attitude of iterative development: Rather than fighting the inevitable change that occurs in software development by trying (usually unsuccessfully) to fully and correctly specify, freeze, and “sign off” on a frozen requirement set and design before implementation, iterative development is based on an attitude of embracing change and adaptation as unavoidable and indeed essential drivers.

This is not to say that iterative development and the UP encourages an uncontrolled and reactive “feature creep”-driven process. Subsequent chapters explore how the UP balances the need—on the one hand—to agree upon and stabilize a set of requirements, with—on the other hand—the reality of changing requirements, as stakeholders clarify their vision or the marketplace changes.

Each iteration involves choosing a small subset of the requirements, and quickly designing, implementing, and testing. In early iterations the choice of requirements and design may not be exactly what is ultimately desired. But the act of swiftly taking a small step, before all requirements are finalized, or the entire design is speculatively defined, leads to rapid feedback—feedback from the users, developers, and tests (such as load and usability tests).

This early feedback is worth its weight in gold; rather than speculating on the correct requirements or design, the feedback from realistic building and testing something provides crucial practical insight and an opportunity to modify or adapt understanding of the requirements or design. End-users have a chance to quickly see a partial system and say, “Yes, that's what I asked for, but now that I try it, what I really want is something slightly different.”^[1] This “yes...but” process is not a sign of failure; rather, early and frequent structured cycles of “yes...buts” are a skillful way to make progress and discover what is of real value to the stakeholders. Yet, as mentioned, this is not an endorsement of chaotic and reactive development in which developers continually change direction—a middle way is possible.

^[1] Or more likely, “You didn't understand what I wanted!”

In addition to requirements clarification, activities such as load testing will prove if the partial design and implementation are on the right path, or if in the next iteration, a change in the core architecture is required. Better to resolve and prove the risky and critical design decisions early rather than late—and iterative development provides the mechanism for this.

Consequently, work proceeds through a series of structured build-feedback-adapt cycles. Not surprisingly, in early iterations the deviation from the “true path” of the system (in terms of its final requirements and design) will be larger than in later iterations. Over time, the system converges towards this path, as illustrated in Figure 2.2.

Benefits of Iterative Development

Benefits of iterative development include:

• early rather than late mitigation of high risks (technical, requirements, objectives, usability, and so forth)

• early visible progress

• early feedback, user engagement, and adaptation, leading to a refined system that more closely meets the real needs of the stakeholders

• managed complexity; the team is not overwhelmed by “analysis paralysis” or very long and complex steps

• the learning within an iteration can be methodically used to improve the development process itself, iteration by iteration

Iteration Length and Timeboxing

The UP (and experienced iterative developers) recommends an iteration length between two and six weeks. Small steps, rapid feedback, and adaptation are central ideas in iterative development; long iterations subvert the core motivation for iterative development and increase project risk. Much less than two weeks, and it is difficult to complete sufficient work to get meaningful throughput and feedback; much more than six or eight weeks, and the complexity becomes rather overwhelming, and feedback is delayed. A very long iteration misses the point of iterative development. Short is good.

A key idea is that iterations are timeboxed, or fixed in length. For example, if the next iteration is chosen to be four weeks long, then the partial system should be integrated, tested, and stabilized by the scheduled date—date slippage is discouraged. If it seems that it will be difficult to meet the deadline, the recommended response is to remove tasks or requirements from the iteration, and include them in a future iteration, rather than slip the completion date. Chapter 37 summarizes reasons for timeboxing.

Massive teams (for example, several hundred developers) may require longer than six-week iterations to compensate for the overhead of coordination and communication; but no more than three to six months is recommended. For example, the successful replacement in the 1990s of the Canadian air traffic control system was developed with an iterative lifecycle and other UP practices. It involved 150 programmers and was organized into six-month iterations.^[2] But note that even in the case of an overall six-month project iteration, a subsystem team of 10 or 20 developers can break down their work into a series of six one-month iterations.

^[2] Philippe Kruchten, who also led the development of the RUP, served as chief architect for the project.

A six-month iteration is the exception for massive teams, not the rule. To reiterate, the UP recommends that normally an iteration should be between two and six weeks in duration.