Chapter 7. Practice 4: Short Iterations
He doubly benefits the needy who gives quickly.
Iterative development delivers functional software to the customer for evaluation at specific intervals. Each iteration adds new functionality to the product and increases the customer’s understanding of how the final product will fulfill his needs. It also provides an opportunity to identify how the product may fall short. The effectiveness of iterative development comes from these opportunities to receive and incorporate customer feedback. Customer feedback drives the next iteration by adding, deleting, and modifying requirements to correct shortcomings, and iteration length determines the number of opportunities available during a development effort. This means that short iterations provide more feedback and increase the possibility that the final product will be what the customer wants.
Short iterations support three of the Lean software development principles described in Chapter 2:
Eliminate waste
Defer commitment
Deliver fast
Two forms of waste that are prevalent in software development are partially completed work and replanning. Partially completed work is any work that has been started but not completed; it includes such things as uncoded requirements and incomplete, untested code. Because partially completed work doesn’t contribute new functionality to the current product, it is non-value-added work. In addition, if the customer decides to stop development, in a specific area or for the entire product, any partially completed work is wasted effort. Planning too far into the future can result in replanning when the customer’s needs change. Effort spent in the initial planning is wasted when work is replanned. Short iterations prevent both situations by focusing efforts on developing the customer’s highest priority requirements quickly and planning just far enough ahead to support that development.
Deferring commitment means putting off irrevocable decisions as long as possible (and reasonable). Delaying such decisions gives developers time to gather the information needed to make them, and short iterations provide opportunities to collect that information in the form of customer feedback. Developers can create prototypes of the final implementation in an early iteration, gather feedback, and use the feedback to decide on the final implementation in a later iteration.
As noted earlier, iterative development puts new functionality in the customer’s hands at specific intervals. Short iterations support the “deliver fast” principle by reducing interval size and delivering new functionality in quick response to customer feedback. Instead of waiting until the final delivery of the product to see the effects of their comments, customers see it in the next iteration.
The use of short iterations in Lean software development relies on both underlying principles and specific techniques for implementing those principles.
Short Iterations Generate Customer Value
Two major principles drive the use of short iterations in Lean software development: increasing feedback opportunities and making course corrections. Getting feedback from the customer and implementing that feedback allow you to develop the product efficiently, get the product delivered quickly, and ensure the final product supports the customer’s needs.
Increase Feedback Opportunities
One of the goals of short iterations is to increase the amount of feedback received from the customer. Short iterations deliver new functionality more often, thus increasing the number of opportunities customers have to evaluate the product. Figure 7-1 shows graphically how shorter iterations increase feedback opportunities. A 12-month development effort consisting of a single iteration provides no opportunity for feedback until after the final product is delivered, at which point it’s too late. Breaking the effort into two iterations provides some feedback, allowing developers to modify software developed during the first six months in the second six months. But if a little feedback is good, more should be better, right? By breaking the development into monthly iterations, developers can get feedback from customers at 11 points during the development effort.
Customer feedback benefits a development effort in several ways. First, assuming we have learned our lesson about prioritized requirements by now, feedback allows developers to focus on the features most important to the customer. Each iteration implements a subset of the requirements, starting with the most important. Following each iteration, the requirements list is modified based on customer feedback; requirements are added, deleted, modified, and reprioritized. The next iteration again implements a subset of the updated requirements in priority order, so developers are always implementing the requirements most important to the customer, even if the priorities are completely different from the previous iteration.
The cycle just described has a couple of important side effects: requirements changes are accounted for automatically, and domain knowledge gained in early iterations can be applied to subsequent iterations, making those iterations more efficient.
Frequent feedback limits over-investing in one area of the project by giving the customer an opportunity to indicate a feature is “good enough.” Although further refinement of the feature may be possible and may improve the final product, continued refinement past a certain point generates diminishing returns. The additional time spent on “good enough” features may be better spent in areas where the return is greater.
Finally, feedback ensures that the right product is built. Initial requirements are often generated without a good understanding of what’s really needed. By providing many interim releases via short iterations, developers allow customers to gain a better understanding of the requirements and to see the realm of possibilities. Customer feedback at regular intervals allows developers to change the direction of development, drop or add features, or stop development in areas with diminishing returns.
Feedback also has a positive effect on the developer-to-customer relationship. A customer who provides feedback and sees that feedback reflected in the next iteration will feel a part of the process. In addition to fostering goodwill between the customer and the development team, this feeling will encourage the customer to spend more time evaluating the product and providing better feedback, since he knows it will be taken seriously.
Make Course Corrections
Getting feedback from customers is half the job. Implementing the feedback in the product is the other half. Short iterations allow frequent course corrections, which can be used to home in on the target.
Two types of problems that can arise during development are miscommunication between customers and developers, and changing requirements. You can address both problems by making course corrections based on customer feedback.
Miscommunication between customers and developers manifests itself as features in the product that don’t behave the way the customer intended. For example, a word processor document may have the following user story, a form of requirement described in Mike Cohn’s book, User Stories Applied (Addison-Wesley Professional, 2004):
As an author, I want to make a copy of the current document so I can save a snapshot before continuing to write.
The story doesn’t specify what to name the copy, so developers might assume that the name should be in the typical Copy (x) of myfile.doc format and implement it this way. At the end of the iteration, the application is shown to the customer, who immediately points out that the copy’s name should be formatted as myfile(x).doc. As shown in Figure 7-2, developers respond to this feedback by adding a note to the user story to bring it in line with the customer’s needs and changing the implementation in the next iteration.
Requirements change when the customer determines that the original requirements no longer reflect the required functionality. Such changes may occur for any number of reasons, but the result is an implementation that is no longer in line with the customer’s requirements. Perhaps the customer has decided to switch from a client-server application to a web services application running in a browser. In this case, the developers are given a new direction and must make a major course correction, as in Figure 7-3.
The ability to make course corrections depends on getting feedback from the customer; the ability to make course corrections quickly and efficiently depends on short iterations, which generate the necessary feedback in a timely fashion.
Developing with Short Iterations
The techniques that support short iterations are straightforward. The first two—prioritized requirements and a fixed iteration length—work together to ensure that developers are always working toward getting the customer’s highest priority requirements implemented. The remaining techniques—demonstrating the iteration’s product and delivering functional software to the customer—help focus the team on meeting its goals of providing value and getting useful feedback.
Work to Prioritized Requirements
Prioritized requirements can help minimize the size of the codebase (see Chapter 6), but they play a role in short iterations as well. Focusing on the most important requirements delivers the most useful functionality to the customer, so short iterations that implement the most important features first deliver the most value in the shortest time.
Set an Iteration Length and Stick to It
Although most Agile methodologies recommend iteration lengths of two to six weeks, several factors influence iteration length, including the volatility of the environment, experience with the methodology in use, the technical ability of the team and its understanding of the technologies in use, and the complexity of the system itself.
Choosing an iteration length within the two- to six-week range, or in some cases outside of it, will depend on each of those factors. However, unless dictated otherwise by experience or knowledgeable recommendation, the best starting point is in the middle of that range—four weeks.
The customer’s understanding of the domain, the level of competition within the domain, and the volatility of the domain itself all combine to drive the volatility of the environment. You will typically find a lower rate of change when developing for a customer who is the major force in a domain and has very little competition than you will when developing for a startup company in a brand new field with many players. Development within a research organization may be quite a bit more hectic, as new technologies, or uses for them, drive frequent requirements changes.
Team experience can also affect iteration length. A team with lots of experience using the development process will be more efficient than a less experienced one, and will be able to get more done in a shorter iteration. The same holds true for the team’s experience with the technologies being used. The need to research a technology before using it in a product can slow the pace of development significantly.
Finally, the complexity of the system can affect the pace of development. Systems with many external components place an integration burden on the team, and systems requiring lots of infrastructure development can generate tasks that span multiple iterations.
Regardless of the iteration length chosen, adherence to that length is very important, so set an end date for each iteration up front. Make sure everyone (especially management) understands that the date is not negotiable and that if development begins to get behind schedule, you will omit functionality in order to make the release date. An inviolable end date does two things: it ensures that the customer gets new functionality to evaluate regularly, and it focuses the development team.
Customer feedback is one of the main drivers for short iterations; without new functionality to evaluate, customers can’t provide useful feedback. If a development team allows the end dates to slip, or even worse, allows date slippage to become a habit, the ability to get and incorporate feedback will be at risk. Better to get feedback on less functionality than on none at all.
Iteration end dates also focus the team on the task at hand. A team that knows when it has to deliver tends to stay focused on the requirements, especially when the iteration always ends with a demonstration in front of the customer.
End Each Iteration with a Demo
An end-of-iteration demonstration serves two purposes: it recognizes the development team’s effort, and it officially marks the end of the iteration.
Ending the iteration with a demo gives the development team a chance to show off what it has been doing over the course of the iteration. A positive reaction from the customer provides a sense of accomplishment and builds enthusiasm for the next iteration. An end-of-iteration demo also focuses the team on the work it’s doing. No one wants to stand up in front of the customer and announce that no new functionality is ready, so the entire team has a vested interest in implementing requirements and avoiding tasks that do not help reach that goal. It also drives home the point that delivering new functionality is important, enough so that finishing a few of the requirements is better than half-finishing all the requirements.
An end-of-iteration demo is the team’s declaration that it has completed the work it agreed to for the iteration and signifies the official handoff to the customer. It indicates to the customer that the team is dedicated to reaching milestones and delivering on promises. Both the team and the customer are afforded a sense of completion that punctuates the team’s progress.
Deliver the Iteration’s Product to the Customer
While the end-of-iteration demo introduces customers to the iteration’s product, it isn’t enough to generate good feedback. Customers need time to dig into the product and apply it to the tasks for which it will eventually be used. Only after customers have had time to really put the product through its paces will they begin to provide useful feedback, so the product must be delivered to the end users.
Delivering the product doesn’t necessarily mean shipping the product in fancy packaging; it can be as simple as giving customers access to a distribution file on an FTP server. The means of delivery will depend on the type of product under development: client applications may be distributed via CD/DVD, web applications may be distributed by updating a server, software for phones or PDAs may require providing hardware with the new application loaded. Whatever the method, getting the software in the customer’s hands so they can use it in their own environment is essential to getting good feedback.
The Fallacy of Iterative Development
Iterative development is an effective approach for creating software products. However, the desire to create deterministic plans and schedules sometimes leads to an implementation that marginalizes the benefits of iterative development. The fallacy of iterative development is the following pair of false assumptions about iterative development:
Iterative development is a series of short Waterfalls.
The content of all iterations should be defined at the outset by divvying up the requirements.
These assumptions lead to a process, shown in Figure 7-4, in which all requirements are gathered at the beginning of the development cycle and an upfront planning activity determines the content of all iterations by divvying up the requirements into iteration-sized pieces, typically with each iteration focused on a different area of the product. The product is then developed via a series of iterations, each of which has the typical Waterfall phases of design, implementation, and test (remember, we eliminated the integration phase with continuous integration). Although this approach is incremental (new features are added each iteration), it doesn’t incorporate customer feedback by reevaluating requirements after each iteration. Features implemented during the iteration are considered complete, and the iteration that follows simply moves on to the next feature.
Two problems can arise from this type of iterative development: customer input is ignored, or the effort is replanned numerous times. Planning the entire development effort at the outset doesn’t leave room for change based on customer feedback. The development team forges ahead based on the plan, even in the face of customer feedback indicating it is headed in the wrong direction. The end result of sticking to the original plan and ignoring customer feedback is that the development team builds the wrong product.
Incorporating customer feedback after each iteration is important, but when that feedback is incorporated through replanning of the remaining iterations, the result is a lot of wasted effort. For a 12-month development effort using 1-month iterations, 12 iterations are planned at the beginning of the effort. Following the first iteration, the remaining 11 iterations are replanned based on customer feedback. After the second iteration, the remaining 10 iterations are replanned, and so on until the end of the effort. Although customer feedback is being incorporated, much of the planning effort is wasted. The final iteration is planned 12 times, but only the last plan is executed.
The solution to both problems is to plan only the next (short) iteration in detail, as shown in Figure 7-5. Customer needs are captured by the prioritized requirements list, so the list is updated at the start of each iteration. Planning for each iteration is simply a matter of selecting a subset of the requirements from the top of the list.
Planning short iterations one at a time from an updated requirements list guarantees that the development team is always working the issues that are most important to the customer and that the final product is what the customer needs. In addition, no effort is wasted trying to plan too far ahead.
Big Tasks in Little Pieces
The single biggest objection to developing in short iterations is that some tasks are just too big to fit. System architecture is often placed in this category, as is the development of complex functionality that “doesn’t work unless all the pieces are there.” On the surface, these may seem like valid arguments, and in traditional Waterfall development, they are. However, changing the development paradigm addresses both arguments.
Emergent design allows the component-level architecture to evolve as requirements become more defined. When evolving an architecture over time, short iterations actually aid development. The quick turnaround times resulting from short iterations provide timely feedback on what works, allow architects to try multiple approaches, and postpone critical decisions until the system is better understood.
Emergent design doesn’t relieve system architects of all need to think ahead. For example, attempting to retrofit security onto an existing system can result in lots of redesign and rewriting. However, adequate consideration of security issues early on doesn’t require implementing a full-blown security component in the first iteration.
The implementation of complex functionality often doesn’t fit in one iteration, which can mean that the development team has no functional software to release at the end of the demo. However, complex functionality can typically be broken into smaller, less complex pieces. Simulating nonexistent components or incomplete functionality through the use of mock objects and method stubs allows development to proceed without all the pieces in place. As development proceeds, components and functionality are integrated incrementally until the mock objects and stubs are eventually replaced with the real thing. This approach allows the team to demonstrate functional software following each iteration.
Summary
Short iterations deliver functional software to the customer regularly and frequently, allowing customer feedback to play an integral role in the ongoing development process.
Short iterations support Lean software development principles such as eliminating waste, deferring commitment until the last reasonable moment, and delivering functional software quickly.
Short iterations are driven by the need to increase customer feedback and make course corrections throughout the development cycle, both of which help ensure that the final product is what the customer really needs.
Working to a prioritized requirements list is one of the enabling techniques for implementing short iterations. Prioritized lists ensure that the team is working on the most important features.
Strictly enforcing a standard iteration length and presenting the iteration output to the customer help keep the development team focused on the requirements scheduled for the current iteration.
Delivering the iteration output to the customer, or at least making it available for testing and experimentation, gives the customer time to determine which features fit their needs and which need more work or should be dropped. More evaluation time generates better feedback and, ultimately, a better product.