Chapter 22. Why Agile Planning Works
“If you want a guarantee, buy a toaster.”
—Clint Eastwood in The Rookie
Having arrived at this point, we are well equipped to look at why agile estimating and planning are successful. First, remember the purpose of estimating and planning. Planning is an attempt to find an optimal solution to the overall product development question: features, resources, and schedule. Changing any one of these causes changes in the others. While planning, we are exploring the entire spectrum of possible solutions of how to mix these three parameters such that we create the best product possible. The estimates and plans that we create must be sufficient for serving as the basis for decisions that will be made by the organization. In this chapter, we consider the agile estimating and planning process described so far by this book to see how it achieves these goals.
Replanning Occurs Frequently
One of the ways in which agile estimating and planning support the efficient exploration of the new product development solution space is through frequent replanning. At the start of each new iteration, a plan for that iteration is created. The release plan is updated either after each iteration or, at worst, after every few iterations. Acknowledging the impossibility of creating a perfect plan goes a long way toward reducing the anxiety that accompanies such a goal (Githens 1998). Knowing that a plan can be revised at the start of the next iteration shifts a team’s focus from creating a perfect plan (an impossible goal) to creating a plan that is useful right now.
For a plan to be useful, it must be accurate, but we accept that early plans will be imprecise. One of the reasons we replan is to remove that imprecision progressively. That is, an early plan may say that a project will deliver 300 to 400 story points in the third quarter. That plan may later turn out to be accurate (305 story points are delivered in August), but this plan is not particularly precise. During the early stages of a project, this plan may very likely be a sufficient basis for making decisions. Later, though, to remain useful, the plan will gain precision, and we may say that the project will deliver 380 to 400 story points in September.
An agile estimating and planning process recognizes that our knowledge is always incomplete and requires that plans be revised as we learn. As a team learns more about the product they are building, new features may be added to the release plan. As the team learns more about the technologies they are using or about how well they are working together, expectations about their rate of progress are adjusted. For a plan to remain useful, this new knowledge needs to be incorporated into the plan.
Estimates of Size and Duration Are Separated
A common planning flaw (on traditional as well as many agile teams) is confusing estimates of size and duration. Clearly, the size of an effort and the time needed to complete that effort are related, but many additional factors affect duration. A project of a given size will take programmers of different skill and experience levels different amounts of time. Similarly, duration is affected by the size of the team working on the project. A four-person team may take six months (twenty-four person-months). An eight-person team may reduce that to four calendar months but thirty-two person-months, even though the project is the same size in both cases.
To see the difference between estimates of size and duration, suppose I show you a book and ask you how long it will take you to read it. You can tell from its title that it’s a novel, but I won’t let you look inside the book to see how many pages are in it, how wide the margins are, or how small the type is. To answer my question about how long it will take you to read this book, you first estimate the number of pages. Let’s assume you say 600. Then you estimate your rate of progress at one page per minute. You tell me it will take you 600 minutes, or 10 hours. In arriving at this estimate of duration (10 hours), you first estimated the size of the job (600 pages).
Agile estimating and planning succeed because estimates of size and duration are separated. We start by estimating the size of a project’s user stories in story points. Because story points are abstract and notional, they are pure estimates of size. We then estimate a rate of progress, which we call velocity. Our estimates of size and velocity are then combined to arrive at an estimate of duration. Our estimating and planning process benefits from this clear and total separation of estimates of size and duration.
Plans Are Made at Different Levels
Because agile plans cover three different levels—the release, the iteration, and the current day—each plan can be made with a different level of precision. Having plans with different time horizons and different levels of precision has two main benefits. First, it conveys the reality that the different plans are created for different reasons. The daily plan, as committed to by each participant in a team’s daily meeting, is fairly precise: Individuals express commitments to complete, or at least make progress on, specific tasks during the day. The iteration plan is less precise, listing the user stories that will be developed during an iteration and the tasks thought necessary to do so. Because each user story is imperfectly specified, there is also some vagueness around what it means to say that the story will be developed in the iteration. Finally, the release plan is the least precise of all, containing only a prioritized list of desired user stories and one or more estimates of how much of the desired functionality is likely to be delivered by the desired release date.
The second benefit of planning at different levels is that it helps the team view the project from different perspectives. An iteration plan is necessary for completing the highly coordinated work of a cross-functional team within a short iteration. A release plan provides a broader perspective on the project and ensures that the team does not lose the forest of the release for the trees of an iteration. A team that works iteration to iteration without awareness of a more distant goal runs the risk of continually pursuing short-term goals while missing out on targeting a truly lucrative longer-term goal. Short-term goals may be inconsistent with the desired long-term result.
Plans Are Based on Features, Not Tasks
A traditional plan in the form of a Gantt chart, PERT chart, or work breakdown structure focuses on the tasks needed to create a product. An agile plan focuses instead on the features that will be needed in the product. This is a key distinction, because it forces the team to think about the product at the right level—the features. When features are developed iteratively, there is less need for up-front thinking about the specific tasks necessary. The work of an iteration emerges, and the team will think of or discover all of the tasks as they are needed. What’s more important is that the team think about the features that are being developed. Colleague Jim Highsmith (2004b) has stated that “agile planning is ‘better’ planning because it utilizes features (stories, etc.) rather than tasks. It is easy to plan an entire project using standard tasks without really understanding the product being built. When planning by feature, the team has a much better understanding of the product.” At a task level, many project plans look the same; every agile plan is specific to the product being developed.
Small Stories Keep Work Flowing
From queuing theory (Poppendieck and Poppendieck 2003; Reinertsen 1997), we learn the importance of focusing on cycle time, the amount of time something takes to go from the start of a process to the end of that process. On a software project, cycle time is the time from when the team begins work on a feature until that feature delivers value to users. The shorter the cycle time, the better.
A key influence on cycle time is the variability in the time it takes to develop a new feature. One of the best ways to reduce variability is to work with reasonably small and similar size units of work. The estimating and planning process outlined in this book supports this by advising that teams estimate their short-term work within approximately one order of magnitude. Larger user stories can exist further down a project’s prioritized requirements list. However, as those features near the top of the list (when they will be scheduled into an iteration that is beginning), they are disaggregated into smaller pieces.
Work in Process Is Eliminated Every Iteration
High amounts of work in process slow a team. On a software project, work in process exists on any feature that the team has started developing but has not yet finished. The more work in process there is, the longer any new feature will take to develop, as the new feature must follow the already-started work. (Or the new feature needs to be expedited, jumping ahead of the work in process, but that just compounds the problem for the next feature that cannot be expedited.) So work in process leads to increased cycle times. And in the previous section, we learned the importance of maintaining a short cycle time.
One of the reasons why agile planning succeeds is that all work in process is eliminated at the end of each iteration. Because work is not automatically rolled forward from one iteration to the next, each iteration is planned afresh. This means that work on a feature not fully implemented in one iteration will not necessarily be continued in the next. It often will be, but there’s no guarantee. This has the effect of eliminating all work in process at the start of each iteration.
Because work in process is eliminated at the start of each iteration, teams are more easily able to work efficiently in short iterations. This means a shorter feedback loop from users to the project team, which leads to faster learning as well as more timely risk mitigation and control.
Tracking Is at the Team Level
Traditional approaches to estimating and planning measure and reward progress at the individual team member level. This leads to a variety of problems. For example, if finishing a task early results in a programmer’s being accused of giving a padded estimate for the task, the programmer will learn not to finish early. Rather than finish early, she won’t report a task as complete until its deadline.
Agile estimating and planning successfully avert this type of problem by tracking progress only at the team level. This is one of the reasons that Chapter 14, “Iteration Planning,” included the advice that individuals should refrain from signing up for specific tasks during iteration planning. Similarly, there are no individual burndown charts prepared—only a teamwide burndown chart.
Uncertainty Is Acknowledged and Planned For
Many traditional, prescriptive plans make the mistake of believing that features can be locked down at the beginning of a project. Plans are then made that do not allow changes or force changes through an onerous change control process. This leads us to delivering projects with features that users don’t want. When we create a plan early in a project and do not update the plan as we acquire new knowledge, we lose the opportunity to synchronize the plan with reality.
With an agile approach to estimating and planning, teams acknowledge both end and means uncertainty. End uncertainty (about the product ultimately being built) is reduced as product increments are shown to potential users and other stakeholders at the end of each iteration. Their feedback and responses are used to fine-tune future plans. Means uncertainty (about how the product will be built) is reduced as the team learns more about the technologies in use and themselves. Early discovery that a particular third-party component cannot meet performance requirements, for example, may lead the team to find an alternative. The time to find and switch to the new component will need to be factored into new plans once the need is identified.
A Dozen Guidelines for Agile Estimating and Planning
With all of these reasons in mind, the following is a list of a dozen guidelines for successful agile estimating and planning.
1. Involve the whole team. Primary responsibility for certain activities may fall to one person or group, as prioritizing requirements is primarily the responsibility of the product owner. However, the whole team needs to be involved and committed to the pursuit of the highest-value project possible. We see this, for example, in the advice that estimating is best done by the whole team, even though it may be apparent that only one or two specific team members will work on the story or task being estimated. The more responsibilities are shared by the team, the more success the team will have to share.
2. Plan at different levels. Do not make the mistake of thinking that a release plan makes an iteration plan unnecessary, or the other way around. The release, iteration, and daily plans each cover a different time horizon with a different level of precision, and each serves a unique purpose.
3. Keep estimates of size and duration separate by using different units. The best way to maintain a clear distinction between an estimate of size and one of duration is to use separate units that cannot be confused. Estimating size in story points and translating size into duration using velocity is an excellent way of doing this.
4. Express uncertainty in either the functionality or the date. No plan is certain. Be sure to include an expression of uncertainty in any release plan you produce. If the amount of new functionality is fixed, state your uncertainty as a date range (“We’ll finish in the third quarter” or “We’ll finish in between seven and ten iterations”). If the date is fixed instead, express uncertainty about the exact functionality to be delivered (“We’ll be done on December 31, and the product will include at least these new features, but probably no more than those other new features”). Use bigger units (iterations, months, and then quarters, for example) when the amount of uncertainty is greater.
5. Replan often. Take advantage of the start of each new iteration to assess the relevancy of the current release plan. If the release plan is based on outdated information or on assumptions that are now false, update it. Use replanning opportunities to ensure that the project is always targeted at delivering the greatest value to the organization.
6. Track and communicate progress. Many of a project’s stakeholders will have a very strong interest in the progress of the project. Keep them informed by regularly publishing simple, very understandable indicators of the team’s progress. Burndown charts and other at-a-glance indicators of project progress are best.
7. Acknowledge the importance of learning. Because a project is as much about generating new knowledge as it is about adding new capabilities to a product, plans must be updated to include this new knowledge. As we learn more about our customers’ needs, new features are added to the project. As we learn more about the technologies we are using or about how well we are working as a team, we adjust expectations about our rate of progress and our desired approach.
8. Plan features of the right size. Functionality that will be added in the near future (within the next few iterations) should be decomposed into relatively small user stories—typically, items that will take one or two days up to no more than ten days. We are best at estimating work that is all within one order of magnitude in size. Working with user stories that fall within these ranges will provide the best combination of effort and accuracy. It will also provide stories that are small enough to be completed during one iteration for most teams. Of course, working with small user stories can become quite an effort on longer projects. To balance this, if you are creating a release plan that will look more than two or three months into the future, either write some larger stories (called epics) or estimate the more distant work at the theme level to avoid decomposing large stories into small ones too far in advance.
9. Prioritize features. Work on features in the order that optimizes the total value of the project. In addition to the value and cost of features when prioritizing, consider the learning that will occur and the risk that will be reduced by developing a feature. Early elimination of a significant risk can often justify developing a feature early. Similarly, if developing a particular feature early will allow the team to gain significant knowledge about the product or their effort to develop it, they should consider developing that feature early.
10. Base estimates and plans on facts. Whenever possible, ground your estimates and plans in reality. Yes, there may be times in some organizations when it will be necessary to estimate things like velocity with very little basis in fact, and Chapter 16, “Estimating Velocity,” presented some valid techniques for doing so. However, whenever possible, estimates and plans should be based on real, observed values. This goes, too, for an estimate of how much of a feature is complete. It’s easy to tell when a feature is 0% done (we haven’t started it), and it’s relatively easy to tell when we’re 100% done (all tests pass for all of the product owner’s conditions of satisfaction). It’s hard to measure anywhere in between—is this task 50% done or 60% done? Because that question is so hard, stick with what you can know: 0% and 100%.
11. Leave some slack. Especially when planning an iteration, do not plan on using 100% of every team member’s time. Just as a highway experiences gridlock when filled to 100% capacity, so will a development team slow down when every person’s time is planned to full capacity.
12. Coordinate teams through lookahead planning. On a project involving multiple teams, coordinate their work through rolling lookahead planning. By looking ahead and allocating specific features to specific upcoming iterations, interteam dependencies can be planned and accommodated.
Summary
The purpose of agile planning is to discover iteratively an optimal solution to the overall product development questions of which features with which resources in what timeline. An agile approach to estimating and planning succeeds in finding such a solution because plans are made at different levels and replanning occurs frequently; because plans are based on features rather than tasks; because size is estimated first and then duration is derived from the size estimate; because small stories keep work flowing, and work in process is eliminated at the end of every iteration; because progress is measured at the team, rather than the individual, level; and because uncertainty is acknowledged and planned for.
Discussion Questions
1. Are there other reasons you can think of why agile estimating and planning succeed?
2. Which of the dozen guidelines apply to your current estimation and planning process? Would that process benefit by following the others?