Chapter 2. Why Planning Fails

“No plan survives contact with the enemy.”

—Field Marshal Helmuth Graf von Moltke

The previous chapter made the argument that the purpose of planning is to arrive iteratively at an optimized answer to the ultimate new product development question of what should be developed. That is, what capabilities should the product exhibit, in what timeframe, and with which and how many resources? We learned that planning supports this by reducing risk, by reducing uncertainty about what the product should be, by supporting better decision making, by establishing trust, and by conveying information.

Unfortunately, the traditional ways in which we plan projects often let us down. In answering the combined scope/schedule/resources question for a new product, our traditional planning processes do not always lead to very satisfactory answers and products. As support of this, consider that:

• Nearly two-thirds of projects significantly overrun their cost estimates (Lederer and Prasad 1992)

• Sixty-four percent of the features included in products are rarely or never used (Johnson 2002)

• The average project exceeds its schedule by 100% (Standish 2001)

In this chapter, we look at five causes of planning failure.

Planning Is by Activity Rather Than Feature

A critical problem with traditional approaches to planning is that they focus on the completion of activities rather than on the delivery of features. A traditionally managed project’s Gantt chart or work breakdown structure identifies the activities that will be performed. This becomes how we measure the progress of the team. A first problem with activity-based planning is that customers get no value from the completion of activities. Features are the unit of customer value. Planning should, therefore, be at the level of features, not activities.

A second problem occurs after a traditional schedule has been created and is being reviewed. When we review a schedule showing activities, we do so looking for forgotten activities rather than for missing features.

Further problems occur because activity-based plans often lead to projects that overrun their schedules. When faced with overrunning a schedule, some teams attempt to save time by inappropriately reducing quality. Other teams institute change-control policies designed to constrain product changes, even highly valuable changes. Some of the reasons why activity-based planning leads to schedule overruns include

• Activities don’t finish early.

• Lateness is passed down the schedule.

• Activities are not independent.

Each of these problems is described in the following sections.

Activities Don’t Finish Early

A few years ago I had two main projects that needed my time. I was programming some interesting new features for a product. I also needed to prepare documentation for an ISO 9001 compliance audit. The programming was fun. Writing documents for the compliance audit wasn’t. Not surprisingly, I managed to expand the scope of the programming work so that it filled almost all my time and left me the bare minimum of time to prepare for the audit.

I’m not the only one who does this. In fact, this behavior is so common that it has a name, Parkinson’s Law (1993), which states:

Work expands so as to fill the time available for its completion.

Parkinson is saying that we take as much time to complete an activity as we think we’ll be allowed. If there’s a Gantt chart hanging on the wall that says an activity is expected to take five days, the programmer assigned to that activity will generally make sure the activity takes the full five days. She may do this by adding a few bells and whistles if it looks like she’ll finish early (a practice known as gold-plating). Or she may split time between the activity and researching some hot new technology she thinks may be useful. What she will not do very often is finish the activity early. In many organizations, if she finishes early, her boss may accuse her of having given a padded estimate. Or her boss may expect her to finish more activities early. Why risk either of these scenarios when a little web surfing can make the activity come in on schedule instead?

When a Gantt chart shows that an activity is expected to take five days, it gives implicit permission to the developer to take up to that long to complete. It is human nature when ahead of that schedule to fill the extra time with other work that we, but perhaps not others, value.

Lateness Is Passed Down the Schedule

Because traditional plans are activity based, in large measure they focus on the dependencies between activities. Consider the Gantt chart shown in Figure 2.1, which shows four activities and their dependencies. An early start for testing requires the fortuitous confluence of these events:

• Coding of the middle tier finishes early, which is influenced by when adding tables to the database is finished.

• Coding of the user interface finishes early.

• The tester is available early.

Figure 2.1. Testing will start late if anything goes worse than planned; it will start early only if everything goes better than planned.

The key here is that even in this simple case there are three things that all must occur for an early start on testing. Although multiple things must occur for testing to start early, any one of the following can cause testing to start late:

• Coding the user interface finishes late.

• Coding the middle tier takes longer than planned to complete and finishes late.

• Coding the middle tier takes the time planned but starts late because adding tables to the database finishes late.

• The tester is unavailable.

In other words, an early start requires a combination of things to go well; a late start can be caused by one thing going wrong.

The problem is compounded because we’ve already established that activities will rarely finish early. This means that activities will start late and that the lateness will get passed down the schedule. Because early completion is rare, it is even more rare that an activity such as testing in Figure 2.1 gets to start early.

Activities Are Not Independent

Activities are said to be independent if the duration of one activity does not influence the duration of another activity. In building a house, the amount of time it takes to excavate the basement is independent of the amount of time it will take to paint the walls. When activities are independent, a late finish on one activity can be offset by an early finish on another. Flipping a coin multiple times is another example of independent activities. A coin that lands on heads on the first flip is no more or less likely to land on heads on the second flip.

Are software development activities independent? Will the variations in completion times tend to balance out? Unfortunately, no. Many software activities are not independent of one another. For example, if I’m writing the client portion of an application and the first screen takes 50% longer than scheduled, there is a good chance that each of the remaining screens is also going to take longer than planned. If the activities of a development effort are not independent, variations in completion time will not balance out.

Many activities in a typical project plan are not independent, yet we continually forget this. When someone is late on the first of a handful of similar items we’ve all heard or given the answer, “Yes, I was late this time, but I’ll make it up on the rest.” This stems from the belief that the knowledge gained from completing the first activity will allow the remaining similar activities to be completed more quickly than called for in the plan. The real knowledge we should gain in a situation like this is that when an activity takes longer than planned, all similar activities are also likely to take longer than planned.

Multitasking Causes Further Delays

A second reason why traditional approaches to planning often fail is multitasking, which is defined as simultaneously working on multiple tasks. Multitasking exacts a horrible toll on productivity. Clark and Wheelwright (1993) studied the effects of multitasking and found that the time an individual spends on value-adding work drops rapidly when the individual is working on more than two tasks. This is illustrated in Figure 2.2, which is based on their results.

Figure 2.2. Effect of multitasking on productivity.

Logically, it makes sense that multitasking helps when you have two things to work on; if you become blocked on one, you can switch to the other. It is also logical that Figure 2.2 shows a rapid decline in time spent on value-adding tasks after a second task. We’re rarely blocked on more than one task at a time; and if we’re working on three or more concurrent tasks, the time spent switching among them becomes a much more tangible cost and burden.

Multitasking often becomes an issue once a project starts to have some activities finish late. At that point dependencies between activities become critical. A developer waiting on the work of another developer will ask that developer to deliver just a subset of work so that he may continue. Suppose I am to spend ten days working on some database changes, then ten days implementing an application programming interface (API) for accessing the database, and then ten days developing a user interface. This is illustrated in the top half of Figure 2.3. Your work is held up until you get the API from me. You ask me to do just enough of the API work so that you can get started. Similarly, the tester asks me to do just enough of the user interface so that she can begin automating tests. I agree, and my schedule becomes as shown in the bottom of Figure 2.3.

Figure 2.3. Multitasking extends the completion date of work and leaves work in process longer.

This often gives the illusion of speed; but as shown in Figure 2.3, my database and API work finish later than originally planned. This is almost certain to ripple through and affect further activities in the plan. Additionally, in this example, each of the desired units of work remains in process for twenty days rather than ten, as was the case when the work was done serially.

To make matters worse, Figure 2.3 assumes that I am not slowed by switching among these activities more frequently. The Clark and Wheelwright study indicates that a loss in productivity will occur.

Multitasking becomes a problem on a traditionally planned project for two primary reasons. First, work is typically assigned well in advance of when the work will begin, and it is impossible to allocate work efficiently in advance. Assigning work to individuals rather than to groups exacerbates the problem. Second, it encourages focusing on achieving a high level of utilization of all individuals on the project rather than on maintaining sufficient slack to cope with the inherent variability in typical project tasks. Loading everyone to 100% of capacity has the same effect as loading a highway to 100% of capacity: No one can make any progress.

Features Are Not Developed by Priority

A third reason why traditional planning fails to lead consistently to high-value products is because the work described by the plan is not prioritized by its value to the users and customer. Many traditional plans are created with the assumption that all identified activities will be completed. This means that work is typically prioritized and sequenced for the convenience of the development team.

Traditional thinking says that if all work will be completed, project customers have no preference about the sequence in which that work is done. This leads to the development team’s working on features in what appears to the customer to be a relatively haphazard order. Then, with the end of the project approaching, the team scrambles to meet the schedule by dropping features. Because there was no attempt to work on features in order of priority, some of the features dropped are of greater value than those that are delivered.

We Ignore Uncertainty

A fourth shortcoming with traditional approaches to planning is the failure to acknowledge uncertainty. We ignore uncertainty about the product and assume that the initial requirements analysis led to a complete and perfect specification of the product. We assume that users will not change their minds, refine their opinions, or come up with new needs during the period covered by the plan.

Similarly, we ignore uncertainty about how we will build the product and pretend we can assign precise estimates (“two weeks”) to imprecise work. We cannot hope to identify every activity that will be needed in the course of a project. Yet we often fail to acknowledge this in the plans we create.

Even with all this uncertainty, schedules are often expressed as a single, unqualified date: “We will ship on June 30,” for example. During the earliest part of a project we are the most uncertain. The estimates we give should reflect our uncertainty. One way of doing this is by expressing the end date as a range: “We’ll ship sometime between June and August,” for example. As the project progresses and as uncertainty and risk are removed from the project, estimates can be refined and made more precise. This was the point of the cone of uncertainty in Chapter 1, “The Purpose of Planning.”

The best way of dealing with uncertainty is to iterate. To reduce uncertainty about what the product should be, work in short iterations, and show (or, ideally, give) working software to users every few weeks. Uncertainty about how to develop the product is similarly reduced by iterating. For example, missing tasks can be added to plans, bad estimates can be corrected, and so on. In this way, the focus shifts from the plan to the planning.

Estimates Become Commitments

Embedded within every estimate is a probability that the work will be completed in the estimated time. Suppose your team has been asked to develop a new high-end word processor. The probability of finishing this by the end of the week is 0%. The probability of finishing it in ten years is 100%. If I ask you for an estimate, and you tell me the end of the week, that estimate comes with a probability of 0%. If the estimate you give me is ten years, that estimate comes with a probability of 100%. Each estimate between the end of the week and ten years from now comes with its own probability between 0% and 100% (Armour 2002).

A problem with traditional planning can arise if the project team or its stakeholders equate estimating with committing. As Phillip Armour (2002) points out, an estimate is a probability, and a commitment cannot be made to a probability. Commitments are made to dates. Normally the date that a team is asked (or told) to commit to is one to which they would assign a less-than-100% probability. Before making such a commitment, the team needs to assess a variety of business factors and risks. It is important that they be given this opportunity, and that every estimate does not become an implicit commitment.

Summary

After looking through this list of problems with traditional approaches to planning, it’s no wonder so many projects are disappointing. Activity-based planning distracts our attention from features, which are the true unit of customer value. A variety of problems then leads to the likelihood of delivering late against a schedule derived from an activity-based plan. With good intentions, project participants view multitasking as a possible cure but are eventually forced even further behind schedule because of the hidden costs of multitasking. When the project schedule runs out of time, features are inevitably dropped. Because features are developed in the order deemed most efficient by the developers, the dropped features are not necessarily those with the lowest value to users.

Ignoring uncertainty about exactly what users will eventually want can lead to completing a project on schedule but without including important capabilities that were identified after the plan was created. Also ignoring uncertainty about how the product will be developed leads to missed activities in the project plan. This in turn increases the likelihood that the project will be late, that features will be dropped at the end, or that inappropriate quality trade-offs may be made.

Many organizations confuse estimates with commitments. As soon as a team expresses an estimate, they are forced to commit to it.

Discussion Questions

1. What problems result from plans being based on activities rather than deliverable features?

2. In your current environment, is an estimate the same as a commitment? What problems does this cause? What could you do to change this misperception?

3. In what ways does multitasking affect your current project? How could you reduce that impact?