Find opportunities for improvement in software development by first figuring out which problems are development problems. XP isn't intended to solve marketing, sales, or management problems. It's not that non-software bottlenecks aren't important, it's just that XP doesn't spread that thin. There may be ways to crank the knobs to 10 in other areas too, but that is outside the scope of XP. XP is a coherent body of values, principles, and practices for addressing software development problems. We need a way to look at software development as a whole.
One approach to looking at the throughput of entire systems is the Theory of Constraints. I'll illustrate the theory with an example from my own laundry room (Figure 9). My washer takes forty-five minutes to clean clothes, my dryer takes ninety minutes to dry them, and folding the clothes takes fifteen minutes.
The bottleneck in this system is drying. If I get two washers, I won't get any more clothes finished. I'll temporarily get more clothes washed, but then I'll have to contend with the stacks of wet clothes all over the place, and I'll probably get fewer clothes finished. If I want to get more clothes finished, I have no choice but to do something about drying.
The Theory of Constraints says that in any system there is one constraint at a time (occasionally two). To improve overall system throughput you have to first find the constraint; make sure it is working full speed; then find ways of either increasing the capacity of the constraint, offloading some of the work onto non-constraints, or eliminating the constraint entirely.
How do you find the constraint in a system? Work piles up in front of the constraint. There are not piles of dried clothes sitting waiting to be folded; there are piles of wet clothes sitting waiting to be dried.
The dryer is the constraint. To make sure it is working at full capacity, I turn on that annoying buzzer that tells me when a dryer load is finished. When I hear the buzzer, I shift a load. I don't need the buzzer on the washer because I only start a load when I shift one out of the dryer. Work is pulled through the system based on actual demand, not pushed through the system based on projected demand.
If I still need more clothes laundered, I need to increase the capacity of the drying step. This could be done by buying a bigger dryer or by speeding drying by keeping the dryer vent unclogged. I could also offload drying work by buying a new washing machine that spins clothes faster, so they don't take as long in the dryer. Or, I could take the clothes outside and hang them in the sun, as shown in Figure 10.
Now the constraint has moved either to the washer or the folding table. The Theory of Constraints says there is always a constraint. When we eliminate one constraint we create another. Micro-optimization is never enough. To improve our results we must look at the whole situation before deciding what to change.
In software, the constraints in the system need to be identified too. Figure 11 is the familiar, often ineffective, waterfall software development style.
Even if I know how long each of the steps takes I still can't identify the bottleneck. To find the bottleneck, I look for work piling up. If the ERD covers lots of features that get dropped late in implementation because there is too much to do, I suspect the implementation process. If many features are implemented, but waiting to be integrated and deployed, I suspect that the integration process is the constraint.
If integration is the constraint, I first look to make sure integration is going as smoothly as it can given its inputs and its environment. Perhaps people could be shifted from implementation to integration. I have seen one overwhelmed deployment person holding up the efforts of twenty programmers. Asking a few programmers to help with deployment would have helped overall throughput.
Once integration is working smoothly, I look for ways to offload integration work upstream, perhaps by automating tests during implementation. This might slow implementation but improve overall system throughput (more likely, it would make implementation faster, but that's a story for a different chapter).
The point made above about work being pulled through the system is true in software as well. The “push” model of development (Figure 12) is to pile up requirements to be done, then designs to be implemented, then code to be integrated and tested; culminating in an aptly-named big bang integration. XP uses a “pull” model (Figure 13). Stories are specified in detail immediately before they are implemented. The tests are pulled from the specifications. The programming interface is designed to match the needs of the test. The code is written to match the tests and the interface. The design is refined to match the needs of the code as written. This leads to a decision about which story to specify next. In the meantime, the rest of the stories remain on the wall until chosen for implementation.
The Theory of Constraints shares with other theories of organizational change the assumption that the whole organization is focused on overall throughput, not on micro-optimization. If everyone is trying to make sure his function is not seen as the constraint, no change will happen. If the developers say, “Yes, writing automated tests would be 'A Good Thing', but it will slow me down and I'm overloaded as it is,” then nothing will change regardless of how beneficial the change would be to the organization and thus to the individuals in it. The reward system and culture need to align with overall throughput instead of individual productivity for the change to stick.
A weakness of the Theory of Constraints is that it is a model: a map, not the territory. People develop software; they are not boxes in a model. The closer an organization is to efficient but superficially chaotic communication the less accurate the mapping will be into the world of the Theory of Constraints and back again. Software development is a human process not a factory. However, the Theory of Constraints is a good way to become aware of your process. Draw your current process as a series of linked activities and look for where work piles up.
The Theory of Constraints can help find bottlenecks, but what if the bottleneck has nothing to do with software? One of my clients deploys non-safety-critical software on airplanes. Their first partner took as long as four months to deploy finished software on the planes because it would only deploy the software when the planes came in for service. My client picked a second partner who had the infrastructure to deploy new software every day, eliminating that constraint (and moving it to marketing). If the bottleneck exists outside of software development, the answer must come from outside of software development.
I am sometimes tempted to expand the scope of XP to include issues like choosing business partners. Sometimes I'm afraid that if XP doesn't cover a wide range of problems, people won't see it as important. In clearer moments, I believe that XP's value will be more apparent to business with a clear, narrow scope.
Sometimes XP can facilitate shifting the constraint out of software development completely, so XP will have ripple effects throughout an organization applying it. For example, you begin having weekly meetings where the business-focused part of the team (product managers and customers, for instance) chooses the features to be added that week. The programmers may learn to go fast enough that product marketing can't specify features fast enough. The constraint has shifted out of software development.
Here's a sad but repeated story: a development team begins applying XP, dramatically improves quality and productivity, but then is disbanded, its leaders fired and the rest of the team scattered. Why does this happen? From the Theory of Constraint perspective, the team's improved performance shifted the constraint elsewhere in the organization. The new constraint (e.g. marketing, who can't decide what they want fast enough) doesn't like the spotlight. Nobody actually cares about organizational throughput. The “source” of the turmoil, XP, is blamed and eliminated.
Executive sponsorship and strong relationships with people outside the team are crucial to applying XP, precisely because applying XP will shift the structure of work in the rest of the organization as soon as software development gets its act together. If you don't have executive sponsorship, be prepared to do a better job yourself without recognition or protection.