4
Why Quality Pays

As described in Chapter 1, quality must come first, even before the schedule. In the subsequent chapters, we have discussed some examples of poor software management. These stories all had an underlying quality theme. Testing was expensive and took a lot of time, the finished products were often highly defective, and the plans and data needed to be accurate and precise. These issues all concern quality.

Software quality is a growing problem and, based on current trends, these quality problems will only get worse—at least until executives start demanding quality software. There are three reasons to insist that software quality be measured and managed:

1. Poor-quality software can cause major property damage and even kill people.

2. Quality work saves time and money.

3. If you don’t manage software quality, nobody else will.

Poor-Quality Software can be Life-Threatening

As software becomes more widely used, the reports of problems are increasing. Although relatively few fatalities have occurred to date, many software defects have caused major disasters. Some examples are the following:

• Six people were killed when an incorrectly programmed medical instrument overexposed cancer patients to radiation.

• A manufacturing worker was killed when a software defect caused a production machine to crush him.

• A software error in the Ford Explorer engine controls limited vehicle speed to 110 mph instead of the specified 99 mph. At 110 mph, the Firestone tires on Ford Explorers had a rated life of ten minutes.

• At Bendix Brakes, a software design omission caused improperly installed brakes to release momentarily at speeds of less than 15 mph. No one was actually hurt, but the recall cost tens of millions of dollars.

• A software design mistake in the Mars orbiter caused this $125 million spacecraft to crash into the planet and be destroyed.

• Poorly designed software caused the pilot of a commercial airliner to mistakenly fly into a mountain, killing all the passengers and crew.

• A three-line programming error in an Intel manufacturing process caused a Pentium recall that cost the company over $400 million.

• The Code Red worm, by taking advantage of a common type of software defect, was able to infect 250,000 machines and cause widespread Internet shutdowns in just nine hours. The worldwide cost of preventing further Code Red infections was $1.2 billion.

Quality Work Saves Time And Money

In deciding how to manage software quality, we must first consider the facts. First, while developing programs, even experienced engineers inject a defect about every ten lines of code^* (LOC) [1]. The programmers are not incompetent or lazy, they are just human. All humans make mistakes, but in software, these mistakes result in defects. This means that a modest-sized program of 100,000 LOC typically would start with about 10,000 defects.

* The line-of-code (LOC) measure refers to the text lines in a printed program listing. One line is typically a short statement of 10 to 25 alpha-numeric characters. The KLOC measure refers to thousands of LOC.

The second key fact about software quality is that the cost of finding and fixing defects increases at every step in the development process. For example, Xerox has measured this time, and its data are shown in Figures 4.1 and 4.2 [2]. The defect find and fix times shown range from 3 minutes in code reviews to 25 minutes in inspections and 1,400 minutes in system testing. Engineers do code reviews by reading and analyzing their programs. In inspections, several engineers review a program to find its problems. In unit testing, each programmer tests the small program modules that he or she developed and fixes all of the defects found. In system testing, all of these small program modules are combined and tested as a complete system.

Figure 4.1 Time to find and Fix Defects

Figure 4.2 Time to find and Fix Defects

To appreciate the significance of these times, consider the time required to find and fix 10,000 defects. Using the Xerox data, the defect-removal time would be 4,170 hours with inspections, 5,330 hours for unit testing, and 234,000 hours for system test. Note that 234,000 hours is over 100 engineering years of work. Clearly, selecting the proper defect-removal method makes a big difference in both software development time and in engineering costs. As we shall see later in this chapter, defect-removal methods also have a big impact on the quality of the finished product.

The third key fact about software quality is that the engineer who developed the program is best able to find and fix its defects. This is demonstrated by the Xerox data shown in Figure 4.2. Here, in a review, an engineer personally reviews his or her program module to find and fix its defects. Instead of 25, 32, or 1,405 minutes per defect, the engineer who wrote the program module can generally find and fix its defects in just a few minutes each. SEI data show that the most efficient way to find and fix defects is for engineers to read and carefully analyze their programs. This takes them about an hour for every 100 lines of code. Engineers who do such careful reviews typically find and fix 5 to 10 defects per review hour. Therefore, counting review time, the average defect-removal time in code reviews is between 6 and 12 minutes per defect [1]. With code reviews, the engineers can find and fix 10,000 defects in only about 1,000 hours instead of about 4,000 hours with inspections, 5,000 hours with unit testing, or over 200,000 hours in system test.

Since reviews are so effective, you might wonder why everyone doesn’t use them. The reason is that good code reviews require disciplined methods and special training. The engineers must gather defect data and use these data to help them find and fix the defects. Because people tend to make the same mistakes repetitively, these data show engineers what kinds of defects to look for. With few exceptions, engineers are not trained in such quality methods and do not believe that reviews and defect data would help them. Methods for convincing engineers to use such methods are discussed further in Chapter 6.

Defect-Removal Strategies

In the normal process of translating a program into testable form, about half of its defects are removed by a program called a compiler. Compilers are much like spell-checkers in that they identify most of the spelling, punctuation, and other syntax-type defects. That means that the compiler would find about half of the 10,000 defects in a typical 100,000 LOC program. To find the remaining 5,000 defects, the three strategies are to test; to inspect and test; and to review, inspect, and test. Since reviews and inspections typically do not find all of a program’s defects, reviewed programs are inspected and inspected programs are tested to find even more of the defects.

Figure 4.3 shows the times and effectiveness of the three strategies for finding and fixing the defects in a 100,000 LOC software system. As you can see, the times required and the finished product defects are dramatically different. For example, the test strategy takes 23,400 hours, while the inspect/test strategy takes only 11,000 hours. Even less time is required by the review/inspect/test strategy, which takes about 6,000 hours, or only about one-quarter as long as the test strategy. The time savings come from finding more of the defects in reviews and inspections and fewer in system test.

Perhaps the most significant point about the test strategy is its ineffectiveness. Even after spending over ten engineer-years of work, the test strategy produces products with lots of defects. The inspection strategy takes half the time and leaves less than half as many defects, while the review strategy takes a quarter of the time and leaves one-eighth as many defects. Each of the defects remaining in a shipped product could damage your prod-uct’s users and cost you hundreds of hours to find and fix.

Figure 4.3 Defect-Removal Strategies

The reason that testing takes so long is that a test reveals only the symptoms of a problem. Then the engineers must determine where the problem is in the system and its underlying cause. This is particularly difficult when there are multiple problems. Then the engineer must isolate the symptoms for the particular problem, identify the troublesome module, find the mistake, and then fix that mistake. After each defect is fixed, the fix must be installed in the system, and the system retested. Because each of these steps can take an engineer an hour to a week or more, the time to find and fix a few hundred defects can be many months.

Quality Work is More Predictable

Another benefit of quality work is a direct consequence of the unpredictability of testing. Since the time to find and fix each defect can vary from a few hours to several weeks, testing time will vary widely when products enter test with more than a very few defects. This means that any project with a large amount of testing work will be nearly impossible to plan and track. Therefore, if you want accurate plans and reliable commitments, you must insist on quality work.

If you don’t Manage Software Quality, nobody else will

An IBM example shows why executives must sponsor quality programs. In 1983, the president of IBM was concerned about the quality of IBM’s software. While software quality was not yet a crisis, software maintenance costs had reached $250,000,000 a year. The president asked me to be IBM’s director of software quality and process. My charter was to fix the software quality problem.

Some years earlier, management had issued a policy called Corporate Instruction 105, or CI-105. It required that the quality of every new product be better than the best previous IBM product of the same general type. The hardware groups had responded to CI-105 and made very substantial quality improvements. Most of these product managers had improved quality by ten or more times, and some had improved it as much as 1,000 times. However, the software community had not responded to CI-105. One division president even met with corporate management to argue that CI-105 could not be used for software. He said that software quality couldn’t be measured, that software was inherently defective, and that they could not afford to do more testing. IBM’s president and the other executives disagreed, and the division president was soon replaced.

To apply CI-105 to software we first needed a software quality measure. We established a technical task force to define this measure and assigned several members of my quality staff. We also included one or two engineers from each of the major software laboratories. After several months of work, this group had reviewed many possible measures. It was soon apparent, however, that the laboratory members were not authorized to agree to any measure. Since the experts from my group had learned enough from the task force to propose a usable quality measure, we decided to disband the task force.

After talking with several laboratory directors, I concluded that the software laboratories would not voluntarily agree with any CI-105 quality measure, so I had my staff draft our own proposal. We reviewed this proposal with corporate and division staffs and then sent it to the software laboratories. We told them either to agree with this measure or to propose a better one. While several laboratories reluctantly agreed, the largest and most important laboratory refused. The director disagreed with the idea of measuring software quality and would not accept any measure.

I scheduled a meeting for this director to explain to IBM’s president why he did not want to measure software quality and what he proposed to do instead. When I told the director about this upcoming meeting, he became very upset. Rather than explain his position to IBM’s president, however, he agreed with our proposal. We then quickly obtained agreement from the other laboratories.

At the time we established the CI-105 quality measure, we also defined how the measure would be used. The product manager was required to make a quality plan before a new or enhanced software product could be announced. After product shipment, my staff gathered defect data from the field engineering division every month, sorted these data by product, calculated the CI-105 measures, and posted the planned and actual data on an executive quality reporting system. Over the next several years, the measured quality of IBM’s new software products improved by an average of 35% a year, or nearly ten times in five years.

I did not appreciate the significance of the CI-105 software-quality initiative until I later heard that marketing representatives at Amdahl Corporation, an IBM competitor, were reporting that IBM’s software quality had improved dramatically. This was significant because Amdahl’s customers used IBM programs to run their systems. When your competitors notice that your product quality is improving, you are making progress.

Summary and Conclusions

The following six principal points are made in this chapter:

1. Defective software is expensive and can even be life-threatening.

2. Software engineers inject defects because they are human.

3. The cost to find and fix a defect increases exponentially as development progresses.

4. The engineer who developed a program is best able to find and fix its defects.

5. Effective defect-removal methods require special training.

6. If you do not manage software quality, nobody else will.

References

1. W. S. Humphrey, A Discipline for Software Engineering. Reading, MA: Addison-Wesley, 1995.

2. Allen Willett, a PSP instructor and TSP coach at Xerox, private communication.