Table 25-3 provides a cross-tabulation of the difficulty in finding and fixing the design and coding faults. Of these faults, 78% took five days or less to fix. In general, the easier-to-find faults were easier to fix; the more difficult-to-find faults were more difficult to fix.

One of the interesting things about Chi-Square analysis is that it is based on the difference between expected and observed values of the paired data. The expected value in this case is the product of the observed find value and the observed fix value. If the two sets of data are independent of each other, the expected percentages will match or be very close to the observed percentages; otherwise, the two sets of data are not independent.

The first row of data is the observed percentages of how long it took to fix the MR; the first column is the observed percentages of how hard it was to find/duplicate the problem. The expected value of easy to find and fixable in a day or less is 67.1% × 30.1% = 20.2%, whereas the actually observed value of 23.7% is 17% more than that expected value.

There were more faults that were easy to find and took less than one day to fix than were expected by the Chi-Square analysis. Interestingly, there were fewer than expected easy to find faults (expected: 12%) that took 6 to 30 days to fix (observed: 10%).

Although the coordinates of the effort to find and fix the faults are non-comparable, we note that the following relationship is suggestive. Collapsing the previous table yields an interesting insight in Table 25-4 that seems counter to the common wisdom that says “once you have found the problem, it is easy to fix it.” There is a significant number of “easy/moderate to find” faults that require a relatively long time to fix.

Table 25-5 shows the fault types of the MRs as ordered by their frequency in the survey independent of any other factors. For the sake of brevity in the subsequent tables, we use the fault type number to represent the fault types.

The first five fault types account for 60% of the faults. That “internal functionality” is the leading fault by such a large margin is somewhat surprising; that “interface complexity” is such a significant problem is not surprising at all. However, that the first five fault types are leading faults is consistent with the nature of the evolution of the system. Adding significant amounts of new functionality to a system easily accounts for problems with “internal functionality,” “low-level logic,” and “external functionality.”

The fact that the system is a very large, complicated real-time system easily accounts for the fact that there are problems with “interface complexity,” “unexpected dependencies” and design/code complexity,” “change coordination,” and “concurrent work.”

C has well-known “language pitfalls” that account for the rank of that fault in the middle of the set. Similarly, “race conditions” are a reasonably significant problem because of the lack of suitable language facilities in C.

That “performance” faults are relatively insignificant is probably due to the fact that this is not an early release of the system, and performance was always a significant concern of code inspections.

There are two interesting relationships to consider in the ordering of the various faults: the effect that the difficulty in finding the faults has on the ordering and the effect that the difficulty of fixing the faults has on the ordering. The purpose of weighting is to provide an adjustment to the observed frequency by how easy or hard the faults are to find or to fix. From the standpoint of “getting the most bang for the buck,” the frequency of a fault is a good prima facie indicator of the importance of one fault relative to another. Table 25-5 shows the fault types ordered by frequency.

Table 25-6 is an attempt to capture the weighted difficulty of finding the various faults. The weighting is done by multiplying the proportion of observed values for each fault with multiplicative weights of 1, 2, 3, and 4 for each find category, respectively, and summing the results.

Obviously it would have been better to have had some duration assigned to the effort to find faults and then correlated the weighting with those durations, as we do subsequently in weighting by effort to fix faults. The weights used are intended to be suggestive, not definitive. We experimented with several different weightings, and the results were pretty much the same. Thus we used the simplest approach.

Better yet would have been effort data associated with each MR that could be used to get a more realistic picture of actual difficulty. But this type of data is seldom available, and an approximation is needed instead.

For example, if a fault was easy to find in 66% of the cases, moderate in 23%, difficult in 11%, and very difficult in 0%, the weight is 145 = (66 * 1) + (23 * 2) + (11 * 3) + (0 * 4). Table 25-6 shows the fault types weighted by difficulty to find, from easiest to most difficult.

Typically, performance faults and race conditions are very difficult to isolate and reproduce. We would expect that “code complexity” and “error handling” faults also would be difficult to find and reproduce. Not surprisingly, “language pitfalls” and “interface complexity” are reasonably hard to detect.

In the Chi-Square analysis, “internal functionality,” “unexpected dependencies,” and “other” tended to be easier to find than expected. “Code complexity” and “performance” tended to be harder to find than expected. There tended to be more significant deviations where the population was larger.

If we weight the proportions by multiplying the number of occurrences of each fault by its weight from Table 25-5 and dividing by the total weighted number of occurrences, we get only a slight change in the ordering of the faults, with “internal functionality,” “code complexity,” and “race conditions” (faults 5, 11, and 13) changing slightly more than the rest of the faults.

Table 25-7 represents the results of weighting the difficulty of fixing the various faults by factoring in the actual time needed to fix the faults. The multiplicative scheme uses the values 1, 3, 15, and 30 for the four average times in fixing a fault. The calculations are performed as in the example of weighting the difficulty of finding the faults.

The weighting according to the difficulty in fixing the fault causes some interesting shifts in the ordering. “Language pitfalls,” “low-level logic,” and “internal functionality” (faults 1, 3, and 5) drop significantly in their relative importance. This coincides with one’s intuition about these kinds of faults. “Design/code complexity,” “resource allocation,” and “unexpected dependencies” (faults 11, 15, and 20) rise significantly in their relative importance; “interface complexity,” “race conditions,” and “performance” (faults 10, 13, 14) also rise, but not significantly so.

Table 25-8 shows the top fix-weighted faults. According to our weighting schemes, these four faults account for 55.2% of the effort expended to fix all the faults and 51% of the effort to find them, but represent 52.1% of the faults by frequency count. Collectively, they are somewhat harder to fix than rest of the faults and slightly easier to find. We again note that although the two scales are not strictly comparable, the comparison is an interesting one nonetheless.

In the Chi-Square analysis, “language pitfalls” and “low-level logic” took fewer days to fix than expected. “Interface complexity” and “internal functionality” took 1 to 6 days to fix more often than expected, and “design/code complexity” and “unexpected dependencies” took longer to fix (that is, 6 to over 30 days) than expected. These deviations reinforce our weighted assessment of the effort to fix the faults.

In Table 25-9, we show the underlying causes of the MRs as ordered by their frequency in the survey, independent of any other factors.

The high proportion of “none given” as an underlying cause requires some explanation. One of the reasons for this is that faults such as “language pitfalls,” “low-level logic,” “race conditions,” and “change coordination” tend to be both the fault and the underlying cause (7.8%—or 33% of the faults in the “none given” underlying cause category in Table 25-12 below). In addition, one could easily imagine that some of the faults, such as “interface complexity” and “design/code complexity,” could also be considered both the fault and the underlying cause (3.4%—or 16% of the faults in the “none given” underlying cause category in Table 25-12). On the other hand, we were surprised that no cause was given for a substantial part of the “internal functionality” faults (3.3%—or 16% of the faults in the “none given” category in Table 25-12). One would expect there to be some underlying cause for that particular fault.

Table 25-10 shows the relative difficulty in finding the faults associated with the underlying causes. The resulting ordering is particularly nonintuitive: the MRs with no underlying cause are the second most difficult to find; those submitted under duress are the most difficult to find.

In the Chi-Square analysis of finding underlying causes, faults caused by “lack of knowledge” tended to be easier to find than expected, whereas faults caused by “submitted under duress” tended to be moderately hard to find more often than expected. This latter finding is interesting, as we know very little about faults “submitted under duress.”

In Table 25-11, we weight the underlying causes by the effort to fix the faults represented by the underlying causes. This yields a few shifts in the proportion of effort: “incomplete/omitted design” increased significantly; “unclear requirements” and “incomplete/omitted requirements” increased less significantly; “none” decreased significantly; and “unclear design” and “other” decreased less significantly. However, the relative ordering of the various underlying causes is unchanged.

The relative weighting of the effort to fix these kinds of underlying causes seems to coincide with one’s intuition very nicely.

In the Chi-Square analysis of fixing underlying causes, faults caused by “none given” tended to take less time to fix than expected, whereas faults caused by “incomplete/omitted design” and “submitted under duress” tended to take more time to fix than expected.

In Table 25-12, we present the cross-tabulation of faults and their underlying causes. Faults are represented by the rows, underlying causes by the columns. The numbers in the matrix are the percentages of the total population of faults. Thus, 1.5% of the total faults were fault 1 with the underlying cause 1. The expected number of faults for fault 1 and underlying cause 1 can be computed by multiplying the total faults for each of those categories: 20.5% * 3.5% = .7%. In this example, the actual number of faults was higher than expected.

For the sake of brevity, we consider only the most frequently occurring faults and their major underlying causes. “Incomplete/omitted design” (cause 4) is the primary underlying cause in all of these major faults. “Ambiguous design” (cause 5), “lack of knowledge” (cause 7), and “none given” (cause 1) were also significant contributors to the presence of these faults.

Again, for the sake of brevity, we consider only the most frequently occurring underlying causes and the faults to which they were most applicable.

Table 25-13 shows the means of prevention of the MRs, as ordered by their occurrence independent of any other factors. We note that the means selected may well reflect a particular approach of the responder in selecting one means over another (for example, see the discussion later in this section about formal versus informal means of prevention).

It is interesting to note that the application-specific means of prevention (“application walk-throughs”) is considered the most effective. This selection of application walk-throughs as the most useful means of error prevention appears to confirm the observation of Curtis, Krasner, and Iscoe [Curtis et al. 1988] that a thin spread of application knowledge is the most significant problem in building large systems.

Further, it is worth noting that informal means of prevention rank higher than formal ones. On the one hand, this may reflect the general bias in the United States against formal methods. On the other hand, the informal means are a nontechnical solution to providing the information that may be supplied by formal representations (and which provide a more technical solution with perhaps higher attendant adoption costs).

The level of effort to find the faults for which these are the means of prevention does not change the order found in Table 25-13, with the exception of “requirements/design templates,” which seems to apply to the easier-to-find faults, and “guideline enforcement,” which seems to apply more to the harder-to-find faults.

In the Chi-Square analysis, the relationship between finding faults and preventing them is the most independent of the relationships, reported here with p=.041. “Application walk-throughs” applied to faults that were marginally easier to find than expected, whereas “guideline enforcement” applied to faults that were less easy to find than expected.

In Table 25-14, the means of prevention is weighted by the effort to fix the associated faults.

It is interesting to note that the faults considered to be prevented by training are the hardest to fix. The formal methods also apply to classes of faults that take a long time to fix.

Weighting the means of prevention by effort to fix their corresponding faults yields a few shifts in proportion: “application walk-throughs,” “better test planning,” and “guideline enforcement” decreased in proportion; “expert person/documentation” and “formal requirements” increased in proportion; and “formal interface specifications” and “other” less so. As a result, the ordering changes slightly to 5, 6, 2, 1, 8, 10, 3, 9, 4, 7: “expert person/documentation” and “formal requirements” (numbers 6 and 1) are weighted significantly higher; “requirements/design templates,” “formal interface specifications,” “training,” and “other” (numbers 2, 3, 4, and 10) are less significantly higher; and “guideline enforcement” and “better test planning” (numbers 8 and 9) are significantly lower.

In the Chi-Square analysis, faults prevented by “application walk-throughs,” “guideline enforcement,” and “other” tended to take fewer days to fix than expected, whereas faults prevented by “formal requirements,” “requirements/design templates,” and “expert person/documentation” took longer to fix than expected.

In Table 25-15, we present the cross-tabulation of faults and their means of prevention. Again, the faults are represented by the rows, and the means of prevention are represented by the columns. The data is analogous to the preceding cross-tabulation of faults and underlying causes.

For the sake of brevity, we consider only the most frequently occurring faults and their major means of prevention. “Application walk-throughs” were felt to be an effective means of preventing these most significant faults. “Expert person/documentation,” “formal requirements,” and “formal interface specifications” were also significant means of preventing these faults.

Again, for the sake of brevity, we consider only the most frequently occurring means of prevention and the faults to which they were most applicable. Not surprisingly, these means were most applicable to “internal functionality” and “interface complexity,” the most prevalent faults. Counterintuitively, they are also strongly recommended as applicable to “low-level logic.”

In Table 25-16, it is interesting to note that in the Chi-Square analysis there are lots of deviations (that is, there is a wider variance between the actual values and the expected values in correlating underlying causes and means of prevention). This indicates that there are strong dependencies between the underlying causes and their means of prevention. Intuitively, this type of relationship is just what we would expect.

We first summarize the means of prevention associated with the major underlying causes. “Application walk-throughs,” “expert person/documentation,” and “guideline enforcement” were considered important in addressing these major underlying causes.

The following summarizes the major underlying causes addressed by the most frequently considered means of prevention. “Lack of knowledge,” “none given,” “incomplete/omitted design,” and “ambiguous design” were the major underlying causes for which these means of prevention were considered important. It is somewhat non-intuitive that the “none given” underlying cause category is so prominent as an appropriate target for these primary means of prevention.