On the one hand, it is frustrating that so few studies of software faults have been published to guide researchers in finding ways of detecting, ameliorating, or preventing these faults from happening. On the other hand, it is not at all surprising that projects are reluctant to make such sensitive data public because of internal politics or external competitiveness in software-intensive businesses.
Despite this paucity of studies and the reluctance of companies to make data available, there is a set of landmark studies about software faults that provide useful foundations for product and process improvements. Endres [Endres 1975], Schneidewind and Hoffmann [Schneidewind and Hoffmann 1979], and Glass [Glass 1981] reported on various fault analyses of software development. A weakness in their work is that they do not delineate interface faults as a specific category.
Thayer, Lipow, and Nelson [Thayer et al. 1978] and Bowen [Bowen 1980] provide extensive categorization of faults, but with a relatively narrow view of interface faults. Basili and Perricone [Basili and Perricone 1984] offer the most comprehensive study of problems encountered in the development phase of a medium-scale system, reporting data on the fault, the number of components affected, the type of the fault, and the effort required to correct the fault. Interface faults were the largest class of faults (39% of the faults).
We note, however, that none of these studies address the kinds of problems that arise in very large-scale software developments, nor do they address the evolutionary phase of developments. Perry and Evangelist [Perry and Evangelist 1985], [Perry and Evangelist 1987] were the first to address fault studies in the evolution of a large real-time system. An extremely important factor in this study is the fact that interface faults were by far the overwhelming and dominant faults (68% of the faults). An important question that was left unanswered was whether these interface faults were the easy or the hard ones to find and fix.
The distinction between an evolutionary software system release and an initial development release is a critical one. In the latter case, the design and implementation choices are much less constrained than in evolutionary development. In the former, you have to make changes to an existing system, and so the choices are far more constrained and there are many more difficulties in understanding the implications of changes. As the evolutionary development part of a system’s life cycle is far greater than its initial development, so too are studies of the faults in that evolutionary part much more important.
In this study, we take a detailed look at one specific release of one ultra-reliable, ultra-large-scale, real-time system rather than a more superficial look at several more moderately sized systems in several domains. The advantage of this approach is that we gain a deeper understanding of the system and its problems. The disadvantage is that we are less able to generalize our results compared to the latter course. This type of trade-off is often encountered in empirical studies.
As we will see, however, this deeper look provides us with a number of practical and useful insights. For example, it is commonly accepted wisdom that “once a bug is found, it is easy to fix.” Unfortunately, our data contradicts this “common wisdom.” Or, in the case of the unanswered question about interface faults, our data supports our original but unsubstantiated intuition that interface faults are harder to fix than implementation faults.