Code coverage is an important metric by which the extent of testing is often quantified. The main assumption behind coverage is that if a branch or a statement contains a flaw, it cannot be detected unless a test at least executes that statement or branch (and obtains an output contrary to the normal expectation). It can be argued that higher coverage should lead to detection of more flaws in the code and, if they are fixed, to better release quality. Although this assumption is widely believed, there is little evidence to show that higher code coverage results in fewer failures.
There are a number of potential flaws with the argument that higher code coverage leads to better quality. First, the coverage measure reflects the percent of statements covered but does not consider whether these statements are likely to contain a flaw. Therefore, it is possible to create two test sets with the same coverage measure but a markedly distinct ability to detect post-release defects. Second, the fact that a statement or a branch was executed does not imply that all possible data values have been exercised by the test. The customer/usage scenario could be totally different from what the tester perceives it to be. This is of particular concern for systems with simple control flow that can be easily covered with a few test cases. In our study on code coverage, based on qualitative interviews with engineers, the reasons for observing such a trend was attributed to several factors [Mockus et al. 2009]:
Code covered is not correct code.
Developers inherently know that certain binaries are complex and will tend to get very high coverage in those binaries. But these binaries may be the ones most used in the system, which can lead to more failures being found in them, i.e., code coverage and usage profiles might not match.
Complexity should tie into code coverage. For example, obtaining a code coverage of 80% on a binary with cyclomatic complexity 1000 is more difficult than getting a coverage of 80% on a binary with cyclomatic complexity 10. The complexity values tied into the code coverage helps exploit the efficacy of the code coverage measures.
Despite these flaws, coverage appears to be a promising measure due to its wide use in practice. However, there has been little published about the relationship between code coverage and quality. Most studies performed to date on this relationship suffer from issues of internal validity (studies of 100-line programs). In order to find evidence of the relationship between code coverage and quality, we mapped the branch and block coverage values for Windows Vista (40+ million lines of code and several thousand binaries and engineers) and field failures for Windows Vista six months after release. We observe a weak positive correlation between coverage and quality and a low prediction precision and recall (precision 83.8% and recall 54.8%; for further reading, please refer to [Mockus et al. 2009]). Our investigation of this straightforward and simple hypothesis leads us to believe that code coverage is not best used in isolation, but instead needs to be combined with other metrics, such as code churn, complexity, etc.