Using the Code Coverage Metric Effectively

Code coverage—the percentage of lines of code exercised by your unit tests—is the new “lines of code” metric, sure to be abused by lazy managers. The naive interpretation of the metric is that 100 percent means you have comprehensive coverage, and 0 percent indicates that you haven’t even tried. You can find a number of code coverage tools for C++; most are not free. COVTOOL (http://covtool.sourceforge.net) is one open source test coverage tool.

A good coverage tool will also annotate your source, showing you specifically which lines of code get executed when you run your tests. The real value in measuring coverage lies here: knowing which lines of code aren’t covered can help you determine where you need to add tests.

If you always follow the simple TDD cycle to drive your code, your coverage metric will approach 100 percent. (It might not ever be 100 percent because of some limitations in the tools and their ability to discern how certain code elements are used. That’s OK.) But you already knew that—if you only write code in the presence of a failing test and you never write more code than necessary to pass the test, you have 100 percent coverage by definition. I personally don’t usually bother with code coverage tools for this reason. Still, if you’re struggling with always following the habit of writing a test first or if you tend to write more code than needed to pass a test, a coverage tool can provide you with valuable feedback.

A system-wide, average coverage number less than 90 percent tells you the codebase wasn’t (fully or at all) developed using TDD, and nothing more. A coverage number of 90 percent or more similarly tells you little—it usually indicates that the codebase was built using TDD, but it’s also possible to write integration tests that blast through a large amount of code. More typically, the coverage across a non-test-driven system is considerably less (and in my experience, rarely more than 70 percent). It’s hard to cover every last branch with integration tests, and it’s also hard to write comprehensive unit tests after you’ve already written the code.

It’s also possible to do TDD, write focused unit tests, attain coverage levels in the 90+ percentile range, and create tests that are difficult to maintain. In other words, high numbers may look good but might not tell the whole story.

Never set coverage numbers as goals. Managers who insist on high coverage numbers get what they ask for—high coverage numbers and little else useful.