Coverage analysis is measuring which lines in a program are run and which lines aren't. This is type of analysis is also known as 'code coverage', or more simply 'coverage'.

A coverage analyzer can be used while running a system in production, but what are the pros and cons, if we used it this way? What about using a coverage analyzer when running test suites? What benefits would this approach provide compared to checking systems in production?

Coverage helps us to see if we are adequately testing our system. But it must be performed with a certain amount of skepticism. This is because, even if we achieve 100 percent coverage, meaning every line of our system was exercised, in no way does this guarantee us having no bugs. A quick example involves a code we write and what it processes is the return value from a system call. What if there are three possible values, but we only handle two of them? We may write two test cases covering our handling of it, and this could certainly achieve 100 percent statement coverage. However, it doesn't mean we have handled the third possible return value; thus, leaving us with a potentially undiscovered bug. 100 percent code coverage can also be obtained by condition coverage but may not be achieved with statement coverage. The kind of coverage we are planning to target should be clear.

Another key point is that not all testing is aimed at bug fixing. Another key purpose is to make sure that the application meets our customer's needs. This means that, even if we have 100 percent code coverage, we can't guarantee that we are covering all the scenarios expected by our users. This is the difference between 'building it right' and 'building the right thing'.

In this chapter, we will explore various recipes to build a network management application, run coverage tools, and harvest the results. We will discuss how coverage can introduce noise, and show us more than we need to know, as well as introduce performance issues when it instruments our code. We will also see how to trim out information we don't need to get a concise, targeted view of things.

This chapter uses several third-party tools in many recipes.