We're going to be working with a module called coverage.py, which is—unsurprisingly—a code coverage tool for Python.

Since coverage.py isn't built in to Python, we're going to need to download and install it. You can download the latest version from the Python Package Index at http://pypi.python.org/pypi/coverage, but it will probably be easier simply to type the following from the command line:

python3 -m pip install --user coverage

We're going to walk through the steps of using coverage.py here, but if you want more information you can find it on the coverage.py home page at http://nedbatchelder.com/code/coverage/.

We're going to create a little toy code module with tests, and then apply coverage.py to find out how much of the code the tests actually use.

Put the following test code into test_toy.py. There are several problems with these tests, which we'll discuss later, but they ought to run:

from unittest import TestCase
import toy

class test_global_function(TestCase):
    def test_positive(self):
        self.assertEqual(toy.global_function(3), 4)

    def test_negative(self):
        self.assertEqual(toy.global_function(-3), -2)

    def test_large(self):
        self.assertEqual(toy.global_function(2**13), 2**13 + 1)

class test_example_class(TestCase):
    def test_timestwo(self):
        example = toy.Example(5)
        self.assertEqual(example.timestwo(), 10)

What we have here is a couple of TestCase classes with some very basic tests in them. These tests wouldn't be of much use in a real-world situation, but all we need them for is to illustrate how the code coverage tool works.

Put the following code into toy.py. Notice the if __name__ == '__main__' clause at the bottom; we haven't dealt with one of these in a while, so I'll remind you that the code inside that block runs doctest if we were to run the module with Python toy.py:

def global_function(x):
    r"""
    >>> global_function(5)
    6
    """
    return x + 1

class Example:
    def __init__(self, param):
        self.param = param

    def timestwo(self):
        return self.param * 2

    def __repr__(self):
        return 'Example({!r})'.format(self.param)

if __name__ == '__main__':
    import doctest
    doctest.testmod()

Here, we have the code that satisfies the tests we just wrote. Like the tests themselves, this code wouldn't be of much use, but it serves as an illustration.

Go ahead and run Nose. It should find the tests, run them, and report that all is well. The problem is that some of the code isn't ever tested. Let's run the tests again, only this time we'll tell Nose to use coverage.py to measure coverage while it's running the tests:

python -m nose --with-coverage --cover-erase

This should give us an error report that looks like this:

.....
Name    Stmts   Miss  Cover   Missing
-------------------------------------
toy        12      3    75%   16, 19-20
----------------------------------------------------------------------
Ran 5 tests in 0.053s

OK

The dots at the top indicate passing tests, and the OK at the bottom says that the testing procedure worked as expected, but the part in between is new. That's our coverage report. Apparently, our tests only cover three quarters of our code: out of the 12 statement lines in toy.py, three didn't get executed. These lines were 16 and 19 through 20.

When we passed --with-coverage and --cover-erase as command-line parameters to Nose, what did they do? Well, --with-coverage is pretty straightforward: it told Nose to look for coverage.py and to use it while the tests get executed. That's just what we wanted. The second parameter, --cover-erase, tells Nose to forget about any coverage information that was acquired during previous runs. By default, coverage information is aggregated across all of the uses of coverage.py. This allows you to run a set of tests using different testing frameworks or mechanisms, and then check the cumulative coverage. You still want to erase the data from previous test runs at the beginning of this process, though, and the --cover-erase command line is how you tell Nose to tell coverage.py that you're starting a new.

Another useful Nose command-line option is --cover-package=PACKAGE, which limits the coverage report to the specific package you're interested in. It didn't show up in our toy because we didn't import anything, but normally the coverage report includes every module or package that has code executed while your tests are running. The percentage of the standard library that is covered by your tests is usually not useful information. It can be convenient to limit the report to the things you actually want to know.

So, back to our coverage report. The missing lines were line 16 and lines 19 through 20. Looking back at our code, we see that line 16 is the __repr__ method. We really should have tested that, so the coverage check has revealed a hole in our tests that we should fix. Lines 19 and 20 are just code to run doctest, though. They're not something that we ought to be using under production conditions, so we can just ignore that coverage hole.

Code coverage can't detect problems with the tests themselves, in most cases. In the previous test code, the test for the timestwo method violates the isolation of units and invokes two different methods of example_class. Since one of the methods is the constructor, this might be acceptable, but the coverage checker isn't in a position to even see that there might be a problem. All it saw was more lines of code being covered. That's not a problem—it's how a coverage checker ought to work—but it's something to keep in mind. Coverage is useful, but high coverage doesn't equal good tests.