Case Study: Testing above_freezing

The first function that we’ll test is above_freezing from ​Testing Your Code Semiautomatically​:

In that section, we ran the example calls from the docstring using doctest. But we’re missing a test: what happens if the temperature is zero? In the next section, we’ll write another version of this function that behaves differently at zero and we’ll discuss how our current set of tests is incomplete.

Choosing Test Cases for above_freezing

Before writing our testing code, we must decide which test cases to use. Function above_freezing takes one argument, a number, so for each test case, we need to choose the value of that argument. There are billions of numbers to choose from and we can’t possibly test them all, so how do we decide which values to use? For above_freezing, there are two categories of numbers: values below freezing and values above freezing. We’ll pick one value from each category to use in our test cases.

Looking at it another way, this particular function returns a Boolean, so we need at least two tests: one that causes the function to return True and another that causes it to return False. In fact, that’s what we already did in our example function calls in the docstring.

In above_freezing’s docstring, the first example call uses 5.2 as the value of the argument, and that value is above freezing so the function should return True. This test case represents the temperatures that are above freezing. We chose that value from among the billions of possible positive floating-point values; any one of them would work just as well. For example, we could have used 100.6, 29, 357.32, or any other number greater than 0 to represent the “above freezing” category. The second example call uses -2, which represents the temperatures that are below freezing. As before, we could have used -16, -294.3, -56.97, or any other value less than 0 to represent the “below freezing” category, but we chose to use -2. Again, our choice is arbitrary.

Are we missing any test case categories? Imagine that we had written our code using the >= operator instead of the > operator:

Both versions of the function produce the expected results for the two docstring examples, but the code is different from before, and it won’t produce the same result in all cases. We neglected to test one category of inputs: temperatures at the freezing mark. Test cases like the one at the freezing mark are often called boundary cases since they lie on the boundary between two different possible behaviors of the function (in this case, between temperatures above freezing and temperatures below freezing). Experience shows that boundary cases are much more likely to contain bugs than other cases, so it’s always worth figuring out what they are and testing them.

Sometimes there are multiple boundary cases. For example, if we had a function that determined which state water was in—solid, liquid, or gas—then the two boundary cases would be the freezing point and the boiling point.

To summarize, the following table shows each category of inputs, the value we chose to represent that category, and the value that we expect the call on the function to return in that case:

Now that all categories of inputs are covered, we need to run the third test. Running the third test in the Python shell reveals that the value returned by above_freezing_v2 isn’t False, which is what we expected:

It took three test cases to cover all the categories of inputs for this function, but three isn’t a magic number. The three tests had to be carefully chosen. If the three tests had all fallen into the same category (say, temperatures above freezing: 5, 70, and 302) they wouldn’t have been sufficient. It’s the quality of the tests that matters, not the quantity.

Testing above_freezing Using unittest

Once you decide which test cases are needed, you can use one of two approaches that you’ve learned about so far to actually test the code. The first is to call the functions and read the results yourself to see if they match what you expected. The second is to run the functions from the docstring using module doctest. The latter approach is preferable because the comparison of the actual value returned by the function to the value we expect to be returned is done by the program and not by a human, so it’s faster and less error prone.

In this section, we’ll introduce another of Python’s modules, unittest. A unit test exercises just one isolated component of a program. Like we did with doctest, we’ll use module unittest to test each function in our module independently from the others. This approach contrasts with system testing, which looks at the behavior of the system as a whole, just as its eventual users will.

In ​Inheritance​, you learned how to write classes that inherit code from others. Now you’ll write test classes that inherit from class unittest.TestCase. Our first test class tests function above_freezing:

The name of our new class is TestAboveFreezing, and it’s saved in the file test_above_freezing.py. The class has three of its own methods, one per each test case. Each test case follows this pattern:

In each test method, there is a call on method assertEqual, which has been inherited from class unittest.TestCase. To assert something is to claim that it is true; here we are asserting that the expected value and the actual value should be equal. Method assertEqual compares its first two arguments (which are the expected return value and the actual return value from calling the function being tested) to see whether they are equal. If they aren’t equal, the third argument, a string, is displayed as part of the failure message.

At the bottom of the file, the call on unittest.main() executes every method that begins with the name test.

When the program in test_above_freezing.py is executed, the following results are produced:

The first line of output has three dots, one dot per test method. A dot indicates that a test was run successfully—that the test case passed.

The summary after the dashed line tells you that unittest found and ran three tests, that it took less than a millisecond to do so, and that everything was successful (OK).

If our faulty function above_freezing_v2 was renamed above_freezing and our test_above_freezing unit test program was rerun, instead of three passes (as indicated by the three dots), there would be two passes and a failure:

The F indicates that a test case failed. The error message tells you that the failure happened in method test_above_freezing_at_zero. The error is an AssertionError, which indicates that when we asserted that the expected and actual value should be equal, we were wrong.

The expression False != True comes from our call on assertEqual: variable expected was False, variable actual was True, and of course those aren’t equal. Additionally, the string that was passed as the third argument to assertEqual is part of that error message: "The temperature is at the freezing mark."

Notice that the three calls on assertEqual were placed in three separate methods. We could have put them all in the same method, but that method would have been considered a single test case. That is, when the module was run, we would see only one result; if any of the three calls on assertEqual failed, the entire test case would have failed. Only when all three passed would we see the coveted dot.

As a rule, each test case you design should be implemented in its own test method.

Now that you’ve seen both doctest and unittest, which should you use? We prefer unittest, for several reasons:

• For large test suites, it is nice to have the testing code in a separate file rather than in a very long docstring.

• Each test case can be in a separate method, so the tests are independent of each other. With doctest, the changes to objects made by one test persist for the subsequent test, so more care needs to be taken to properly set up the objects for each doctest test case to make sure they are independent.

• Because each test case is in a separate method, we can write a docstring that describes the test case tested so that other programmers understand how the test cases differ from each other.

• The third argument to assertEqual is a string that appears as part of the error message produced by a failed test, which is helpful for providing a better description of the test case. With doctest, there is no straightforward way to customize the error messages.