Unittest provides the means to test the code through a series of assertions. I have often felt the temptation to exercise many aspects of a particular piece of code within a single test method. If any part fails, it becomes obscured as to which part failed. It is preferable to split things up into several smaller test methods, so that when some part of the code under test fails, it is obvious.
With these steps, we will investigate what happens when we put too much into a single test method.
- Create a new file named
recipe8.pyin which to put our application code for this recipe. - Pick a class to test. In this case, we will use an alternative version of the Roman numeral converter, which converts both ways.
class RomanNumeralConverter(object): def __init__(self): self.digit_map = {"M":1000, "D":500, "C":100, "L":50, "X":10, "V":5, "I":1} def convert_to_decimal(self, roman_numeral): val = 0 for char in roman_numeral: val += self.digit_map[char] return val def convert_to_roman(self, decimal): val = "" while decimal > 1000: val += "M" decimal -= 1000 while decimal > 500: val += "D" decimal -= 500 while decimal > 100: val += "C" decimal -= 100 while decimal > 50: val += "L" decimal -= 50 while decimal > 10: val += "X" decimal -= 10 while decimal > 5: val += "V" decimal -= 5 while decimal > 1: val += "I" decimal -= 1 return val - Create a new file called
recipe8_obscure.pyin which to put some longer test methods. - Create some test methods that combine several test assertions.
import unittest from recipe8 import * class RomanNumeralTest(unittest.TestCase): def setUp(self): self.cvt = RomanNumeralConverter() def test_convert_to_decimal(self): self.assertEquals(0, self.cvt.convert_to_decimal("")) self.assertEquals(1, self.cvt.convert_to_decimal("I")) self.assertEquals(2010, self.cvt.convert_to_decimal("MMX")) self.assertEquals(4000, self.cvt.convert_to_decimal("MMMM")) def test_convert_to_roman(self): self.assertEquals("", self.cvt.convert_to_roman(0)) self.assertEquals("II", self.cvt.convert_to_roman(2)) self.assertEquals("V", self.cvt.convert_to_roman(5)) self.assertEquals("XII", self.cvt.convert_to_roman(12)) self.assertEquals("MMX", self.cvt.convert_to_roman(2010)) self.assertEquals("MMMM", self.cvt.convert_to_roman(4000)) if __name__ == "__main__": unittest.main() - Run the obscure tests. Why did it fail? Where is the bug? It reports that
IIis not equal toI, so something appears to be off. If this the only bug?
- Create another file called
recipe8_clear.pyto create a more fine-grained set of test methods. - Split up the assertions into separate test methods to give a higher fidelity of output.
import unittest from recipe8 import * class RomanNumeralTest(unittest.TestCase): def setUp(self): self.cvt = RomanNumeralConverter() def test_to_decimal1(self): self.assertEquals(0, self.cvt.convert_to_decimal("")) def test_to_decimal2(self): self.assertEquals(1, self.cvt.convert_to_decimal("I")) def test_to_decimal3(self): self.assertEquals(2010, self.cvt.convert_to_decimal("MMX")) def test_to_decimal4(self): self.assertEquals(4000, self.cvt.convert_to_decimal("MMMM")) def test_convert_to_roman1(self): self.assertEquals("", self.cvt.convert_to_roman(0)) def test_convert_to_roman2(self): self.assertEquals("II", self.cvt.convert_to_roman(2)) def test_convert_to_roman3(self): self.assertEquals("V", self.cvt.convert_to_roman(5)) def test_convert_to_roman4(self): self.assertEquals("XII", self.cvt.convert_to_roman(12)) def test_convert_to_roman5(self): self.assertEquals("MMX", self.cvt.convert_to_roman(2010)) def test_convert_to_roman6(self): self.assertEquals("MMMM", self.cvt.convert_to_roman(4000)) if __name__ == "__main__": unittest.main() - Run the clearer test suite. Is it a bit clearer where the bug is? What did we trade in to get this higher degree of test failure? Was it worth the effort?
In this case, we created a modified Roman numeral converter that converts both ways. We then started creating test methods to exercise things. Since each of these tests were a simple, one-line assertion, it was convenient to put them all in the same test method.
In the second test case, we put each assertion into a separate test method. Running it exposes the fact that there are multiple bugs in this Roman numeral converter.
When we started off writing tests, it was very convenient to bundle all these assertions into a single test method. After all, if everything is working, there is no harm, right? But what if everything does not work, what do we have to deal with? An obscure error report!
The obscured test runner may not be clear. All we have to go on is II != I. Not much. The clue is that it is only off by one. The clear test runner gives more clues. We see that V != IIII, XII != XI, and some more. Each of these failures shows things being off by one.
The bug involves the various Boolean conditions in the while checks:
while decimal > 1000:
while decimal > 500:
while decimal > 100:
while decimal > 50:
while decimal > 10:
while decimal > 5:
while decimal > 1:Instead of testing greater than, it should test for greater than or equal to. This causes it to skip out of each Roman numeral before counting the last one.
In this recipe, we broke things down to a single assertion per test. But I wouldn't advise thinking along these lines.
If we look a little closer, each test method also involves a single usage of the Roman numeral API. For the converter, there is only one result to examine when exercising the code. For other systems, the output may be more complex. It is completely warranted to use several assertions in the same test method to check the outcome by making that single call.
When we proceed to make more calls to the Roman numeral API, it should signal us to consider splitting it off into a new test method.
This opens up the question: what is a unit of code? There has been much debate over what defines a unit of code, and what makes a good unit test. There are many opinions. Hopefully, reading this chapter and weighing it against the other test tactics covered throughout this book will help you enhance your own opinion and ultimately improve your own testing talent.
Unittest can easily help us write both unit tests as well as integration tests. Unit tests exercise smaller blocks of code. When writing unit tests, it is best to keep the testing as small and fine grained as possible.
When we move up to a higher level (such as integration testing), it makes sense to test multiple steps in a single test method. But this is only recommended if there are adequate low-level unit tests. This will shed some light on whether it is broken at the unit level, or whether there is a sequence of steps that causes the error.
Integration tests often extend to things like external systems. For example, many argue that unit testing should never connect to a database, talk to an LDAP server, or interact with other systems.