The test runner recognizes any method whose name starts with test as a test method. The test runner will call these methods and collect the results.

assert 2 == 3, "Two is not equal to three"

Assertions are made by calling assert methods inherited from the TestCase class.

You need something to test; and, in order to test it, you need to know what it’s supposed to do. Let’s create a simple Value class for performing operations on numbers. This class should meet the following specifications:

From this specification, you can write the tests. Writing the tests first is a process called test-driven development.^[4] With appropriately named tests that record the specification (and clearly written tests as well, of course), tests can act as a specification for the code.

Listing 7.1 shows a TestCase, called ValueTest, which tests the specification just listed.

import unittest
class ValueTest(unittest.TestCase):
    def testConstructorShouldStoreValue(self):
       value = Value(6)
       self.assertEquals(value.value, 6,
          "value attribute not set correctly")

    def testAddShouldReturnArgumentAddedToValue(self):
       value = Value(6)
       self.assertEquals(value.add(3), 9,
                   "add returned the wrong answer")

    def testIsEvenShouldReturnTrueForEvenNumbers(self):
       value = Value(6)
       self.assertTrue(value.isEven(),
          "Wrong answer for isEven with an even number")

    def testIsEvenShouldReturnFalseForOddNumbers(self):
       value = Value(7)
       self.assertFalse(value.isEven(),
          "Wrong answer for isEven with an odd number")

 if __name__ == '__main__':
    unittest.main()

The test methods all create an instance of our Value class and then test its state or the result of calling methods. The calls to the assert methods do the actual testing, and they all follow a pattern similar to the one in figure 7.1.

Figure 7.1. The anatomy of an assert method on TestCase

Several assert methods are available on TestCase. In creating a test framework customized for your application, you’ll build higher-level tests on these primitive assert methods, or possibly directly with the assert statement. Table 7.1 shows the standard assert methods.

So what happens when you run the test we’ve just created? Take a look at figure 7.2.

Figure 7.2. The unit tests for Value, run without the Value class in place

All the tests error out with a NameError because we haven’t yet written the Value class. This is the TDD approach—to write the tests before the implementation. You’ll know when we have a complete implementation; all the tests will pass.

unittest formats the results of running the tests in the output. The first line of the output is a line of characters, with one character representing the result of each test.

The three possible results of calling a test method are as follow:

The last line of the output is the summary of the test run; here all four tests have failed with the same error. Obviously, the missing component is the Value class. If we were following strict TDD, you’d implement one method at a time, running the tests in between each method. To save space, listing 7.2 is a full implementation of our enormously complex Value class.

class Value(object):
   def __init__(self, value):
      self.value = value

   def add(self, number):
      return self.value + number

    def isEven(self):
       return (self.value % 2) == 0

When you run the tests now, the output should be much improved (figure 7.3).

Figure 7.3. The unit tests for Value, run with the Value class in place

The ‘EEEE’ from the first run has been replaced with four dots (....), and the summary says OK. Great.

Four out of our five tests start with the same line: value = Value(6). Code duplication is always bad, right? The next section looks at how you can reduce boilerplate in unit tests using setUp and tearDown.

Some classes can’t be tested in isolation; they need either support classes configuring or state initializing before they can be tested. If this initialization includes creating test files or opening external connections, then you may need to both set up tests and clean up after them.

unittest makes this possible through the setUp and tearDown methods on TestCase. As you might surmise, setUp is run before each test method and tearDown is run afterward. Any exceptions raised in setUp and tearDown will count as a test failure.

Listing 7.3 demonstrates how you can use the setUp and tearDown methods with our simple tests.

Example 7.3. The unit tests for Value rewritten to use setUp and tearDown

The tearDown shown in this listing isn’t strictly necessary, but we wanted to demonstrate how it works. Both setUp and tearDown call up to the base class methods. When creating a test framework for a project, it’s common to create a hierarchy of test case classes for different types of tests. Forgetting to call up to the base class setUp is an easy (and sometimes hard to diagnose) mistake to make.

The approach to unittest we’ve used so far is fine when you have all your tests in a single module. The next section will look at how to create test suites from multiple test files.

When writing tests for a project, you’ll want to create different test files for different classes and modules in your project. We’ve been running tests using the unittest.main function. This runs all the tests in whatever is running as the main script. This approach is fine for running each test file separately, but you’ll usually want to run all your tests in one go and collect the results.

unittest supports running many tests by allowing you to create test suites from multiple test classes, and passing them to a test runner. (This is what main does under the hood.) Figure 7.4 illustrates the different classes available in unittest to help automate your test running.

Figure 7.4. Collecting TestCases together in a TestSuite and running them with the TextTestRunner

Using the following components, it’s easy to collect our tests and run them all together:

To test a large project, you want to be able to collect all the tests together and run them in one go. Using the __name__ == '__main__' pattern, it will still be possible to run tests contained in individual modules while developing. Using introspection, you can import the test modules and automatically add all the test classes they contain to a suite. You can then pass the suite to a runner and check the results. Let’s create a set of utility functions and keep them in a module called testutils (listing 7.4), which we’ll be using shortly to test MultiDoc.

Example 7.4. testutils: module to support test running

The easiest way of explaining these functions is to work from the bottom up.

RunTests runs a test suite and returns the results. It accepts keyword arguments that are passed through to the TextTestRunner. MakeSuite creates a test suite , and initializes it with any test classes or modules that you pass in . AddTests adds tests to a suite . If you pass in a module (recognized because its type is ModuleType), then you check every object it contains by iterating over its __dict__ dictionary , as follows:

for entry in test.__dict__.values()

Objects are checked to see if they’re tests by calling IsTestCase, which checks to see if an object is a test class or not . type(test) == type is only true for classes. You can tell if a class is a test class by checking if it’s a subclass of TestCase, as follows:

issubclass(test, unittest.TestCase)

Listing 7.5 is an example of using these functions to run the unit tests for the Value class that you created in the previous sections. It assumes you’ve put these tests in a module called ValueTestModule that you can import into our test runner code.

import sys
from testutils import MakeSuite, RunTests
import ValueTestModule    
suite = MakeSuite(ValueTestModule)
results = RunTests(suite, verbosity=2)    
if results.failures or results.errors:
    sys.exit(1)    

This listing imports the module containing our tests and creates the test suite. If you have several test modules, then you can use something like this code segment:

import testmodule1
import testmodule2
import testmodule3

suite = MakeSuite(testmodule1, testmodule2, testmodule3)

MakeSuite adds all the modules you pass it to the suite. After creating the suite, you run the tests . Listing 7.5 passes in the optional keyword argument verbosity=2. This increases the amount of information output while tests are running—which is useful for keeping track of tests as they run. Figure 7.5 shows the output of running tests with a higher verbosity level.

Figure 7.5. Running tests with verbosity level set to 2

Listing 7.5 checks the results of these tests; if there are any errors or failures, it exits with an exit code of 1 . A correct exit code allows you to integrate running your tests into your build process (for example using msbuild)^[5] and stop the build if tests fail.

So far, we’ve concentrated on getting familiar with the basics of the unittest framework. This is background to the really interesting part—exploring different testing techniques with IronPython. Many of these techniques make great use of the dynamic nature of Python. To demonstrate this, we write tests for MultiDoc.

Working with mock objects is one place where you’ll see the advantage of using a dynamically typed language. In a statically typed language, the compiler won’t let you run any code unless the types match those declared in the production code. The objects you use in tests must be real objects, or objects that inherit from the expected type.

In a dynamically typed language, you can take advantage of duck typing. The objects you use for testing need only implement the attributes and methods used in the tests. Let’s see this in action by testing the execute method of MultiDoc’s OpenCommand.

Listing 7.6 gives a quick refresher of what the execute code looks like.

def execute(self):
   fileName = self.mainForm.document.fileName
   directory = Path.GetDirectoryName(fileName)
   directoryExists = Directory.Exists(directory)
   openFileDialog = self.openFileDialog
if fileName is not None and directoryExists:
   openFileDialog.InitialDirectory = directory
   openFileDialog.FileName = fileName

if openFileDialog.ShowDialog() == DialogResult.OK:
   document = self.getDocument(openFileDialog.FileName)
   if document:
      self.mainForm.document = document

The first behavior of execute to test is that it correctly sets the Filename and InitialDirectory attributes on the OpenFileDialog. OpenCommand has a reference to the MainForm, so it can check whether the current document has a fileName set on it or not. If this reference isn’t None, then the filename should be set on the OpenFileDialog, so that the dialog opens in the same folder as the current file.

Where possible, classes should be tested in isolation. This is one of the ways that testing encourages you to write better code. By making classes easier to test, you end up making them more modular and decoupled from other classes (which is usually in the right direction along the road to better).

The only attribute of MainForm that OpenCommand uses is the document. Instead of using a real instance and a real document, you can use a mock object. With a statically typed language, you’d have to provide an object that the compiler recognizes as being of a suitable type. Plenty of libraries could help you to do this, but it’s a lot simpler with a dynamic language such as Python. Listing 7.7 shows just about the simplest possible mock object, and how you can use it to create a mock MainForm for testing the OpenCommand.

class Mock(object):
   pass
mainform = Mock()
document = Mock()
document.fileName = None
mainform.document = document
command = OpenCommand(mainform)

If you call execute as it stands, then the call to openFileDialog.ShowDialog() will block—which isn’t ideal for an automated test suite. Instead of a real OpenFileDialog, we need a mock one with a ShowDialog method.

The real dialog box returns either DialogResult.OK or DialogResult.Cancel. If the user chooses a file, then the returned value is DialogResult.OK, which triggers the creation of a new document. In this test, we don’t want this result, so we want a class that returns DialogResult.Cancel and has Filename and InitialDirectory attributes (which we do want to test). Next, we’ll want to test what happens when the user accepts the dialog box. Listing 7.8 shows a MockDialog class that allows you to control what’s returned from ShowDialog.

Example 7.8. A mock OpenFileDialog class

The OpenCommand stores the dialog as an instance attribute, so replacing it with the mock dialog is easy.

Now to use this in a test. In listing 7.9, the test name reflects the behavior we’re testing. Because you’ll need an OpenCommand initialized with a mock MainForm and dialog in the next few tests, we place this code in a setUp method.

Example 7.9. Testing the use of OpenFileDialog

The mock objects we’ve created in this section are pretty specific to testing the OpenCommand. Instead of creating all your mocks from scratch, you could use any of the following Python mock libraries:

Perhaps the reason for the many different mock libraries is that they’re so easy to write. The first one in the list is my favorite, because I wrote it, but it’s often simpler to write mocks as you need them as we’ve been doing.

Even though creating mocks is easy, that doesn’t mean that there aren’t reusable patterns. In the next section, we look at a slightly different testing pattern using a Listener class.

Next, we need to test the behavior of execute when the dialog is accepted or canceled. If the dialog is accepted, it calls the getDocument method with the filename from the dialog and sets the returned document onto the MainForm. If the dialog is canceled, then it doesn’t.

You could test this by providing a known filename and then checking that the returned document is valid and complete. The test would need a real file, and would also depend on the Document class remaining the same. If you changed Document, then you’d also need to change the way you test OpenCommand. This is why testing in isolation is preferred—tests for one part of the code become much less brittle to changes in another part of the code.

You can get around this by replacing the getDocument method with a custom object that returns a mock document and also allows you to confirm whether it has been called or not.

Adding methods at runtime is another feature of dynamic languages, but is known among the Python community by the slightly pejorative term of monkey patching. The reason it’s frowned on is that it can make your code hard to read. If a class defines a method, and then later on you see that method being called, you’ll assume you know what code is being executed. If in fact that method has been replaced, it’s difficult to know what code is being executed.

One place where monkey patching is both accepted and useful is in testing. Because methods are looked up dynamically, you can add methods at runtime. In order to understand monkey patching, it will be useful to take a brief look at the Python attribute lookup rules.

Attribute Lookup Rules

When you call a method on an instance, the method is looked up using the normal order shown in figure 7.6.^[8]

Figure 7.6. How a method call becomes an attribute lookup followed by a call

You can confirm these rules at the interactive interpreter by adding a new method to a class. All instances of the class then gain the method.

>>> class AClass(object):
...    pass
...
>>> instance = AClass()
>>> def method(self):
...    print 'Hello'
...
>>> AClass.method = method
>>> instance.method()
Hello

In our case, you have the choice of patching the replacement object on the class or on the instance. The disadvantage of patching the class is that it’s effectively a global and modifying it will also modify it for other tests. You can override methods used at runtime by patching the instance—which is what we need for testing. Again, this is easy to show in an interactive interpreter session.

>>> def method2():
...    print 'Hello 2'
...
>>> instance.method = method2
>>> instance.method()
Hello 2

We’ve now covered most of the basic principles of testing in Python, useful knowledge to apply whether you’re programming in CPython or IronPython. Coming soon is functional testing, but first we put monkey patching into practice with a useful test class.

Bound and unbound methods

You may have noticed that when you patched the class with a function, it needed to take the argument self like normal methods. When you look up a method defined on a class, it gets wrapped as a bound method object—which is how self is passed in. (This happens because functions are descriptors.) When you attach a function to an instance, it’s just an ordinary attribute and so doesn’t get self passed to it.

Bound methods are named because of the way self is bound to the method.

This distinction between bound methods (methods looked up on an instance and wrapped as a bound method) and unbound methods (the same object fetched directly from the class) isn’t only deep Python; it can also be very useful.

.NET has a similar distinction with closed static and open instance delegates. Good references are hard to come by, but the best we’ve found^[9] is at http://peisker.net/dotnet/languages2005.htm.

class Listener(object):
   def __init__(self):
      self.reset()

   def reset(self):
      self.returnVal = None
      self.triggered = False
      self.triggerArgs = None
      self.triggerKeyWargs = None

   def __call__(self, *args, **keywargs):
      self.triggered = True
      self.triggerArgs = args
      self.triggerKeyWargs = keywargs
      return self.returnVal

def testExecuteShouldNotCallGetDocumentForCancelledDialog(self):
   listener = Listener()
   self.command.getDocument = listener

   self.command.execute()
   self.assertFalse(listener.triggered, "getDocument called incorrectly")

def testExecuteWithAcceptedDialogShouldCallGetDocument(self):
   listener = Listener()
   self.command.getDocument = listener

   originalDocument = self.command.mainForm.document
   self.command.mainForm.document.fileName = __file__
   self.command.openFileDialog.returnVal = DialogResult.OK
   self.command.execute()
   self.assertEquals(listener.triggerArgs, (__file__,),
               "getDocument not called with filename")
   self.assertEquals(self.command.openFileDialog.InitialDirectory,
               Path.GetDirectoryName(__file__),
               "FileName incorrectly set")
   self.assertEquals(self.command.mainForm.document,
               originalDocument,
               "document incorrectly changed")

def testNewDocumentFromGetDocumentShouldBeSetOnMainForm(self):
   listener = Listener()
   self.command.getDocument = listener
   self.command.mainForm.document.fileName = __file__
   self.command.openFileDialog.returnVal = DialogResult.OK
   newDocument = object()
   listener.returnVal = newDocument
   self.command.execute()
   self.assertEquals(self.command.mainForm.document,
               newDocument,
               "document not replaced")

In dependency injection, dependencies are supplied to components rather than being used directly. Dependency injection makes testing easier, because you can supply mocks instead of the real dependencies and test that they’re used as expected. A common way to do dependency injection in Python is to provide dependencies as default arguments in object constructors.^[10]

Let’s look at testing a simple scheduler class to see how this works (listing 7.14).

import time

class Scheduler(object):
   def __init__(self, tm=time.time, sl=time.sleep):
      self.time = tm
      self.sleep = sl

   def schedule(self, when, function):
      self.sleep(when - self.time())
      return function()

Scheduler has a single method, schedule, that takes a callable and a time for the callable to be fired. The schedule method blocks by sleeping until the correct time (when) using the time.time and time.sleep standard library functions; but, because it obtains them with dependency injection, it’s easy to test. The injection is set up in the Scheduler constructor, so the first thing you need to test is that the default constructor does the right thing. Setting up the default dependency in the constructor is the extra layer that dependency injection introduces into your code. Listing 7.15 shows the test for the constructor.

import time
from unittest import TestCase
from dependency_injection import Scheduler

class DependencyInjectionTest(TestCase):

   def testConstructor(self):
      scheduler = Scheduler()
      self.assertEquals(scheduler.time, time.time,
                   "time not initialized correctly")
      self.assertEquals(scheduler.sleep, time.sleep,
                   "sleep not initialized correctly")

Having tested that the dependency injection is properly initialized in the default case, you can use it to test the schedule method. Listing 7.16 uses a fake time module that records calls to time, sleep, and the function you pass into schedule. Methods on FakeTime are passed into the constructor instead of the defaults. You can then assert that calls are made in the right order, with the right arguments, and that schedule returns the right result.

def testSchedule(self):
   class FakeTime(object):
      calls = []

      def time(self):
         self.calls.append('time')
         return 100

      def sleep(self, howLong):
         self.calls.append(('sleep', howLong))

   faketime = FakeTime()
   scheduler = Scheduler(faketime.time, faketime.sleep)

   expectedResult = object()
   def function():
      faketime.calls.append('function')
      return expectedResult

   actualResult = scheduler.schedule(300, function)

   self.assertEquals(actualResult, expectedResult,
    "schedule did not return result of calling function")

   self.assertEquals(faketime.calls,
          ['time', ('sleep', 200), 'function'],
          "time module and functions called incorrectly")

Because the fake time function is set up to return 100, and the function is scheduled to be called at 300, sleep should be called with 200. Dependency injection can easily be done using setter properties, or even with simple attributes. Yet another approach is to use factory methods or functions for providing dependencies, which can be needed where fresh instances of dependencies are required for each use. Dependency injection is useful for both unit testing and subclassing, or overriding the behavior of classes; subclassing is significantly easier to do with Python than some other languages.

One of the problems with unit testing is that, although it’s good for testing components in isolation, it isn’t so good at testing that they’re wired together correctly. To make sure that this is covered, you need some higher-level tests; this is where functional testing comes in.

You need to start MultiDoc, perform actions, and then make assertions about the state of MultiDoc. But, starting MultiDoc means starting the Windows Forms event loop, which will seize the control flow of the thread. The logical thing to do is start MultiDoc on another thread. As if we didn’t have enough problems already, doing this will create another one—any interaction with Windows Forms controls has to be on the thread on which they were created. Fortunately, this is all relatively simple to do. You need to know the following three facts:

Listing 7.17 shows a simple example of starting the event loop on another thread.

Example 7.17. Interacting with Windows Forms controls from another thread

The delegate CallTarget0 wraps functions that don’t take any arguments. A corresponding CallTarget1 wraps functions taking one argument, CallTarget2 for functions that take two arguments, and so on up to CallTarget5.^[14]. We find it simpler to use CallTarget0 and pass in lambda functions where we need a function called with multiple arguments.

Listing 7.18 puts this knowledge to work with a new base class for tests: FunctionalTest. The setUp method starts MultiDoc and tearDown stops it. FunctionalTest also provides a convenience method invokeOnGUIThread for interacting with MultiDoc by executing functions on the GUI thread.

Example 7.18. A FunctionalTest base class for interacting with a running MultiDoc

This new test case is nice, and it will work fine (trust me; we’ve already tried it). But it isn’t quite sufficient for what we want to achieve. We want to test the New Page dialog; and, if you invoke a function on the control thread that opens the dialog, Invoke will block until the dialog is closed again. You need a way to asynchronously perform actions on the control so that you can interact with the Name Tab dialog. This means more fun with dialogs.

You can use a similar pattern to interact asynchronously with the GUI thread. You can launch the action from yet another thread that has the job of calling Invoke. This won’t block the test thread while it’s waiting for Invoke to return. You may want to be able to retrieve a return value, and to be able to join to the new thread to check that it exits, you can encapsulate this functionality in an object. Listing 7.19 shows the AsyncExecutor object along with a convenience method to use it from functional tests.

Example 7.19. A FunctionalTest base class for interacting with a running MultiDoc

Now you have all the infrastructure you need to write the functional test. Ideally, the test would know nothing about the internal structure of MultiDoc; but in order to make assertions about the state of MultiDoc, it needs to know something. This makes the test brittle against changes to the structure of MultiDoc. In the next section, we turn our user story into a functional test while looking at how you can mitigate against this potential brittleness.

You need to create a test file and add it to runtests.py. The new test inherits from the test case, FunctionalTest. setUp automatically launches MultiDoc and gives you access to the main class (the MainForm instance) as self.mainForm. We still need to work out how you’ll interact with MultiDoc.

One way would be to insert fake mouse and key events into the event loop. A managed class is available for sending key presses to the Windows Forms message loop. There are no managed classes for sending mouse movements and button presses, but you can do this using unmanaged classes.^[15] Even taking this route, you’d still need access to the controls to get the locations to send clicks to. If you’re going to have access to the controls anyway, then you might as well trigger them programmatically. This approach still tests that event handlers are wired correctly, and is a good compromise between fully black-box testing and testing completely below the level of the GUI. The advantage of simulating mouse moves and clicks is that it only works if your GUI components are accessible to the mouse (that is, visible). The cost is having to maintain a more complex test framework.

The first important step in our user story is clicking the New Page toolbar button. This button is the fourth button in the toolbar, so you can access it using code like mainForm.toolBar.Items[3].PerformClick() (which must be executed on the control thread, of course). This suffers from the brittleness we mentioned earlier. If you change the order of buttons in the toolbar, then you have to modify everywhere that uses this code. A simple solution is to access the button through a single method. This has all the usual advantages of avoiding duplication and means that you can change the way you access toolbar buttons from a single place. Listing 7.20 shows the start of the functional test and a method for clicking the New Page button.

Example 7.20. A FunctionalTest base class for interacting with a running MultiDoc

Putting the access to UI components into methods also has the advantage of making the functional test more readable. As you write more functional tests, and abstract out of them methods for common actions, you effectively create a Domain Specific Language (DSL) for writing your tests. The ideal would be to have one line of code per line of user story.

An alternative way of making tests less susceptible to breakage caused by layout changes is to give controls a descriptive Name attribute. It’s then easy to provide a findControlByName method that recursively iterates through child controls looking for a specific control. You don’t need to store a reference to all the controls you might want to access through functional tests, and changing your layout won’t (necessarily) break all your functional tests.

Executing our functional test will cause MultiDoc to appear along with dialog and new tab pages. Following the user story, MultiDoc will dance under the invisible hand of Harold. (Because this is automated, the actual dance is very quick, but it will look like figure 7.7.)

Figure 7.7. MultiDoc dancing under the invisible hand of Harold, our mythical user

The full functional test is shown in listing 7.21. It contains the user story as comments in the test method, each line followed by the code that implements it.

Example 7.21. FunctionalTest base class for interacting with running MultiDoc

As you can see, several pieces of code in the test still poke inside MultiDoc to test its state. If you were to implement more tests, you’d find that a lot of this code could be shared between tests and moved up to become methods on FunctionalTest. In this way, the tests become more DSL-like, and you build a comprehensive test framework.

That was fun, but it might give you the impression that creating a functional test suite for your application is easy. We didn’t have to deal with timing or threading issues at all. Despite potential difficulties, getting functional tests working is the most satisfying part of writing tests. We’ve now finished with testing, so it’s time to wrap up.