Like visits to the dentist, thorough testing of any program is something that you should be doing if you want to avoid the pain of having to trace a problem that you thought you'd taken care of. This lesson is one that normally takes a programmer many years to learn, and to be honest, you're still going to be working on it for many years. However, the one thing that is of the utmost importance is that testing must be organized; and to be the most effective, you must start writing your programs knowing that it will be tested as you go along, and plan around having the time to write and confirm your test cases.
Fortunately, Python offers an excellent facility for organizing your testing called PyUnit. It is a Python port of the Java JUnit package, so if you've worked with JUnit you're already on firm ground when testing in Python — but if not, don't worry.
In this chapter you learn:
The concept and use of assertions
The basic concepts of unit testing and test suites
A few simple example tests to show you how to organize a test suite
Thorough testing of the search utility from Chapter 11
The beauty of PyUnit is that you can set up testing early in the software development life cycle, and you can run it as often as needed while you're working. By doing this, you can catch errors early on, before they're painful to rework — let alone before anybody else sees them. You can also set up test cases before you write code, so that as you write, you can be sure that your results match what you expect! Define your test cases before you even start coding, and you'll never find yourself fixing a bug only to discover that your changes have spiraled out of control and cost you days of work.
Note that PyUnit is not the only framework available for testing your Python programs. There are literally dozens of others out there. At the time of this writing, the vast majority of those have not been updated to work with Python 3.1, but they are definitely worth a look once they get updated.
An assertion in Python is in practice similar to an assertion in day-to-day language. When you speak and you make an assertion, you have said something that isn't necessarily proven but that you believe to be true. Of course, if you are trying to make a point, and the assertion you made is incorrect, your entire argument falls apart.
In Python, an assertion is a similar concept. Assertions are statements that can be made within the code while you are developing it that you can use to test the validity of your code, but if the statement doesn't turn out to be true, an AssertionError is raised, and the program will be stopped if the error isn't caught (in general, they shouldn't be caught, because AssertionErrors should be taken as a warning that you didn't think something through correctly!)
Assertions enable you to think of your code in a series of testable cases. That way, you can make sure that while you develop, you can make tests along the lines of "this value is not None" or "this object is a String" or "this number is greater than zero." All of these statements are useful while developing to catch errors in terms of how you think about the program.
Assert lacks a couple of things by itself. First, assert doesn't provide you with a structure in which to run your tests. You have to create a structure, and that means that until you learn what you want from tests, you're liable to make tests that do more to get in your way than confirm that your code is correct.
Second, assertions just stop the program and they provide only an exception. It would be more useful to have a system that would give you summaries, so you can name your tests, add tests, remove tests, and compile many tests into a package that let you summarize whether or not your program tests out. These ideas and more make up the concepts of unit tests and test suites.
Unit testing revolves around the test case, which is the smallest building block of testable code for any circumstances that you're testing. When you're using PyUnit, a test case is a simple object with at least one test method that runs code; and when it's done, it then compares the results of the test against various assertions that you've made about the results.
Note
PyUnit is the name of the package as named by its authors, but the module you import is called the more generic-sounding name unittest.
Each test case is subclassed from the TestCase class, which is a good, memorable name for it. The simplest test cases you can write just override the runTest method of TestCase and enable you to define a basic test, but you can also define several different test methods within a single test case class, which can enable you to define things that are common to a number of tests, such as setup and cleanup procedures.
A series of test cases run together for a particular project is called a test suite. You can find some simple tools for organizing test suites, but they all share the concept of running a bunch of test cases together and recording what passed, what failed, and how, so you can know where you stand.
Because the simplest possible test suite consists of exactly one test case, and you've already had the simplest possible test case described to you, in the following Try It Out you write a quick testing example so you can see how all this fits together. In addition, just so you really don't have anything to distract you, you test arithmetic, which has no external requirements on the system, the file system, or, really, anything.
TestCase classes beginning with fail, such as failUnless, failIf, and failUnlessEqual, come in additional varieties to simplify setting up the conditions for your tests. When you're programming, you'll likely find yourself resistant to writing tests (they can be very distracting; sometimes they are boring; and they are rarely something other people notice, which makes it harder to motivate yourself to write them). PyUnit tries to make things as easy as possible for you.
After the unit test is defined in ArithTest, you may like to define the suite itself in a callable function, as recommended by the PyUnit developer, Steve Purcell, in the modules documentation. This enables you to simply define what you're doing (testing) and where (in the function you name). Therefore, after the definition of ArithTest, you have created the suite function, which simply instantiates a vanilla, unmodified test suite. It adds your single unit test to it and returns it. Keep in mind that the suite function only invokes the TestCase class in order to make an object that can be returned. The actual test is performed by the returned TestCase object.
As you learned in Chapter 6, only when this is being run as the main program will Python invoke the TextTestRunner class to create the runner object. The runner object has a method called run that expects to have an object of the unittests.TestSuite class. The suite function creates one such object, so test_suite is assigned a reference to the TestSuite object. When that's finished, the runner.run method is called, which uses the suite in test_suite to test the unit tests defined in test_suite.
The actual output in this case is dull, but in that good way you'll learn to appreciate because it means everything has succeeded. The single period tells you that it has successfully run one unit test. If, instead of the period, you see an F, it means that a test has failed. In either case, PyUnit finishes off a run with a report. Note that arithmetic is run very, very fast.
Now, see what failure looks like.
Well, this is all well and good, but real-world tests usually involve some work to set up your tests before they're run (creating files, creating an appropriate directory structure, generally making sure everything is in shape, and other things that may need to be done to ensure that the right things are being tested). In addition, cleanup also often needs to be done at the end of your tests.
In PyUnit, the environment in which a test case runs is called the test fixture, and the base TestCase class defines two methods: setUp, which is called before a test is run, and tearDown, which is called after the test case has completed. These are present to deal with anything involved in creating or cleaning up the test fixture.
Note
You should know that if setUp fails, tearDown isn't called. However, tearDown is called even if the test case itself fails.
Remember that when you set up tests, the initial state of each test shouldn't rely on a prior test having succeeded or failed. Each test case should create a pristine test fixture for itself. If you don't ensure this, you're going to get inconsistent test results that will only make your life more difficult.
To save time when you run similar tests repeatedly on an identically configured test fixture, subclass the TestCase class to define the setup and cleanup methods. This will give you a single class that you can use as a starting point. Once you've done that, subclass your class to define each test case. You can alternatively define several test case methods within your unit case class, and then instantiate test case objects for each method. Both of these are demonstrated in the next example.
A good way to see how all of this fits together is to use a test suite during the development of an extended coding project. This strategy underlies the XP (Extreme Programming) methodology, which is a popular trend in programming: First, you plan the code; then you write the test cases as a framework; and only then do you write the actual code. Whenever you finish a coding task, you rerun the test suite to see how closely you approach the design goals as embodied in the test suite. (Of course, you are also debugging the test suite at the same time, and that's fine!) This technique is a great way to find your programming errors early in the process, so that bugs in low-level code can be fixed and the code made stable before you even start on higher-level work, and it's extremely easy to set up in Python using PyUnit, as you see in the next example.
This example includes a realistic use of text fixtures as well, creating a test directory with a few files in it and then cleaning up the test directory after the test case is finished. It also demonstrates the convention of naming all test case methods with test followed by the name, such as testMyFunction, to enable the unittest.main procedure to recognize and run them automatically.
The first step in this programming methodology, as with any, is to define your objectives — in this case, a general-purpose, reusable search function that you can use in your own work. Obviously, it would be a waste of time to anticipate all possible text-processing functionality in a single search utility program, but certain search tasks tend to recur a lot. Therefore, if you wanted to implement a general-purpose search utility, how would you go about it? The UNIX find command is a good place to look for useful functionality — it enables you not only to iterate through the directory tree and perform actions on each file found, but also to specify certain directories to skip, to specify rather complex logic combinations on the command line, and a number of other things, such as searching by file modification date and size.
On the other hand, the find command doesn't include any searching on the content of files (the standard way to do this under UNIX is to call grep from within find) and it has a lot of features involving the invocation of post-processing programs that we don't really need for a general-purpose Python search utility.
What you might need when searching for files in Python could include the following:
Return values you can use easily in Python: A tuple including the full path, the file name, the extension, and the size of the file is a good start.
Specification of a regular expression for the file name to search for and a regular expression for the content (if no content search is specified, the files shouldn't be opened, to save overhead).
Optional specifications of additional search terms: The size of the file, its age, last modification, and so on are all useful.
A truly general search utility might include a function to be called with the parameters of the file, so that more advanced logic can be specified. The UNIX find command enables very general logic combinations on the command line, but frankly, let's face it — complex logic on the command line is hard to understand. This is the kind of thing that really works better in a real programming language like Python, so you could include an optional logic function for narrowing searches as well.
In general, it's a good idea to approach this kind of task by focusing first on the core functionality, adding more capability after the initial code is already in good shape. That's how the following example is structured — first you start with a basic search framework that encapsulates the functionality you covered in the examples for the os and re modules, and then you add more functionality once that first part is complete. This kind of incremental approach to software development can help keep you from getting bogged down in details before you have anything at all to work with, and the functionality of something like this general-purpose utility is complicated enough that it would be easy to lose the thread.
Because this is an illustration of the XP methodology as well, you'll follow that methodology and first write the code to call the find utility, build that code into a test suite, and only then will you write the find utility. Here, of course, you're cheating a little. Ordinarily, you would be changing the test suite as you go, but in this case, the test suite is already guaranteed to work with the final version of the tested code. Nonetheless, you can use this example for yourself.
Remember that this is an illustration of an incremental programming approach, so the first example was a good place to stop and give an explanation, but there are plenty of other search parameters it would be nice to include in this general search utility, and of course there are still two unit cases to go in the test suite you wrote at the outset. Because Python gives you a keyword parameter mechanism, it's very simple to add new named parameters to your function definition and toss them into the search context dictionary, and then use them in find_file as needed, without making individual calls to the find function unwieldy.
The next example shows you how easy it is to add a search parameter for the file's extension, and throws in a logic combination callback just for good measure. You can add more search parameters at your leisure; the following code just shows you how to get started on your own extensions (one of the exercises for the chapter asks you to add search parameters for the date on which the file was last modified, for instance).
Though the file extension parameter, as a single simple value, is easy to conceive and implement — it's really just a matter of adding the parameter to the search context and adding a filter test in find_file — planning a logic combination callback parameter requires a little thought. The usual strategy for specification of a callback is to define a set of parameters — say, the file name, size, and modification date — and then pass those values in on each call to the callback. If you add a new search parameter, you're faced with a choice — you can arbitrarily specify that the new parameter can't be included in logical combinations, you can change the callback specification and invalidate all existing callbacks for use with the new code, or you can define multiple categories of logic callbacks, each with a different set of parameters. None of these alternatives is terribly satisfying, and yet they're decisions that have to be made all the time.
In Python, however, the dictionary structure provides you with a convenient way to circumvent this problem. If you define a dictionary parameter that passes named values for use in logic combinations, unused parameters are simply ignored. Thus, older callbacks can still be used with newer code that defines more search parameters, without any changes to code you've already got being necessary. In the updated search code found in the next Try It Out, the callback function is defined to be a function that takes a dictionary and returns a flag — a true filter function. You can see how it's used in the example section and in the next chapter, in test case 5 in the search test suite.
Adding a logical combination callback also makes it simple to work with numerical parameters such as the file size or the modification date. It's unlikely that a caller will search on the exact size of a file; instead, one usually searches for files larger or smaller than a given value, or in a given size range — in other words, most searches on numerical values are already logical combinations. Therefore, the logical combination callback should also get the size and dates for the file, so that a filter function can already be written to search on them. Fortunately, this is simple — the results of os.stat are already available to copy into the dictionary.
The result of the test suite shown in the preceding example is clean and stable code in a somewhat involved programming example, and well-defined test cases that are documented as working correctly. This is a quick and easy process in the case of a software "product" that is some 30 lines long, although it can be astounding how many programming errors can be made in only 30 lines!
In a real-life software life cycle, of course, you will have thousands of lines of code. In projects of realistic magnitude like this, nobody can hope to define all possible test cases before releasing the code. It's true that formal testing during the development phase will dramatically improve both your code and your confidence in it, but there will still be errors in it when it goes out the door.
During the maintenance phase of the software life cycle, bug reports are filed after the target code is placed in production. If you're taking an integrated testing approach to your development process, you can see that it's logical to think of bug reports as highlighting errors in your test cases as well as errors in the code itself. Therefore, the first thing you should do with a bug report is to use it to modify an existing test case, or to define a new test case from scratch, and only then should you start to modify the target code itself.
By doing this, you accomplish several things. First, you're giving the reported bugs a formal definition. This enables you to agree with other people regarding what bugs are actually being fixed, and it enables further discussion to take place as to whether the bugs have really been understood correctly. Second, by defining test fixtures and test cases, you are ensuring that the bugs can be duplicated at will. As I'm sure you know if you've ever need to reproduce elusive bugs, this alone can save you a lot of lost sleep. Finally, the third result of this approach might be the most significant: If you never make a change to code that isn't covered by a test case, you will always know that later changes aren't going to break fixes already in place. The result is happier users and a more relaxed you. And you'll owe it all to unit testing.
Testing is a discipline best addressed at the very outset of the development life cycle. In general, you will know that you've got a firm grip on the problem you're solving when you understand it enough to write tests for it.
The most basic kind of test is an assertion. Assertions are conditions that you've placed inside of your program confirming that conditions that should exist do in fact exist. They are for use while you're developing a program to ensure that conditions you expect are met.
Assertions will be turned off if Python is run with the -O option. The -O indicates that you want Python to run in a higher performance mode, which would usually also be the normal way to run a program in production. This means that using assert is not something that you should rely on to catch errors in a running system.
PyUnit is the default way of doing comprehensive testing in Python, and it makes it very easy to manage the testing process. PyUnit is implemented in the unittest module.
When you use PyUnit to create your own tests, PyUnit provides you with functions and methods to test for specific conditions based on questions such as "is value A greater than value B," giving you a number of methods in the TestCase class that fail when the conditions reflected by their names fail. The names of these methods all begin with "fail" and can be used to set up most of the conditions for which you will ever need to test.
The TestCase class should be subclassed — it's the run method that is called on to run the tests, and this method needs to be customized to your tests. In addition, the test fixture, or the environment in which the tests should be run, can be set up before each test if the TestCase's setUp and tearDown methods are overridden, and code is specified for them.
You've seen two approaches to setting up a test framework for yourself. One subclasses a customized class, and another uses separate functions to implement the same features but without the need to subclass. You should use both and find out which ones work for your way of doing things. These tests do not have to live in the same file as your modules or programs; they should be kept separate so they don't bloat your code.
As you go through the remainder of this book, try to think about writing tests for the functions and classes that you see, and perhaps write tests as you go along. It's good exercise; better than having exercises here.
The key things to take away from this chapter are:
Assertions are statements made within your code that allow you to test the validity of the code. If the test fails, an
AssertionErroris raised. You can use assert to create your tests.PyUnit is the name of the package as named by its authors, but the module you import is called the more generic-sounding name unittest.
A test suite is a series of test cases run together for a particular project.
In PyUnit, the environment in which a test case runs is called the test fixture, and the base
TestCaseclass defines two methods:setUp, which is called before a test is run; andtearDown, which is called after the test case has completed. These are present to deal with anything involved in creating or cleaning up the test fixture.
In the following chapter, we will discuss GUI (graphical user interface) programming, and learn to make simple, interactive programs.