Chapter 7
xUnit Basics

About This Chapter

Chapter 6, Test Automation Strategy, introduced the "hard to change" decisions that we need to get right early in the project. The current chapter serves two purposes. First, it introduces the xUnit terminology and diagramming notation used throughout this book. Second, it explains how the xUnit framework operates beneath the covers and why it was built that way. This knowledge can help the builder of a new Test Automation Framework (page 298) understand how to port xUnit. It can also help test automaters understand how to use certain features of xUnit.

An Introduction to xUnit

The term xUnit is how we refer to any member of the family of Test Automation Frameworks used for automating Hand-Scripted Tests (see Scripted Test on page 285) that share the common set of features described here. Most programming languages in widespread use today have at least one implementation of xUnit; Hand-Scripted Tests are usually automated using the same programming language as is used for building the SUT. Although this is not necessarily the case, this strategy is usually much easier because our tests have easy access to the SUT API. By using a programming language with which the developers are familiar, less effort is required to learn how to automate Fully Automated Tests (page 26).¹

Common Features

Given that most members of the xUnit family are implemented using an object-oriented programming language (OOPL), they are described here first and then places where the non-OOPL members of the family differ are noted.

All members of the xUnit family implement a basic set of features. They all provide a way to perform the following tasks:

• Specify a test as a Test Method (page 348)
• Specify the expected results within the test method in the form of calls to Assertion Methods (page 362)
• Aggregate the tests into test suites that can be run as a single operation
• Run one or more tests to get a report on the results of the test run

Because many members of the xUnit family support Test Method Discovery (see Test Discovery on page 393), we do not have to use Test Enumeration (page 399) in these members to manually add each Test Method we want to run to a test suite. Some members also support some form of Test Selection (page 403) to run subsets of test methods based on some criteria.

The Bare Minimum

Here is the bare minimum we need to understand about how xUnit operates (Figure 7.1):

• How we define tests using Test Methods on Testcase Classes (page 373)
• How we can build up arbitrary Suites of Suites (see Test Suite Object on page 387)²

• How we run the tests
• How we interpret the test results

Figure 7.1. The static test structure as seen by a test automater. The test automater sees only the static structure as he or she reads or writes tests. The test automater writes one Test Method with four distinct phases for each test in the Testcase Class. The Test Suite Factory (see Test Enumeration) is used only for Test Enumeration. The runtime structure (shown grayed out) is left to the test automater's imagination.

Defining Tests

Each test is represented by a Test Method that implements a single Four-Phase Test (page 358) by following these steps:

• Setting up the test fixture using either In-line Setup (page 408), Delegated Setup (page 411), or Implicit Setup (page 424)
  
• Exercising the SUT by interacting with methods in its public or private interface
  
• Verifying that the expected outcome has occurred using calls to Assertion Methods
  
• Tearing down the test fixture using either Garbage-Collected Teardown (page 500), In-line Teardown (page 509), Implicit Teardown (page 516), or Automated Teardown (page 503)
  
  

The most common types of tests are the Simple Success Test (see Test Method), which verifies that the SUT has behaved correctly with valid inputs, and the Expected Exception Test (see Test Method), which verifies that the SUT raises an exception when used incorrectly. A special type of test, the Constructor Test (see Test Method), verifies that the object constructor logic builds new objects correctly. Both "simple success" and "expected exception" forms of the Constructor Test may be needed. The Test Methods that contain our test logic need to live somewhere, so we define them as methods of a Testcase Class.³ We then pass the name of this Testcase Class (or the module or assembly in which it resides) to the Test Runner (page 377) to run our tests. This may be done explicitly—such as when invoking the Test Runner on a command line—or implicitly by the integrated development environment (IDE) that we are using.

What's a Fixture?

The test fixture is everything we need to have in place to exercise the SUT. Typically, it includes at least an instance of the class whose method we are testing. It may also include other objects on which the SUT depends. Note that some members of the xUnit family call the Testcase Class the test fixture—a preference that likely reflects an assumption that all Test Methods on the Testcase Class should use the same fixture. This unfortunate name collision makes discussing test fixtures particularly problematic. In this book, I have used different names for the Testcase Class and the test fixture it creates. I trust that the reader will translate this terminology to the terminology of his or her particular member of the xUnit family.

Defining Suites of Tests

Most Test Runners "auto-magically" construct a test suite containing all of the Test Methods in the Testcase Class. Often, this is all we need. Sometimes we want to run all the tests for an entire application; at other times we want to run just those tests that focus on a specific subset of the functionality. Some members of the xUnit family and some third-party tools implement Testcase Class Discovery (see Test Discovery) in which the Test Runner finds the test suites by searching either the file system or an executable for test suites.

If we do not have this capability, we need to use Test Suite Enumeration (see Test Enumeration), in which we define the overall test suite for the entire system or application as an aggregate of several smaller test suites. To do so, we must define a special Test Suite Factory class whose suite method returns a Test Suite Object containing the collection of Test Methods and other Test Suite Objects to run.

This collection of test suites into increasingly larger Suites of Suites is commonly used as a way to include the unit test suite for a class into the test suite for the package or module, which is in turn included in the test suite for the entire system. Such a hierarchical organization supports the running of test suites with varying degrees of completeness and provides a practical way for developers to run that subset of the tests that is most relevant to the software of interest. It also allows them to run all the tests that exist with a single command before they commit their changes into the source code repository [SCM].

Running Tests

Tests are run by using a Test Runner. Several different kinds of Test Runners are available for most members of the xUnit family.

A Graphical Test Runner (see Test Runner) provides a visual way for the user to specify, invoke, and observe the results of running a test suite. Some Graphical Test Runners allow the user to specify a test by typing in the name of a Test Suite Factory; others provide a graphical Test Tree Explorer (see Test Runner) that can be used to select a specific Test Method to execute from within a tree of test suites, where the Test Methods serve as the tree's leaves. Many Graphical Test Runners are integrated into an IDE to make running tests as easy as selecting the Run  As  Test command from a context menu.

A Command-Line Test Runner (see Test Runner) can be used to execute tests when running the test suite from the command line, as in Figure 7.2. The name of the Test Suite Factory that should be used to create the test suite is included as a command-line parameter. Command-Line Test Runners are most commonly used when invoking the Test Runner from Integration Build [SCM] scripts or sometimes from within an IDE.

Figure 7.2. Using a Command-Line Test Runner to run tests from the command line.

>ruby  testrunner.rb  c:/examples/tests/SmellHandlerTest.rb
Loaded  suite  SmellHandlerTest
Started
.....
Finished  in  0.016  seconds.
5  tests,  6  assertions,  0  failures,  0  errors
>Exit  code:  0

Test Results

Naturally, the main reason for running automated tests is to determine the results. For the results to be meaningful, we need a standard way to describe them. In general, members of the xUnit family follow the Hollywood principle ("Don't call us; we'll call you"). In other words, "No news is good news"; the tests will "call you" when a problem occurs. Thus we can focus on the test failures rather than inspecting a bunch of passing tests as they roll by.

Test results are classified into one of three categories, each of which is treated slightly differently. When a test runs without any errors or failures, it is considered to be successful. In general, xUnit does not do anything special for successful tests—there should be no need to examine any output when a Self-Checking Test (page 26) passes.

A test is considered to have failed when an assertion fails. That is, the test asserts that something should be true by calling an Assertion Method, but that assertion turns out not to be the case. When it fails, an Assertion Method throws an assertion failure exception (or whatever facsimile the programming language supports). The Test Automation Framework increments a counter for each failure and adds the failure details to a list of failures; this list can be examined more closely later, after the test run is complete. The failure of a single test, while significant, does not prevent the remaining tests from being run; this is in keeping with the principle Keep Tests Independent (see page 42).

A test is considered to have an error when either the SUT or the test itself fails in an unexpected way. Depending on the language being used, this problem could consist of an uncaught exception, a raised error, or something else. As with assertion failures, the Test Automation Framework increments a counter for each error and adds the error details to a list of errors, which can then be examined after the test run is complete.

For each test error or test failure, xUnit records information that can be examined to help understand exactly what went wrong. As a minimum, the name of the Test Method and Testcase Class are recorded, along with the nature of the problem (whether it was a failed assertion or a software error). In most Graphical Test Runners that are integrated with an IDE, one merely has to (double-) click on the appropriate line in the traceback to see the source code that emitted the failure or caused the error.

Because the name test error sounds more drastic than a test failure, some test automaters try to catch all errors raised by the SUT and turn them into test failures. This is simply unnecessary. Ironically, in most cases it is easier to determine the cause of a test error than the cause of a test failure: The stack trace for a test error will typically pinpoint the problem code within the SUT, whereas the stack track for a test failure merely shows the location in the test where the failed assertion was made. It is, however, worthwhile using Guard Assertions (page 490) to avoid executing code within the Test Method that would result in a test error being raised from within the Test Method⁴ itself; this is just a normal part of verifying the expected outcome of exercising the SUT and does not remove useful diagnostic tracebacks.

Under the xUnit Covers

The description thus far has focused on Test Methods and Testcase Classes with the odd mention of test suites. This simplified "compile time" view is enough for most people to get started writing automated unit tests in xUnit. It is possible to use xUnit without any further understanding of how the Test Automation Framework operates—but the lack of more extensive knowledge is likely to lead to confusion when building and reusing test fixtures. Thus it is better to understand how xUnit actually runs the Test Methods. In most⁵ members of the xUnit family, each Test Method is represented at runtime by a Testcase Object (page 382) because it is a lot easier to manipulate tests if they are "first-class" objects (Figure 7.3). The Testcase Objects are aggregated into Test Suite Objects, which can then be used to run many tests with a single user action.

Figure 7.3. The runtime test structure as seen by the Test Automation Framework. At runtime, the Test Runner asks the Testcase Class or a Test Suite Factory to instantiate one Testcase Object for each Test Method, with the objects being wrapped up in a single Test Suite Object. The Test Runner tells this Composite [GOF] object to run its tests and collect the results. Each Testcase Object runs one Test Method.

Test Commands

The Test Runner cannot possibly know how to call each Test Method individually. To avoid the need for this, most members of the xUnit family convert each Test Method into a Command [GOF] object with a run method. To create these Testcase Objects, the Test Runner calls the suite method of the Testcase Class to get a Test Suite Object. It then calls the run method via the standard test interface. The run method of a Testcase Object executes the specific Test Method for which it was instantiated and reports whether it passed or failed. The run method of a Test Suite Object iterates over all the members of the collection of tests, keeping track of how many tests were run and which ones failed.

Test Suite Objects

A Test Suite Object is a Composite object that implements the same standard test interface that all Testcase Objects implement. That interface (implicit in languages lacking a type or interface construct) requires provision of a run method. The expectation is that when run is invoked, all of the tests contained in the receiver will be run. In the case of a Testcase Object, it is itself a "test" and will run the corresponding Test Method. In the case of a Test Suite Object, that means invoking run on all of the Testcase Objects it contains. The value of using a Composite Command is that it turns the processes of running one test and running many tests into exactly the same process.

To this point, we have assumed that we already have the Test Suite Object instantiated. But where did it come from? By convention, each Testcase Class acts as a Test Suite Factory. The Test Suite Factory provides a class method called suite that returns a Test Suite Object containing one Testcase Object for each Test Method in the class. In languages that support some form of reflection, xUnit may use Test Method Discovery to discover the test methods and automatically construct the Test Suite Object containing them. Other members of the xUnit family require test automaters to implement the suite method themselves; this kind of Test Enumeration takes more effort and is more likely to lead to Lost Tests (see Production Bugs on page 268).

xUnit in the Procedural World

Test Automation Frameworks and test-driven development became popular only after object-oriented programming became commonplace. Most members of the xUnit family are implemented in object-oriented programming languages that support the concept of a Testcase Object. Although the lack of objects should not keep us from testing procedural code, it does make writing Self-Checking Tests more labor-intensive and building generic, reusable Test Runners more difficult.

In the absence of objects or classes, we must treat Test Methods as global (public static) procedures. These methods are typically stored in files or modules (or whatever modularity mechanism the language supports). If the language supports the concept of procedure variables (also known as function pointers), we can define a generic Test Suite Procedure (see Test Suite Object) that takes an array of Test Methods (commonly called "test procedures") as an argument. Typically, the Test Methods must be aggregated into the arrays using Test Enumeration because very few non-object-oriented programming languages support reflection.

If the language does not support any way of treating Test Methods as data, we must define the test suites by writing Test Suite Procedures that make explicit calls to Test Methods and/or other Test Suite Procedures. Test runs may be initiated by defining a main method on the module.

A final option is to encode the tests as data in a file and use a single Data-Driven Test (page 288) interpreter to execute them. The main disadvantage of this approach is that it restricts the kinds of tests that can be run to those implemented by the Data-Driven Test interpreter, which must itself be written anew for each SUT. This strategy does have the advantage of moving the coding of the actual tests out of the developer arena and into the end-user or tester arena, which makes it particularly appropriate for customer tests.

What's Next?

In this chapter we established the basic terminology for talking about how xUnit tests are put together. Now we turn our attention to a new task—constructing our first test fixture in Chapter 8, Transient Fixture Management.