There are two aspects of time and testing I want to cover here. The first is where the time comes from to write these automated tests. I am always looking for win-win-win programming practices, practices that provide:

The win for me in the long term comes from being able to change my code long after I’ve written it without accidentally breaking anything. Code that I can continue to change for a long time is code that I can charge for changing for a long time. The win for my teammates comes from the ease with which well-tested code can be integrated with the rest of the system. The win for my customers comes from the reduction in the number of defects in tested code and also from the option to continue adding features to tested code indefinitely.

Okay, that takes care of two of the wins. What is the short-term win in automating tests? If a programmer automating tests takes more time today than a programmer not automating tests today or even this week, then programmers won’t automate tests. Where does the time come from today? In a word, debugging.

It wasn’t until I had been automating tests for several years that I noticed that I didn’t use the debugger. It wasn’t that I used the debugger a little less. I only used the debugger every few months, and this from a programmer who used to live in the debugger. What was going on?

First, the tests act as a defect detector. Many times when I introduce a defect accidentally, a test immediately fails. When I say “immediately,” I mean I know within seconds or minutes that I have broken something. When this happens, I often have several alternatives in mind, one of which I choose to implement. “Oh, just incrementing that doesn’t work. I guess I’ll have to recalculate.” Instead of having to go into a debugger and figure out what went wrong, I can solve my problem a different way.

When I cause a problem in code I’m not working on directly, I still get immediate feedback about the problem. In this case, though, I won’t necessarily have an alternative in mind. I do know, however, to stop right then and dig deeper into what is going on. The pattern of test failures gives me valuable clues into the problem, so often I don’t have to resort to a debugger to find the problem. The tests themselves provide the tea leaves I can read to find the problem.

Which brings me to the second aspect of time I want to discuss with respect to automating tests: the Defect Cost Increase (DCI). DCI states that the sooner you test after the creation of an error, the greater your chance of finding the error and the less it costs to find and fix the error, as illustrated in Figure 1-1. This theory explains why leaving testing until just before releasing software is so expensive and error-prone. Most errors don’t get caught at all. The cost of finding and then fixing the ones you do catch is so high that you have to play triage with the errors because you just can’t afford to fix them all.

Figure 1-1. Frequent testing leaves fewer defects at the end

It is not possible to catch all the defects. However, testing more frequently reduces the number of defects left at the end. The value of fewer defects has to be traded off against the cost of testing. (This was always assumed to be a tradeoff.) If developers are more efficient writing automated tests, there is no tradeoff. Frequent testing doesn’t cost; it pays.

Developers automating tests don’t spend much time debugging. If you find an error in seconds or minutes and fix it, it is fixed. The time it takes is not significant enough to track or name. If it takes hours or days to track down a problem, fix it, and gain reasonable confidence that you haven’t broken anything else, then we give that activity the name “debugging” (however long it takes).

When the person doing the testing and the person committing the errors are separate people, there is a natural limit to how close the testing can get to the introduction of the error. This places a limit on the benefits you can gain from the DCI. By having the same person code and test, you can bring the proximity from days or weeks down to minutes. If the benefit gained from closer proximity is great enough, the programmer can visibly get more done even in a day, thus satisfying our need for the third, short-term, win.

The goal in most programming is not perfect code or perfect technique, but code and technique that are continually improving. One objection to programmers automating tests that I often hear is Dijkstra’s saying that “Tests can prove the presence of defects but not their absence.” While I would not presume to contradict Dijkstra, what he says is true but irrelevant. Most programmers make many mistakes. Automating tests is a way to make fewer mistakes. While your code will still not be perfect, even with excellent tests; you will likely see a dramatic reduction in defects once you start automating tests. And your effectiveness will continue to improve.

As testers have been telling us for years, there is a real art to testing effectively. The tests you write in your first month will be nowhere near as effective at catching defects as the tests you write after a year.

So, the goal of automating tests is justified confidence. You can use the confidence dividend to take more daring leaps in design, get along with your teammates better, improve relations with your customers, and go home every night with proof that the system is better now than it was this morning because of your efforts.

setUp( ) and tearDown( ) are nicely symmetrical in theory but not in practice. In practice, you only need to implement tearDown( ) if you have allocated external resources like files or sockets in setUp( ). If your setUp( ) just allocates plain Java objects, you can generally ignore tearDown( ). However, if you allocate many megabytes of objects, you might want to clear the variables pointing to those objects so they can be garbage-collected because the test case objects are not garbage-collected at any predictable time.

What happens when you have two tests with slightly different setups? There are two possibilities:

JUnit doesn’t provide convenient support for suite-level setup. There are a few good reasons to share fixtures between tests, but the vast majority of times I’ve seen shared fixtures, it was because there was an unresolved design problem.

For example, you might want to log in to a database once and then use that database connection in several tests. This makes your tests run faster. To do this, wrap the test suite returned by the static suite( ) method in a TestSetup, which overrides setUp( ) to open the database connection and tearDown( ) to close the connection.

class DatabaseTests extends TestSetup {
    DatabaseConnection connection;
    protected void setUp(  ) throws Exception {
        connection= new DatabaseConnection("kent",
"password")
    }
    protected void tearDown(  ) throws Exception {
        connection.close(  );
    }
}
public static Test suite(  ) {
    Test suite= new TestSuite(this.class);
    return new DatabaseTests(suite);
}

I can’t emphasize enough that sharing fixtures between tests reduces the value of your tests. The underlying design problem is that objects are too closely bound together. You will have better results solving the design problem and then writing tests using stubs, rather than creating dependencies between tests at runtime and ignoring the opportunity to improve your design.

Most of the time you will encounter five classes or interfaces when you are using JUnit:

Figure 1-4 shows the relationship of the five basic classes in JUnit: Assert, Test, TestCase, TestSuite, and TestResult.

Figure 1-4. The five basic classes in JUnit

Most test cases written for JUnit are derived indirectly from the class Assert, which contains methods for automatically checking values and reporting discrepancies. The methods are declared static so you can write design-by-contract style assertions in your methods and have them reported through JUnit:

add(Object item) {
    junit.framework.Assert.assertNotNull(item);
    ...

Most of the time, though, you will be checking the assertions inside of tests.

There are two variants of each of the assertion methods: one takes as a parameter a message to be displayed with the error and one doesn’t. The optional message is typically displayed when a failure is displayed. This can make debugging easier. Some people use the messages commonly; others (like me) don’t. Here is the error message you get if you call assertTrue(“Zero should equal zero”, false):

1) testWithMessage(VectorTest)
junit.framework.AssertionFailedError: Zero should equal
zero
    at VectorTest.testWithMessage(VectorTest.java:7)

Table 1-1 shows all the varieties of assertions.

You may find that you need other assertions than these to compare objects specific to your project. Create your own Assert class to contain these assertions to simplify your tests.

Failing assertions all call a single bottleneck method, fail(String message), which throws an AssertionFailedError. There is also a variant which takes no parameters. Call fail( ) explicitly when your test encounters an error. The test for an expected exception is an example. Table 1-2 lists the bottleneck methods in JUnit.

Test is the generic interface implemented by all objects that can act as tests. It’s a pity the interface is called Test, because implementors actually represent one or more tests. The two methods are shown in Table 1-3.

TestCase and TestSuite are the two most prominent implementors of Test. You can implement Test yourself. We deliberately kept it small so it would be easy to implement. For example, if you wanted to implement a test scripting language you could create a TestScript class and override the two methods in Test. CountTestCases( ) would return 1. run(TestResult) would run a script with the results reported on the TestResult.

Your test case classes will inherit from junit.framework.TestCase. Most of the time you will run tests from automatically created test suites. In this case each of your tests should be represented by a method named (by convention) test*.

TestCase implements Test. CountTestCases( ) returns 1. Run(TestResult) runs setUp( ), runs the test method, and then runs tearDown( ), reporting any exceptions to the TestResult.

Table 1-4 shows the external protocol implemented by TestCase.

There are two template methods, setUp( ) and tearDown( ), you can override to create and dispose of the objects against which you are going to test. Table 1-5 shows these methods.

A TestSuite is a Composite of Tests. At its simplest, it contains a bunch of test cases, all of which are run when the suite is run. Since it is a Composite, however, a suite can contain suites which can contain suites and so on, making it easy to combine tests from various sources and run them together.

Generally, you will never see TestSuites directly. Most IDEs create suites for you. In case you have to create suites yourself, here is the public interface.

In addition to the Test protocol—run(TestResult) and countTestCases( )—TestSuite contains protocol to create named or unnamed instances. Table 1-6 shows the instance creation protocol for TestSuite.

TestSuite also contains protocol for adding and retrieving Tests, as shown in Table 1-7.

While you are running all these tests you need somewhere to store all the results: how many tests ran, which failed, and how long they took. TestResult collects results. A single TestResult is passed around the whole tree of tests. When a test runs or fails, the fact is noted in the TestResult. At the end of the run, the TestResult contains a summary of all the tests.

TestResult is also a subject that can be observed by other objects wanting to report testing progress. A graphical test runner might, for example, observe the TestResult and update a progress bar every time a test starts.

JUnit distinguishes between failures and errors. A failure is a violated JUnit assertion. An error is an unexpected exception. Sometimes this distinction proves useful since errors are generally easy to fix (e.g., NullPointerException means a variable wasn’t initialized), while failures are generally harder to fix. If you have a big list of problems, it’s best to tackle the errors first and see if you have any failures left when they are all fixed. Sometimes I wonder if this distinction is worth the trouble, though, as it complicates the interface. Table 1-8 summarizes the methods for reporting failures and errors.

(Enumeration is the old style of iterator. It remains in JUnit so we maintain compatibility with as many versions of Java as possible.)

If you want to register as an observer of a TestResult, you need to implement TestListener. To register, call addListener( ) as shown in Table 1-9.

TestListeners implement the four methods shown in Table 1-10.

All the classes above come from junit.framework. Here are all the packages in JUnit:

Here is a short example of programming a factorial function test-first. I always begin a test-first programming session with an outline of the tests I want to write. For factorial, I can only think of two important tests:

I always start with a simple example. My goal is never to have a failing test for more than a few seconds. That way I know exactly where I am. If I’m feeling adventuresome, or I’m just exploring, I might have a failing test for longer than that, but there is a useful discipline in being able to stay really close to your tests. The first test looks like this:

public void testOne(  ) {
    assertEquals(1, factorial(1));
  }

Notice that I’ve left the interface as simple as possible. In this case I don’t need an interface any more complicated than a private method in the test-case class. I don’t expect this interface to last, but leaving the interface simple for the moment allows me to gain experience with my code at the least possible cost, giving me as much information as possible to make better design choices later.

The implementation of factorial( ) can be simple since we only have the one test to satisfy so far:

private int factorial(int i) {
    return 1;
  }

We run the test and it passes. Now we can write the second test. What input should we use? There is an art and science to correctly picking the domain of tests. Having read the books about testing, I rely on my gut to tell me how many inputs I need to test and which inputs I need to test. My tendency is to pick the simplest inputs that force me to write the code I think I need. Some people would pick a larger number, such as 5. For my purposes, it is just as good to pick 2, and it makes it easy for me to manually calculate the right answer (2). This introduces the risk that I pick simplistic inputs that don’t really exercise the code, but I haven’t found this to be a problem in practice. The attitude I take toward writing code is far more important than the precise test inputs I pick. If I set out to write solid code, I do, and even simple tests help me. If, for whatever reason, I’m not committed to solid code, no amount of ceremony around test writing is going to result in solid code. It’s my attitude that needs adjusting. So for the sake of simplicity, my second test looks like this:

public void testTwo(  ) {
    assertEquals(2, factorial(2));
  }

Now I can write the recursive call to factorial:

private int factorial(int i) {
    if (1 == i) return 1;
    return i * factorial(i-1);
  }

And the tests both pass. But what about inputs less than 1? I expect an exception to be thrown. Here’s a test for that:

public void testZero(  ) {
    try {
      factorial(0);
      fail(  );
    } catch (IllegalArgumentException e) {
    }
  }

This requires that we check the parameter before making the recursive call:

private int factorial(int i) {
    if (i < 1) throw new IllegalArgumentException(  );
    if (1 == i) return 1;
    return i * factorial(i-1);
  }

With these tests in place, we can confidently change the implementation of factorial, for example by replacing the recursive call with a loop. If we accidentally break the code, one of the tests should break.

Every program ever written has expected behavior in the face of expected inputs. Sometimes it can be a challenge to figure out how to measure the expected behavior, or to segregate the expected inputs into meaningful equivalence classes, but in the end it’s always been possible for me, and it’s always been valuable.

A recent challenge was figuring out how to write tests first for the code for Contributing to Eclipse. The example was a plug-in that extended the functionality of Eclipse. How do you test a plug-in that doesn’t yet exist? It took us a couple of months of ordinary poke-and-peek exploratory testing before the lightbulb went on. First you make a test plug-in that depends on the plug-in you really want to create. This causes an error. To resolve the error, you have to create the real plug-in. Then you start writing tests in the test plug-in that expect the real plug-in to behave in certain ways that it doesn’t, like displaying some information in a particular format.

Not all examples of test-first programming can be resolved this easily. One of the biggest challenges in all kinds of unit testing is figuring out how to deal with expensive, volatile external resources such as databases or servers. Whether you are writing tests before or after coding, stubs (the topic of the next section), will help you keep writing the valuable parts of tests and running these tests quickly and reliably.

Test-driven development (TDD) joins test-first programming and incremental design. In TDD you only add as much design as necessary to a system to pass the current tests with clean code. With test-first programming, you can take an upfront design and implement it, one test at a time. With TDD, you delay commitment to any particular design until the code calls for it. If you can get smooth enough at incremental design, this can lead to designs with no “fat,” nothing put in out of speculation or from fear.

One of the effects of using stubs is that your designs tend to improve. Widely used resources are accessed through a single façade so you can easily replace the resource with the stub. For example, instead of having direct database calls scattered throughout the code, you have a single Database object, an implementor of the IDatabase interface. Then you can create a stub implementation of IDatabase and install it before running your test. Functionality that needs to be stubbed at the same time tends to cluster in the same object, improving cohesion. By presenting the functionality with a single, coherent interface you reduce coupling with the rest of the system.

Sometimes you need to check that an object has been called correctly. You can create a full-blown stub of the object to be called, but then it can be inconvenient to check for correct results. A simpler solution is to use the test-case object itself as a stub, which is called self-shunting. The name is taken from the medical practice of installing a tube that takes blood from an artery and returns it to a vein to provide a convenient place for injecting drugs.

Here is an example. Suppose we want to test that the correct SQL query is generated when we ask a Librarian object to fetch us some accounts. First, we make our test case class an implementor of IDatabase:

public class QueryTest
    extends TestCase
    implements IDatabase

Next we implement the one IDatabase method, query( ), to check to make sure that the correct SQL is generated. We also set a flag so we can detect if the database is not called at all:

public boolean wasCalled= false;

    public void query(String query) {
        assertEquals("select * from Accounts", query);
        wasCalled= true;
    }

Now we can write our test. Notice that we create the Librarian with a particular database instead of relying on a single global database. This style of design is encouraged by stubbing. It reduces the coupling between objects and improves reuse.

public void testQuery(  ) {
        Librarian l= new Librarian(this);
        l.fetchAccounts(  );
        assertTrue(wasCalled);
    }

The biggest problem with shunting is that if you are unfamiliar with the technique, the tests can be hard to read. What is going on here? Why is a test case also a database? Once you get used to the idiom, the tests are easy to read. Everything you need to understand the test is in one class. Another problem occurs when shunting interfaces with many methods. If there are three tests and 100 stubbed-out methods, few of which even have stubbed implementations, it can be hard to find the real tests amongst all that noise. The best solution to this problem is to not have interfaces with hundreds of methods. If you do have wide interfaces, you will have to choose between cluttering the test-case class with lots of trivial methods and the additional complexity of a separate class for the stub.

Testing listeners is another good use for shunting. Refer to junit.tests.framework.TestListenerTest for an example.

When I get a defect report, my impulse is to fix the defect as quickly as possible. This impulse has never served me well. I’m likely to cause another defect with my “fix.”

With tests, I have a ritual I can use to hold my impulse in check.

The search for the smallest reliable reproduction of the defect gives me a chance to really look at the cause of the defect. The test I write improves the chances that when I fix the defect, I really fix it (without tests, I have a tendency to cause a new defect when I fix a defect). All the other tests I have reduce the chances of inadvertently causing another problem. Because I add the test to the global test suite, I reduce the chances of accidentally undoing a fix—another persistent problem I’ve experienced.

I recommend finding a colleague and adopting the above ritual for fixing defects together. When you just can’t think of a test for a defect, that’s a good time to work on it with a partner. After a while, “thinking in tests” will become a habit. In the meantime, it’s good to share the pain, frustration, eventual success, and satisfaction with a colleague.

Another way I use tests is when I need to learn a new API. If I use a new API to solve a new problem, I have two variables in play. I don’t know whether problems are caused because my overall approach to solving the problem is wrong or whether it’s my use of the API. Exercising the API with tests removes one of those variables.

For example, suppose I’ve just been handed a new set of collection classes. One, BoundedList, is only supposed to hold a fixed number of elements. An attempt to add too many elements should result in an exception. I think I understand this description, but do I really? Only experience will tell. How are you going to get that experience? Writing tests is a simple way that gives you immediate and concrete feedback.

What’s my first test? I want to be sure the collection behaves like other collections, unless I add too many items. Here’s what I write:

public void testOverflow(  ) {
    List bounded= new BoundedList(1);
    Object element= new Object(  );
    bounded.add(element);
    assertTrue(bounded.contains(element));
    try {
        bounded.add(element);
        fail("Should have thrown exception");
    } catch (CollectionOverflowException e) {
    }
}

I can go through the rest of the API, writing tests whenever I hit a method I want to make sure I understand. When I’m trying to learn a new API and use it for real at the same time, my mind is divided between the two tasks. Writing tests to learn an API goes more quickly, because I’m concentrating on learning. With the learning firmly in place, I can go ahead and solve my real problem with confidence.

Suppose you decide to use an external package as part of your development. You are now exposed to the risks that the package won’t behave as you expect and that future versions of the package will change in subtle ways that will break your code without you knowing it. How can you inexpensively address these risks?

One way is to write a test every time you make an assumption about how the package works. If you test passes, your assumption is valid.

Suppose I am writing a test runner that is going to use the automatic test suite creation built into JUnit. I assume that if I have a test-case class with one test in it and I create a suite for that class, the suite returned should have a single test in it. I can say this with a test:

public class ExampleSuite extends TestCase {
    public void testNothing(  ) {
    }
}
public void testSuiteBuilding(  ) {
    TestSuite suite= new TestSuite(ExampleSuite.class);
    assertEquals(1, suite.countTestCases(  ));
}

If I have the discipline to document all of my assumptions with tests, then future releases of packages hold no fear for me. The first thing I do is run my assumption suite. If it passes, I expect my system to continue working. If it doesn’t, I need to track down what has changed because it is pretty much guaranteed that my system won’t work any more.

When you document assumptions with tests, you own the tests. This assumes an arms-length relationship between you and the supplier of the package on which you are building. If you want or have a closer relationship with a supplier, you can use the tests as a way of coordinating your activities.

When you want to agree on an API with a supplier, you can jointly write and maintain the tests. Sit down together and code the tests so you reveal as many assumptions as possible. Hidden assumptions are the death of cooperation. The tests tell the supplier exactly what is expected of them. They should come back when the tests all run. If it is impossible to get a test to run as written, then it’s time to renegotiate the API. If no one changes the tests unilaterally, the chances of smooth integration go way up.

By adding stubs to tests, you can further decouple two teams. If you not only write the tests but also get them to run with stub objects too, two teams can develop independently. The supplier team’s job is to make the tests run by replacing the stub objects with real implementations. The customer’s job is to make their own code work with the stub objects until such time as they have the real implementation.

For example, suppose I want a simple little XML parser from you. Together we write this test:

public void testParsing(  ) {
        IXMLParser parser= new StubXMLParser(  );
        INode tree= parser.
            parse("<parent><child/></parent>");
        assertEquals("parent", tree.getName(  ));
        assertEquals("child",
            tree.getChild(0).getName(  ));
    }

There are two interfaces in the API, IXMLParser and INode. We can provide trivial implementations of them, enough so I can begin coding as if parsing really works. The parser just returns a hardwired node:

public class StubXMLParser implements IXMLParser {
        public INode parse(String source) {
            return new Node("parent");
        }
    }

The INode implementation is more complete, but it still returns a hardwired subnode:

class StubNode implements INode {
        protected String name;

        public StubNode(String name) {
            this.name= name;
        }

        public String getName(  ) {
            return name;
        }

        public INode getChild(int index) {
            return new StubNode("child");
        }
    }

I talked to a team who used this approach of combining tests with stub implementations. Their supplier was a hardware team creating customer chips. When a new version of the chip arrived, the first thing they did was run the tests. If the tests didn’t run, it wasn’t worth the cost of testing any of the rest of the system, because the tests documented the assumptions the software made about the hardware. With the stub implementation, though, the software team could begin developing as if the hardware was already complete (subject to the usual discovery of deeply buried assumptions when the real hardware became available).

We have tried to keep JUnit as simple as possible, to accommodate as many different styles of usage as possible. There are many useful extensions that others have created for more specialized purposes. Here are a few you may want to try:

So far we’ve seen only the most basic way to run tests using the test element. The test element also gives you the opportunity to override the values of haltonfailure and haltonerror for a given test.

The batchtest element creates and runs a test element for each file in a file set. With two test classes, SampleTest and AnotherTest, the following batchtest runs them both:

<junit>
    <formatter type="brief" usefile="false"/>
    <batchtest>
        <fileset dir=".">
            <include name="**/*Test.java"/>
        </fileset>
    </batchtest>
</junit>

The output is just as if each test class had been declared in its own test element.

Buildfile: ...uild.xml
test:
    [junit] Testsuite: guide.AnotherTest
    [junit] Tests run: 1, Failures: 0, Errors: 0,
Time elapsed: 0 sec
    [junit] Testsuite: guide.SampleTest
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time
elapsed: 0 sec
BUILD SUCCESSFUL

The JUnit task includes a nested element describing a formatter for printing or otherwise processing feedback from tests. There are three built-in formatters, or you can write your own. The simplest formatter is called brief, and it is invoked with the type attribute of the formatter element like this:

<junit>
    <formatter type="brief" usefile="false"/>
    <test name="guide.SampleTest"/>
</junit>

I have set the usefile attribute to false so the formatter writes output to the console. The results look just like the results of setting the printsummary attribute to on:

Buildfile: ...uild.xml
test:
    [junit] Testsuite: guide.SampleTest
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time
elapsed: 0 sec
BUILD SUCCESSFUL

If I insert a call to fail( ) in my test case, the formatter prints out a stack trace of where the error occurred:

Buildfile: ...uild.xml
test:
    [junit] Testsuite: guide.SampleTest
    [junit] Tests run: 1, Failures: 1, Errors: 0, Time
elapsed: 0 sec
    [junit] Testcase: testEmpty(guide.SampleTest):FAILED
    [junit] null
    [junit] junit.framework.AssertionFailedError
    [junit] at guide.SampleTest.testEmpty(SampleTest.java:
8)
    ...
    [junit] TEST guide.SampleTest FAILED
BUILD SUCCESSFUL

The plain formatter prints out how long each test case took to run:

<junit>
    <formatter type="plain" usefile="false"/>
    <test name="guide.SampleTest"/>
</junit>
Buildfile: ...uild.xml
test:
    [junit] Testsuite: guide.SampleTest
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time
elapsed: 0 sec
    [junit] Testcase: testEmpty took 0 sec
BUILD SUCCESSFUL

The XML formatter writes the same information, but in a format suitable for post processing. Set the type attribute to xml to obtain it. Notice that I’ve eliminated the usefile attribute. By default, the output goes to a file whose name is derived from the name of the class called for in the test element.

<junit>
    <formatter type="xml"/>
    <test name="guide.SampleTest"/>
</junit>

Here is the output for a test that fails:

<testsuite name="guide.SampleTest" tests="1" failures="1"
errors="0" time="0.02">
  <properties>
   ...
  </properties>
  <testcase name="testEmpty" classname="guide.SampleTest"
time="0.0">
    <failure type="junit.framework.AssertionFailedError">
    junit.framework.AssertionFailedError
    at guide.SampleTest.testEmpty(SampleTest.java:8)
    ...
    </failure>
  </testcase>
</testsuite>

You are not limited to the built-in formatters. You can write your own by creating a class that implements the interface org.apache.tools.ant.taskdefs.optional.junit.JUnitResultFormatter. Here is a simple formatter that beeps whenever a test fails:

public class BeepFormatter
    implements JUnitResultFormatter {
    public void addFailure(
        Test test, AssertionFailedError t) {
            Toolkit.getDefaultToolkit().beep(  );
    }
    ...// stub implementations of other methods
}

Invoke this formatter with the classname attribute of the formatter element:

<junit>
    <formatter classname="guide.BeepFormatter"/>
    <test name="guide.SampleTest"/>
</junit>

You can declare as many formatters as you want in the build file.

By integrating JUnit tests with Ant, you can have the advantages of automatically building an entire system with the advantages of fine-grained feedback from automated tests. The result is more confidence in the system being built and quicker identification of errors.

The text user interface gives you a command-line interface for running tests and textual feedback about test progress and results. Invoke the text test runner this way:

java junit.textui.TestRunner [-wait] TestClassName

The —wait option doesn’t terminate the test runner until a character has been typed at the console. This is useful if you are running the tests in a window that stays open only as long as the test runner is active.

Here are the results of running JUnit’s own tests. The periods represent the start of a new test case. “F” or “E” represent a failure or error. After all the tests are run, the stack traces of all failures and errors are printed followed by a summary of the number of tests run and the number of failures and errors.

.........................................
.................................F........
.........
Time: 1.061
There was 1 failure:
1) testJarClassLoading(junit.tests.runner.
TestCaseClassLoaderTest)junit.framework.
AssertionFailedError: Cannot find test.jar
    at junit.tests.runner.TestCaseClassLoaderTest.
testJarClassLoading(TestCaseClassLoaderTest.java:31)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
    at sun.reflect.NativeMethodAccessorImpl.
invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.
invoke(DelegatingMethodAccessorImpl.java:25)

FAILURES!!!
Tests run: 91,  Failures: 1,  Errors: 0

The output of the text test runner is ugly, but it is still useful from time to time. Its best feature is that it starts up very quickly. If you want to run tests every ten seconds, opening a window is too slow.

Figure 1-5 shows the AWT runner, the first graphical JUnit runner. Its primary advantage is that it is extremely simple. The original implementation was by André Weinand, with logo design by Erich Gamma and me.

Figure 1-5. AWT-based Test Runner

The top Run button runs all the tests found in the class named in the fill-in field at the top. Errors and failures are listed below. The bottom Run button reruns the selected failing test case, with the results printed in the status bar at the bottom of the window.

The checkbox Reload classes every run deserves some explanation. It is inconvenient to have to open a new window every time you want to run tests. When you check “Reload,” JUnit uses a custom class loader to throw away all of the test and model classes every time Run is pressed. This has the effect, under the right circumstances, of letting you leave the test runner window up. When you recompile, you need only press Run to get updated results.

I say “under the right circumstances” because this feature is the source of many problems. Some Java code expects to be using its own class loader and won’t work when used with JUnit’s loader. The short-term solution is to put a pattern that identifies classes relying on their own class loader into the excluded.properties file. The default excluded.properties file contains the following entries:

excluded.0=sun.*
excluded.1=com.sun.*
excluded.2=org.omg.*
excluded.3=javax.*
excluded.4=sunw.*
excluded.5=java.*
excluded.6=org.w3c.dom.*
excluded.7=org.xml.sax.*
excluded.8=net.jini.*

Any class matching one of these patterns will be loaded by its original class loader, not JUnit’s. You don’t want to put any classes on this list that will be recompiled between test runs, or you’ll run your tests with obsolete classes and get inconsistent results.

The real solution to this problem is to start up a new JVM every time Run is pressed, so you are guaranteed to get the latest class files loaded. This feature is scheduled for JUnit 3.9. In the meantime, you need to be aware of class-loader-sensitive code and be sure to put those classes on the excluded list.

Figure 1-6, the Swing runner, is a transliteration of the AWT UI to use Swing widgets.

Figure 1-6. Swing-based Test Runner

The top and bottom Run buttons in the Swing runner operate just like the same buttons in the AWT runner, rerunning all tests or a single test, respectively. The Swing runner also remembers the last several tests run. You can load one of these with the ... button.

The Results section shows a list of failures and errors and also a hierarchical view of your entire test suite, as shown in Figure 1-7.

The Test Hierarchy tab allows you to see all of your tests in context. Failed tests are highlighted with a red X.

Figure 1-7. Test Hierarchy tab

The JUnit support in Eclipse was written by Erich Gamma and co-op students. Eclipse works to make test writing and test running natural parts of the programming experience.

Setting up a project so you can write tests

To write JUnit tests in an Eclipse project, you must first make JUnit visible. When you create a project, the second page of the project wizard gives you the option to change the build settings. Click the Libraries tab and press Add External JARs to add the JUnit JAR files to the project’s build path. In the file browser navigate to your Eclipse installation directory, then to plugins/org.junit. Select junit.jar, as shown in Figure 1-8.

Figure 1-8. Project wizard change options

To add a test to an existing project, you can make JUnit visible from the Project Properties page. Select the Java Build Path page, the Libraries tab, and use Add External JARs to add junit.jar as above.

In Eclipse 3.0 this is even simpler. If you add a test case using the test case wizard, the wizard will offer to add the junit.jar for you.

Writing tests

To create a test-case class, use subclass junit.framework.TestCase. You can use the test-case wizard (Figure 1-9) to fill in the superclass for you, as shown in Figure 1-10.

Figure 1-9. Invoking the TestCase Wizard

Figure 1-10. Creating a new TestCase

There is an editor template for writing a new test method. To invoke it, type test in a Java editor and press Ctrl-Space. Select test and you’ll have a blank test method with the name of the method selected (see Figure 1-11).

Figure 1-11. Creating a new Test Method

Eclipse’s Quick Fix facility saves a lot of work when used in conjunction with JUnit and TDD. You can write tests that refer to classes, methods, and members that don’t exist yet, then use Quick Fix to create stubs for the missing elements with a minimum of fuss. Here I’ve written a test that calls Factorial.calculate( ), a nonexistent method on a nonexistent class. Clicking the light bulb in the left margin brings up a menu that contains exactly the activity we wish: to create a class (see Figure 1-12).

Figure 1-12. Creating a class with QuickFix

It takes only a few clicks to resolve the compile error, leaving us with a failing test ready to be satisfied.

Running tests

To run tests, select a project, package, class, or test method in the Package Explorer or Outline and choose Run → Run As → JUnit Test. The JUnit view will appear, showing you the results (see Figure 1-13).

Figure 1-13. JUnit view with results

Double-clicking a failing test opens the definition of that test in the Java editor. Double-clicking a stack frame in the failure trace navigates to that line in the Java editor.

The Hierarchy tab in the JUnit view shows you all the tests run along with their status (pass, fail, or error). Double clicking one of the tests shows you the source for that test in the Java source code editor.

My setup when programming in Eclipse is to dock the JUnit view (drag it to the fast view bar) and set the JUnit preference to show only the view on failures or errors. The JUnit view icon fills up to indicate the progress being made, so I can go on with other tasks while the tests run. I am interrupted with the details only if an error occurs (as shown in Figure 1-14).

Figure 1-14. JUnit preferences: view only on error

Then I use ctrl-F11 to rerun the same tests after each programming change.

JBuilder, from Borland, includes support for writing and running JUnit tests, as well as extensions to JUnit to help create reusable test fixtures.

Setting up a project so you can write tests

To write JUnit tests in JBuilder, you must first make JUnit visible inside the project. On the second page of the project creation wizard, select the Required Libraries tab and press Add.... From there you can add JBuilder/JUnit (see Figure 1-15).

Figure 1-15. Adding the JUnit Library to a new project

For an existing project, you can edit the project properties to add JUnit to the Required Libraries, as shown in Figure 1-16.

Figure 1-16. Adding the JUnit Library to an existing project

Writing tests

JBuilder provides a wizard for creating new test cases. Select File → New... → Test → Test Case. You can type in the name of the test-case class (see Figure 1-17).

Figure 1-17. New Test Case Wizard

JBuilder’s ErrorInsight combines smoothly with JUnit for test-driven development. Type in a test that refers to classes or methods you have not yet created, and JBuilder will offer to create the missing elements for you (see Figure 1-18).

Figure 1-18. ErrorInsight creates a missing class for you

One of the design decisions we made in JUnit was keeping the testing methods in the same class as the code for creating and deleting test fixtures. This is potentially risky if you want to use the same fixture with test methods in several different classes or the same test methods with several different fixtures. In practice, this isn’t usually a problem. You can create an abstract superclass for the common test fixture and put various sets of test methods in the different subclasses. However, this is a pragmatic solution, not necessarily the cleanest design.

JBuilder supports pure fixture classes, classes that create and delete predictable sets of objects. The fixture classes support setUp( ) and tearDown( ) methods that you can override. JBuilder has wizards for fixtures that create JDBC connections, perform JNDI lookups, build own custom fixtures. The test case wizard contains a page to specify which fixtures to create and initialize when an instance of your test class is initialized.

Running tests

JBuilder integrates a convenient test runner with links to the rest of the IDE, as shown in Figure 1-19.

Figure 1-19. Test Runner Output

The icons on the left show:

• The hierarchy of tests that were run

• The output of the tests

• The failures and errors

The failures and errors view on the right shows links to source files using the familiar browser link format. For example, clicking on SampleTest.java:8 above navigates to that file and line.

IntelliJ IDEA, from JetBrains, has also embraced writing and running JUnit tests as part of the normal flow of programming. The JUnit integration makes it natural to write and run tests, and to code test-first.

Setting up a project so you can write tests

You need to add the JUnit library as a library for your module (a project in IDEA can have several modules.) Select a module in which you want to write tests. Choose File → Module Settings.... Select the Libraries (Classpath) tab in the resulting dialog (Figure 1-20).

Figure 1-20. Adding the JUnit JAR

Select Add Jar/Directory.... Navigate to the JUnit JAR file. IDEA comes with a version of JUnit, illustrated in Figure 1-21.

Figure 1-21. Choosing the JUnit JAR

Click OK twice. Now you are ready to start writing tests.

Writing tests

Writing tests in IDEA is just the same as writing any other Java class. There are no special templates or shortcuts. Create a class using the New → Class context menu item on a package. When you edit the generated class code, you will have to extend junit.framework.TestCase. IDEA helps insert the correct import for you, as shown in Figure 1-22.

Figure 1-22. Automatically inserting the import statement

Write test methods just as you would write any other Java method. Again, there are no templates or shortcuts to help you, but typing in a new class or method is not that much work. Code completion gives you a handy overview of the variations of the assertXXX methods available, as shown in Figure 1-23.

Figure 1-23. Automatically completing the assertEquals call

Intention Actions in IDEA help when you are coding test-first. They allow you to quickly create classes and methods referred to in your tests. Figure 1-24 shows IDEA offering to create a missing method.

Figure 1-24. Automatically creating a stub method

Figure 1-25 shows the stub method IDEA created. IDEA chooses reasonable defaults for types and variables names and offers alternatives.

Figure 1-25. Assistance filling in the correct return type

Running tests

Running any Java program in IDEA is controlled by a “run configuration,” a description of all the parameters to be used to run the program. There are several flavors of run configuration. One runs tests and reports the results. Run configurations can be created manually or automatically from the various places in the user interface. This is described later. To create a JUnit run configuration manually, select the menu item Run → Edit Configurations. Press the JUnit tab, and you’ll see the dialog shown in Figure 1-26.

Figure 1-26. Run configuration for a JUnit test

You have three options:

• Run all the tests in a given package

• Run all the tests in a given class

• Run a single test in a given class

IDEA will create a run configuration for you. Figure 1-27 shows the Run "All Tests" menu item on a package. This item creates a run configuration containing all the tests in the selected package and all dependents and then runs it. When you press the Run button after this, these tests will be rerun.

Figure 1-27. Running the tests

IDEA will also create and execute a run configuration for you from inside the text editor. If the insertion point is inside the text of a test class, you will see the Run menu item in the context menu, as shown in Figure 1-28.

Figure 1-28. Running a particular test

The test runner in IDEA has the most features of the IDEs covered here. Figure 1-29 shows the output of a failed test.

Figure 1-29. Feedback from a failed test

While the tests are running, the controls along the left side allow you to pause, stop the tests, or rerun a single test. The controls along the top allow you to filter the test results and control how selecting test results affects the text editor—for example, scrolling to the source of a selected test.

The Output tab of the right pane of the test runner shows the raw output of the selected test (any text written to the console) and the stack trace of any failure or error. The Statistics tab, shown in Figure 1-30, shows how long each test took, its result, and how much memory it consumed. I find it useful in finding out which tests are slow.

Figure 1-30. Tabular feedback from a test run