7

Test Setup and Teardown

Have you ever worked on a project where you needed to prepare your work area first? Once ready, you can finally do some work. Then, after a while, you need to clean up your area. Maybe you use the area for other things and can’t just leave your project sitting around or it would get in the way.

Sometimes, tests can be a lot like that. They might not take up table space, but sometimes they can require an environment setup or some other results to be ready before they can run. Maybe a test makes sure that some data can be deleted. It makes sense that the data should exist first. Should the test be responsible for creating the data that it is trying to delete? It would be better to wrap up the data creation inside its own function. But what if you need to test several different ways to delete the data? Should each test create the data? They could call the same setup function.

If multiple tests need to perform similar preparation and cleanup work, not only is it redundant to write the same code into each test, but it also hides the real purpose of the tests.

This chapter is going to allow tests to run preparation and cleanup code so that they can focus on what needs to be tested. The preparation work is called setup. And the cleanup is called teardown.

We’re following a TDD approach, so that means we’ll start with some simple tests, get them working, and then enhance them for more functionality.

Initially, we’ll let a test run the setup code and then the teardown at the end. Multiple tests can use the same setup and teardown, but the setup and teardown will be run each time for each test.

Once that is working, we’ll enhance the design to let a group of tests share setup and teardown code that runs just once before and after the group of tests.

In this chapter, we will cover the following main topics:

  • Supporting test setup and teardown
  • Enhancing test setup and teardown for multiple tests
  • Handling errors in setup and teardown

By the end of the chapter, the tests will be able to have both individual setup and teardown code along with setup and teardown code that encapsulates groups of tests.

Technical requirements

All the code in this chapter uses standard C++, which builds on any modern C++ 20, or later, compiler and standard library. The code is based on and continues from the previous chapters.

You can find all of the code for this chapter at the following GitHub repository:

https://github.com/PacktPublishing/Test-Driven-Development-with-CPP

Supporting test setup and teardown

In order to support test setup and teardown, we only need to arrange for some code to run before a test begins and for some code to run after a test finishes. For the setup, we might be able to simply call a function near the beginning of the test. The setup doesn’t actually have to run before the test, as long as it runs before the test needs the setup results. What I mean is that the unit test library doesn’t really need to run the setup before a test begins. As long as the test itself runs the setup at the very beginning of the test, then we get the same overall result. This would be the simplest solution. It’s not really a new solution at all, though. A test can already call other functions.

The biggest problem I see with simply declaring a standalone function and calling it at the start of a test is that the intent can get lost. What I mean is that it’s up to the test author to make sure that a function called within a test is clearly defined to be a setup function. Because functions can be named anything, unless it has a good name, just calling a function is not enough to identify the intention to have a setup.

What about the teardown? Can this also be a simple function call? Because the teardown code should always be run at the end of a test, the test author would have to make sure that the teardown runs even when an exception is thrown.

For these reasons, the test library should provide some help with the setup and teardown. How much help and what that help will look like is something we need to decide. Our goal is to keep the tests simple and make sure that all the edge cases are handled.

Following a TDD approach, which was first explained in Chapter 3, The TDD Process, we should do the following:

  • First, think about what the desired solution should be.
  • Write some tests that use the solution to make sure it will meet our expectations.
  • Build the project and fix the build errors without worrying about getting tests to pass yet.
  • Implement a basic solution with passing tests.
  • Enhance the solution and improve the tests.

One option to help with setup and teardown would be to add new parameters to the TEST and TEST_EX macros. This would make the setup and teardown part of the test declaration. But is this necessary? If possible, we should avoid relying on these macros. They’re already complicated enough without adding more features if we can avoid it. Modifying the macros shouldn’t be needed for test setup and teardown.

Another possible solution is to create a method in the TestBase class like we did to set the expected failure reason in Chapter 3, The TDD Process. Would this work? To answer that, let’s think about what the setup and teardown code should do.

The setup should get things ready for the test. This likely means that the test will need to refer to data or resources such as files that the setup code prepares. It might not seem like much of a setup if the test doesn’t get something it can use, but who knows? Maybe the setup does something related but unnoticed by the test code. The main point I’m getting at is that the setup code could do almost anything. It might require its own arguments to customize. Or it might be able to run without any input at all. It might generate something that the test uses directly. Or it might work behind the scenes in a way that is useful but unknown to the test.

Also, the teardown might need to refer to whatever was set up so that it can be undone. Or maybe the teardown just cleans everything up without worrying about where it came from.

Calling a method in TestBase to register and run the setup and teardown seems like it might make the interaction with the test code more complicated because we would need a way to share the setup results. All we really want is to run the setup, gain access to whatever the setup provides, and then run the teardown at the end of the test. There’s an easy way to do this that allows whatever interaction is needed between the setup, teardown, and the rest of the test code.

Let’s start by creating a new .cpp file in the tests folder called Setup.cpp. The project structure will look like this:

MereTDD project root folder
    Test.h
    tests folder
        main.cpp
        Confirm.cpp
        Creation.cpp
        Setup.cpp

Here is a test in Setup.cpp that we can use to get started:

TEST_EX("Test will run setup and teardown code", int)
{
    int id = createTestEntry();
    // If this was a project test, it might be called
    // "Updating empty name throws". And the type thrown
    // would not be an int.
    updateTestEntryName(id, "");
    deleteTestEntry(id);
}

The test uses three functions: createTestEntry, updateTestEntryName, and deleteTestEntry. The comment explains what the test might be called and what it would do if this was a test for an actual project instead of a test for the test library. The idea of the test is to call createTestEntry to set up some data, try to update the name with an empty string to ensure that is not allowed, and then call deleteTestEntry to tear down the data created at the beginning of the test. You can see that the setup provides an identity called id, which is needed by the test and by the teardown.

The test expects the call to updateTestEntryName to fail because of the empty name. This will result in an exception being thrown. We’re just going to throw an int here, but in an actual project, the exception type would normally be something else. The exception will cause the teardown call to deleteTestEntry to be skipped.

Additionally, the test could use confirmations to verify its own results if needed. And a failed confirmation will also throw an exception. We need to make sure that the teardown code is run in all cases. Right now, it will always be skipped because the whole purpose of the test is to expect an exception to be thrown from updateTestEntryName. But other tests might still skip the teardown if they fail a confirmation.

Even if we fix the problem that deleteTestEntry doesn’t get called, we still have a test that’s unclear. What’s really being tested? The only thing in this test that should stand out as the intent of the test is the call to updateTestEntryName. The calls to createTestEntry and deleteTestEntry only hide the real purpose of the test. And if we add a try/catch block to make sure that deleteTestEntry gets called, then the real purpose will only be hidden further.

The three functions in the test are the type of functions that would be found in a project. We don’t have a separate project, so they can be placed in Setup.cpp because they are helper functions for our purposes. They look like this:

#include "../Test.h"
#include <string_view>
int createTestEntry ()
{
    // If this was real code, it might open a
    // connection to a database, insert a row
    // of data, and return the row identifier.
    return 100;
}
void updateTestEntryName (int /*id*/, std::string_view name)
{
    if (name.empty())
    {
        throw 1;
    }
    // Real code would proceed to update the
    // data with the new name.
}
void deleteTestEntry (int /*id*/)
{
    // Real code would use the id to delete
    // the temporary row of data.
}

The id parameter name is commented out because the helper functions don’t use them.

We can wrap up the calls to createTestEntry and deleteTestEntry in the constructor and destructor of a class. This helps simplify the test and ensures that the teardown code gets called. The new test looks like this:

TEST_EX("Test will run setup and teardown code", int)
{
    TempEntry entry;
    // If this was a project test, it might be called
    // "Updating empty name throws". And the type thrown
    // would not be an int.
    updateTestEntryName(entry.id(), "");
}

The TempEntry class contains the setup and teardown calls along with the identifier needed by the test and the teardown. It can go in Setup.cpp right after the three helper methods:

class TempEntry
{
public:
    TempEntry ()
    {
        mId = createTestEntry();
    }
    ~TempEntry ()
    {
        deleteTestEntry(mId);
    }
    int id ()
    {
        return mId;
    }
private:
    int mId;
};

Writing a class like this is a great way to make sure code gets executed when an instance goes out of scope, and we can use it to make sure that the teardown code always gets run at the end of the test. It’s simple and can maintain its own state such as the identifier. Additionally, it only needs a single line to create an instance at the beginning of the test so that it doesn’t distract from what the test is trying to do.

You can go this route anytime you have a specific need that the library code doesn’t meet. But is there a way that the test library can help make this even better?

I’ve seen classes that let you pass lambdas or functions to the constructor that do something similar. The constructor will call the first function right away and will call the second when the instance gets destroyed. That’s just like what the TempEntry class does except for one detail. TempEntry also manages the identity that is needed by the teardown code. None of the lambda solutions I can think of are as clean as a class written just for this purpose, such as TempEntry. But maybe we can still improve this a little more.

The problem with TempEntry is that it’s not clear what is the setup and what is the teardown. It’s also not clear in the test that the first line that creates a TempEntry class has anything to do with setup and teardown. Sure, a little studying will let you realize that the setup is in the constructor and the teardown is in the destructor. It would be nice if we had methods called setup and teardown and if the test itself clearly identified the use of the setup and teardown code being run.

One solution that comes to mind would be a base class that calls virtual setup and teardown methods. But we can’t use normal inheritance because we need to call them from the constructor and destructor. Instead, we can use a design pattern called policy-based design.

A policy class implements one or more methods that a derived class will make use of. The methods that the derived class use are called the policy. It’s like inheritance only backward. We’ll turn the TempEntry class into a policy class that implements the setup and teardown methods by modifying it like this:

class TempEntry
{
public:
    void setup ()
    {
        mId = createTestEntry();
    }
    void teardown ()
    {
        deleteTestEntry(mId);
    }
    int id ()
    {
        return mId;
    }
private:
    int mId;
};

The only real change was to turn the constructor into the setup method and the destructor into the teardown method. That’s only because we were using those methods previously to do the work. Now we have a class that is clear and easy to understand. But how will we use it? We no longer have the setup code running automatically when the class is constructed and the teardown code that runs on destruction. We’ll need to create another class, and this one can go in Test.h because it will be used for all the setup and teardown needs for all the tests. Add this template class inside the MereTDD namespace in Test.h like this:

template <typename T>
class SetupAndTeardown : public T
{
public:
    SetupAndTeardown ()
    {
        T::setup();
    }
    ~SetupAndTeardown ()
    {
        T::teardown();
    }
};

The SetupAndTeardown class is where we tie the calls to setup and teardown back into the constructor and destructor. You can use any class you want for the policy as long as that class implements the two setup and teardown methods. Also, a nice benefit is that because of the public inheritance, you have access to other methods you define in the policy class. We use this to still be able to call the id method. A policy-based design lets you extend the interface to whatever you need as long as you implement the policy. In this example, the policy is just the two setup and teardown methods.

One other thing about using a policy-based design, and specifically about the inheritance, is that this pattern goes against the is-a relationship of object-oriented design. If we were using public inheritance in a normal way, then we could say that SetupAndTeardown is a TempEntry class. In this case, that doesn’t make sense. That’s okay because we’re not going to use this pattern to create instances that can be substituted for one another. We use public inheritance just so that we can call methods such as id inside the policy class.

Now that we have all this, what does the test look like? The test can now use the SetupAndTeardown class like this:

TEST_EX("Test will run setup and teardown code", int)
{
    MereTDD::SetupAndTeardown<TempEntry> entry;
    // If this was a project test, it might be called
    // "Updating empty name throws". And the type thrown
    // would not be an int.
    updateTestEntryName(entry.id(), "");
}

This is a big improvement because of the following list of reasons:

  • It’s clear at the beginning of the test that we have setup code and teardown code attached to the test
  • The teardown code will be run at the end of the test, and we don’t need to complicate the test code with a try/catch block
  • We don’t need to mix calls to setup and teardown into the rest of the test
  • We can interact with the setup results through methods that we write such as the id method

Anytime you need setup and/or teardown code within a test, all you have to do is write a class that implements the setup and teardown methods. If there is no work needed by one of these methods, then leave the method empty. However, both methods need to exist because they are the policy. Implementing the policy methods is what creates a policy class. Then, add an instance of MereTDD:SetupAndTeardown that uses the policy class as its template parameter. The test should declare the SetupAndTeardown instance at the beginning of the test in order to get the most benefit from this design.

While we can declare the setup and teardown code that runs at the beginning and end of each test like this, we’ll need a different solution to share the setup and teardown code so that the setup runs before a group of tests and the teardown code runs after the group of tests complete. The next section will enhance the setup and teardown to meet this expanded need.

Enhancing test setup and teardown for multiple tests

Now that we have the ability to set things up for a test and cleanup after a test, we can do things such as preparing temporary data in the setup that a test needs in order to run and then removing the temporary data in the teardown after a test has run. If there are many different tests using data like this, they can each create similar data.

But what if we need to set up something for a whole group of tests, and then tear it down after all the tests finish? I’m talking about something that remains in place across multiple tests. For the temporary data, maybe we need to prepare a place to hold the data. If the data is stored inside a database, this would be a good time to open the database and make sure the necessary tables are ready to hold the data that each test will be creating. Even the connection to the database itself can remain open and used by the tests. And once all the data tests are done, then the teardown code can close the database.

The scenario applies to many different situations. If you are testing something related to files on a hard drive, then you might want to ensure the proper directories are ready so that the files can be created. The directories can be set up before any of the file tests begin, and the tests only need to worry about creating the files they will be testing.

If you are testing a web service, maybe it makes sense to make sure that your tests have a valid and authenticated login before they begin. It might not make sense for each test to repeat the login steps each time. Unless, of course, that’s the purpose of the test.

The main idea here is that while it’s good to have some code run as setup and teardown for each test, it can also be good to have different setup and teardown code run only once for a group of tests. That’s what this section is going to explore.

We’ll call a group of tests that are related by common setup and teardown code that runs once for the entire group a test suite. Tests don’t have to belong to a test suite, but we’ll create an internal and hidden test suite to group all of the individual tests that don’t have a specific suite of their own.

We were able to add setup and teardown code to an individual test completely within that test because the setup and teardown code within a test is just like calling a couple of functions. However, in order to support setup and teardown for a test suite, we’re going to have to do work outside of the tests. We need to make sure that the setup code runs before any of the related tests run. And then run the teardown code after all of the related tests are complete.

A test project that contains and runs all the tests should be able to support multiple test suites. This means that a test will need some way to identify what test suite it belongs to. Also, we’ll need some way to declare the setup and teardown code for a test suite.

The idea works like this: we’ll declare and write some code to be a test suite setup. Or maybe we can let the setup code be optional if the only thing needed is the teardown code. Then, we’ll declare and write some code for the teardown. The teardown should also be optional. Either the setup or the teardown, or both, need to be defined in order to have a valid test suite. And each test needs some way to identify the test suite it belongs to. When running all the tests in the project, we need to run them in the proper order so that the test suite setup runs first, followed by all the tests in the test suite, and then followed by the test suite teardown.

How will we identify test suites? The test library automatically generates unique names for each test, and these names are hidden from the test author. We could use names for the test suites, too, but let the test author specify what the name should be for each test suite. That seems understandable and should be flexible enough to handle any situation. We’ll let the test author provide a simple string name for each test suite.

When dealing with names, one edge case that always comes up is what to do about duplicate names. We’ll need to decide. We could either detect duplicate names and stop the tests with an error, or we could stack the setup and teardown so that they all run.

Did we have this duplicate problem with the individual test setup and teardown? Not really because the setup and teardown weren’t named. But what happens if a test declares multiple instances of SetupAndTeardown? We actually didn’t consider that possibility in the previous section. In a test, it might look like this:

TEST("Test will run multiple setup and teardown code")
{
    MereTDD::SetupAndTeardown<TempEntry> entry1;
    MereTDD::SetupAndTeardown<TempEntry> entry2;
    // If this was a project test, it might need
    // more than one temporary entry. The TempEntry
    // policy could either create multiple data records
    // or it is easier to just have multiple instances
    // that each create a single data entry.
    updateTestEntryName(entry1.id(), "abc");
    updateTestEntryName(entry2.id(), "def");
}

It is an interesting ability to have multiple setup and teardown instances and should help simplify and let you reuse the setup and teardown code. Instead of creating special setup and teardown policy classes that do many things, this will let them be stacked so that they can be more focused. Maybe one test only needs a single piece of data set up and torn down at the end while another test needs two. Instead of creating two different policy classes, this ability will let the first test declare a single SetupAndTeardown instance, while the second test reuses the same policy class by declaring two.

Now that we allow individual test setup and teardown code to be composed, why not allow the test suite setup and teardown to be composed, too? This seems reasonable and might even simplify the test library code. How is that possible?

Well, now that we know about the ability, we can plan for it and will likely be able to avoid writing code to detect and throw errors instead. If we notice two or more test suite setup definitions with the same name, we can add them to a collection instead of treating the situation as a special error case.

If we do have multiple setup and teardown definitions with the same name, let’s not rely on any particular order between them. They could be defined in different .cpp files just like how the tests can be split between different .cpp files. This will simplify the code because we can add them to a collection as we find them without worrying about a particular order.

The next thing to consider is how to define the test suite setup and teardown code. They probably can’t be simple functions because they need to register themselves with the test library. The registration is needed so that when a test gives a suite name, we will know what the name means. The registration seems very similar to how the tests register themselves. We should be able to add an extra string for the suite name. Additionally, the tests will need this new suite name even if they are not part of a specific test suite. We’ll use an empty suite name for tests that want to run outside of any test suite.

The registration will need to let tests register themselves with a suite name even if the setup and teardown code for that suite name hasn’t yet been registered. This is because tests can be defined in multiple .cpp files, and we have no way of knowing in what order the initialization code will register the tests and test suite setup and teardown code.

There’s one more important requirement. We have a way to interact with the setup results in individual test setup and teardown code. We’ll need this ability in the test suite setup and teardown, too. Let’s say that the test suite setup needs to open a database connection that will be used for all of the tests in the suite. The tests will need some way to know about the connection. Additionally, the test suite teardown will need to know about the connection if it wants to close the connection. Maybe the test suite setup also needs to create a database table. The tests will need the name of that table in order to use it.

Let’s create a couple of helper functions in Setup.cpp that will simulate creating and dropping a table. They should look like this:

#include <string>
#include <string_view>
std::string createTestTable ()
{
    // If this was real code, it might open a
    // connection to a database, create a temp
    // table with a random name, and return the
    // table name.
    return "test_data_01";
}
void dropTestTable (std::string_view /*name*/)
{
    // Real code would use the name to drop
    // the table.
}

Then, in the Setup.cpp file, we can make our first test suite setup and teardown look like this:

class TempTable
{
public:
    void setup ()
    {
        mName = createTestTable();
    }
    void teardown ()
    {
        dropTestTable(mName);
    }
    std::string tableName ()
    {
        return mName;
    }
private:
    std::string mName;
};

This looks a lot like the policy class used in the previous section to define the setup and teardown methods and to provide an interface to access any additional methods or data provided by the setup code. That’s because this is going to be a policy class, too. And we might as well make the policies the same no matter if the setup and teardown code is being used for an individual test or an entire test suite.

When we declare that a test has setup and teardown code for just that test, we declare an instance of MereTDD::SetupAndTeardown that is specialized with a policy class. This is enough to run the setup code right away and to make sure that the teardown code is run at the end of the test. But in order to gain access to the other information, it’s important to give the SetupAndTeardown instance a name. The setup and teardown code is fully defined and accessible through the local named instance.

However, with the test suite setup and teardown, we need to put the instances of the policy class into a container. The container will want everything inside it to be of a single type. The setup and teardown instances can no longer be simple local named variables in a test. Yet, we still need a named type because that’s how a test in the test suite can access the resources provided by the setup code.

We need to figure out two things. The first is where to create instances of test suite setup and teardown code. And the second is how to reconcile the need for a container to have everything be of a single type with the need for the tests to be able to refer to named instances of specific types that can be different from one policy class to another.

The first problem is the easiest to solve because we need to consider lifetime and accessibility. The test suite setup and teardown instances need to exist and be valid for multiple tests within a test suite. They can’t exist as local variables within a single test. They need to be somewhere that they will remain valid for multiple tests. They could be local instances inside of main – that would solve the lifetime issue. But then they would only be accessible to main. The test suite setup and teardown instances will need to be global. Only then will they exist for the duration of the test application and be accessible to multiple tests.

For the second problem, we’re going to first declare an interface that the collection will use to hold all the test suite setup and teardown instances. And the test library will also use this same interface when it needs to run the setup and teardown code. The test library needs to treat everything the same because it doesn’t know anything about the specific policy classes.

We’ll come back to all of this later. Before we get too far, we need to consider what our proposed usage will look like. We are still following a TDD approach, and while it’s good to think of all the requirements and what is possible, we’re far enough along to have a good idea of what a test suite setup and teardown usage would look like. We even have the policy class ready and defined. Add this to Setup.cpp as the intended usage we’re going to implement:

MereTDD::TestSuiteSetupAndTeardown<TempTable>
gTable1("Test suite setup/teardown 1", "Suite 1");
MereTDD::TestSuiteSetupAndTeardown<TempTable>
gTable2("Test suite setup/teardown 2", "Suite 1");
TEST_SUITE("Test part 1 of suite", "Suite 1")
{
    // If this was a project test, it could use
    // the table names from gTable1 and gTable2.
    CONFIRM("test_data_01", gTable1.tableName());
    CONFIRM("test_data_01", gTable2.tableName());
}
TEST_SUITE_EX("Test part 2 of suite", "Suite 1", int)
{
    // If this was a project test, it could use
    // the table names from gTable1 and gTable2.
    throw 1;
}

There are a few things to explain with the preceding code. You can see that it declares two instances of MereTDD::TestSuiteSetupAndTeardown, each specialized with the TempTable policy class. These are global variables with a specific type, so the tests will be able to see them and use the methods in the policy class for each. You can use a different policy class for each if you want. Or if you use the same policy class, then there should normally be some difference. Otherwise, why have two instances? For creating temp tables, as this example shows, each table would likely have a unique random name and be able to use the same policy class.

The constructor needs two strings. The first is the name of the setup and teardown code. We’re going to treat test suite setup and teardown code as if it was a test itself. We’ll include the test suite setup and teardown pass or fail results in the test application summary and identify it with the name provided to the constructor. The second string is the name of the test suite. This can be anything except for an empty string. We’ll treat an empty test suite name as a special value for all the tests that do not belong to a test suite.

In this example, both instances of TestSuiteSetupAndTeardown use the same suite name. This is okay and supported, as we decided earlier. Anytime there are multiple test suite setup and teardown instances with the same name, they will all be run before the test suite begins.

The reason for a new TestSuiteSetupAndTeardown test library class instead of reusing the existing SetupAndTeardown class will become clear later. It needs to merge a common interface together with the policy class. The new class also makes it clear that this setup and teardown are for a test suite.

Then come the tests. We need a new macro called TEST_SUITE so that the name of the test suite can be specified. Other than the test suite name, the macro will behave almost the same as the existing TEST macro. We’ll need a new macro for tests belonging to a test suite that also expects an exception. We’ll call that one TEST_SUITE_EX; it behaves similarly to TEST_EX except for the additional test suite name.

There are a lot of changes needed in Test.h to support test suites. Most of the changes are related to how the tests are registered and run. We have a base class for tests called TestBase, which does the registration by pushing a pointer to TestBase to a vector. Because we also need to register test suite setup and teardown code and run tests grouped by their test suite, we’ll need to change this. We’ll keep TestBase as the base class for all tests. But it will now be a base class for test suites, too.

The collection of tests will need to change to a map so that the tests can be accessed by their test suite name. Tests without a test suite will still have a suite name. It will just be empty. Additionally, we need to find the test suite setup and teardown code by the suite name, too. We’ll need two collections: one map for the tests and one map for the test suite setup and teardown code. And because we need to refactor the existing registration code out of TestBase, we’ll create a class called Test that will be used for the tests and a class called TestSuite that will be used for the test suite setup and teardown code. Both the Test and TestSuite classes will derive from TestBase.

The maps will be accessed with the existing getTests function that will be modified to use a map and a new getTestSuites function. First, include a map at the top of Test.h:

#include <map>
#include <ostream>
#include <string_view>
#include <vector>

Then, further down, change the part that forward-declares the TestBase class and implements the getTests function to look like this:

class Test;
class TestSuite;
inline std::map<std::string, std::vector<Test *>> & getTests ()
{
    static std::map<std::string, std::vector<Test *>> tests;
    return tests;
}
inline std::map<std::string, std::vector<TestSuite *>> & getTestSuites ()
{
    static std::map<std::string,            std::vector<TestSuite *>> suites;
    return suites;
}

The key to each map will be the test suite name as a string. The value will be a vector of either pointers to Test or pointers to TestSuite. When we register a test or test suite setup and teardown code, we will do so by the test suite name. The first registration for any test suite name will need to set up an empty vector. Once the vector has been set up, the test can be pushed to the end of the vector just like it was done previously. And the test suite setup and teardown code will do the same thing. To make this process easier, we’ll create a couple of helper methods that can go in Test.h right after the getTestSuites function:

inline void addTest (std::string_view suiteName, Test * test)
{
    std::string name(suiteName);
    if (not getTests().contains(name))
    {
        getTests().try_emplace(name, std::vector<Test *>());
    }
    getTests()[name].push_back(test);
}
inline void addTestSuite (std::string_view suiteName, TestSuite * suite)
{
    std::string name(suiteName);
    if (not getTestSuites().contains(name))
    {
        getTestSuites().try_emplace(name,            std::vector<TestSuite *>());
    }
    getTestSuites()[name].push_back(suite);
}

Next is the refactored TestBase class, which has been modified to add a test suite name, stop doing the test registration, remove the expected failure reason, and remove the running code. Now the TestBase class will only hold data that is common between the tests and the test suite setup and teardown code. The class looks like this after the changes:

class TestBase
{
public:
    TestBase (std::string_view name, std::string_view suiteName)
    : mName(name),
      mSuiteName(suiteName),
      mPassed(true),
      mConfirmLocation(-1)
    { }
    virtual ~TestBase () = default;
    std::string_view name () const
    {
        return mName;
    }
    std::string_view suiteName () const
    {
        return mSuiteName;
    }
    bool passed () const
    {
        return mPassed;
    }
    std::string_view reason () const
    {
        return mReason;
    }
    int confirmLocation () const
    {
        return mConfirmLocation;
    }
    void setFailed (std::string_view reason,          int confirmLocation = -1)
    {
        mPassed = false;
        mReason = reason;
        mConfirmLocation = confirmLocation;
    }
private:
    std::string mName;
    std::string mSuiteName;
    bool mPassed;
    std::string mReason;
    int mConfirmLocation;
};

The functionality that was pulled out of the previous TestBase class goes into a new derived class, called Test, which looks like this:

class Test : public TestBase
{
public:
    Test (std::string_view name, std::string_view suiteName)
    : TestBase(name, suiteName)
    {
        addTest(suiteName, this);
    }
    virtual void runEx ()
    {
        run();
    }
    virtual void run () = 0;
    std::string_view expectedReason () const
    {
        return mExpectedReason;
    }
    void setExpectedFailureReason (std::string_view reason)
    {
        mExpectedReason = reason;
    }
private:
    std::string mExpectedReason;
};

The Test class is shorter because a lot of the basic information now lives in the TestBase class. Also, we used to have a TestExBase class, which needs to be changed slightly. It will now be called TestEx and looks like this:

template <typename ExceptionT>
class TestEx : public Test
{
public:
    TestEx (std::string_view name,
        std::string_view suiteName,
        std::string_view exceptionName)
    : Test(name, suiteName), mExceptionName(exceptionName)
    { }
    void runEx () override
    {
        try
        {
            run();
        }
        catch (ExceptionT const &)
        {
            return;
        }
        throw MissingException(mExceptionName);
    }
private:
    std::string mExceptionName;
};

The only thing that has really changed with TestEx is the name and the base class name.

Now, we can get to the new TestSuite class. This will be the common interface that will be stored in the map and serve as the common interface for running the setup and teardown code by the test library.

The class looks like this:

class TestSuite : public TestBase
{
public:
    TestSuite (
        std::string_view name,
        std::string_view suiteName)
    : TestBase(name, suiteName)
    {
        addTestSuite(suiteName, this);
    }
    virtual void suiteSetup () = 0;
    virtual void suiteTeardown () = 0;
};

The TestSuite class doesn’t have a runEx method like the Test class. Test suites exist to group tests and to prepare an environment for a test to use, so it doesn’t make sense to write setup code that is expected to throw an exception. A test suite doesn’t exist to test anything. It exists to prepare for one or more tests that will use the resources that the suiteSetup method gets ready. Also, the teardown code is not intended to test anything. The suiteTeardown code is just supposed to clean up whatever was set up. If there are any exceptions during test suite setup and teardown, we want to know about them.

Additionally, the TestSuite class does not have a run method like the Test class because we need to clearly define what the setup is versus what the teardown is. There is no single block of code to run. There are now two separate blocks of code to run, one at setup time and one at teardown time. So, while a Test class is designed to run something, a TestSuite class is designed to prepare a group of tests with a setup and then clean up after the tests with a teardown.

You can see that the TestSuite constructor registers the test suite setup and teardown code by calling addTestSuite.

We have a function called runTests that currently goes through all the tests and runs them. If we put the code to actually run a single test inside a new function, we can simplify the code that goes through all of the tests and then displays the summary. This will be important because we’ll need to run more than tests in the new design. We’ll also need to run the test suite setup and teardown code.

Here is a helper function to run a single test:

inline void runTest (std::ostream & output, Test * test,
    int & numPassed, int & numFailed, int & numMissedFailed)
{
    output << "------- Test: "
        << test->name()
        << std::endl;
    try
    {
        test->runEx();
    }
    catch (ConfirmException const & ex)
    {
        test->setFailed(ex.reason(), ex.line());
    }
    catch (MissingException const & ex)
    {
        std::string message = "Expected exception type ";
        message += ex.exType();
        message += " was not thrown.";
        test->setFailed(message);
    }
    catch (...)
    {
        test->setFailed("Unexpected exception thrown.");
    }
    if (test->passed())
    {
        if (not test->expectedReason().empty())
        {
            // This test passed but it was supposed
            // to have failed.
            ++numMissedFailed;
            output << "Missed expected failure
"
                << "Test passed but was expected to fail."
                << std::endl;
        }
        else
        {
            ++numPassed;
            output << "Passed"
                << std::endl;
        }
    }
    else if (not test->expectedReason().empty() &&
        test->expectedReason() == test->reason())
    {
        ++numPassed;
        output << "Expected failure
"
            << test->reason()
            << std::endl;
    }
    else
    {
        ++numFailed;
        if (test->confirmLocation() != -1)
        {
            output << "Failed confirm on line "
                << test->confirmLocation() << "
";
        }
        else
        {
            output << "Failed
";
        }
        output << test->reason()
            << std::endl;
    }
}

The preceding code is almost identical to what was in runTests. Slight changes have been made to the beginning output that displays the test name. This is to help distinguish a test from the setup and teardown code. The helper function also takes references to the record-keeping counts.

We can create another helper function to run the setup and teardown code. This function will perform almost the same steps for setup and teardown. The main difference is which method to call on the TestSuite pointer, either suiteSetup or suiteTeardown. The helper function looks like this:

inline bool runSuite (std::ostream & output,
    bool setup, std::string const & name,
    int & numPassed, int & numFailed)
{
    for (auto & suite: getTestSuites()[name])
    {
        if (setup)
        {
            output << "------- Setup: ";
        }
        else
        {
            output << "------- Teardown: ";
        }
        output << suite->name()
            << std::endl;
        try
        {
            if (setup)
            {
                suite->suiteSetup();
            }
            else
            {
                suite->suiteTeardown();
            }
        }
        catch (ConfirmException const & ex)
        {
            suite->setFailed(ex.reason(), ex.line());
        }
        catch (...)
        {
            suite->setFailed("Unexpected exception thrown.");
        }
        if (suite->passed())
        {
            ++numPassed;
            output << "Passed"
                << std::endl;
        }
        else
        {
            ++numFailed;
            if (suite->confirmLocation() != -1)
            {
                output << "Failed confirm on line "
                    << suite->confirmLocation() << "
";
            }
            else
            {
                output << "Failed
";
            }
            output << suite->reason()
                << std::endl;
            return false;
        }
    }
    return true;
}

This function is slightly simpler than the helper function to run a test. That’s because we don’t need to worry about missed exceptions or expected failures. It does almost the same thing. It tries to run either the setup or teardown, catch exceptions, and update the pass or fail counts.

We can use the two helper functions, runTest and runSuite, inside the runTests function, which will need to be modified like this:

inline int runTests (std::ostream & output)
{
    output << "Running "
        << getTests().size()
        << " test suites
";
    int numPassed = 0;
    int numMissedFailed = 0;
    int numFailed = 0;
    for (auto const & [key, value]: getTests())
    {
        std::string suiteDisplayName = "Suite: ";
        if (key.empty())
        {
            suiteDisplayName += "Single Tests";
        }
        else
        {
            suiteDisplayName += key;
        }
        output << "--------------- "
            << suiteDisplayName
            << std::endl;
        if (not key.empty())
        {
            if (not getTestSuites().contains(key))
            {
                output << "Test suite is not found."
                    << " Exiting test application."
                    << std::endl;
                return ++numFailed;
            }
            if (not runSuite(output, true, key,
                numPassed, numFailed))
            {
                output << "Test suite setup failed."
                    << " Skipping tests in suite."
                    << std::endl;
                continue;
            }
        }
        for (auto * test: value)
        {
            runTest(output, test,
                numPassed, numFailed, numMissedFailed);
        }
        if (not key.empty())
        {
            if (not runSuite(output, false, key,
                numPassed, numFailed))
            {
                output << "Test suite teardown failed."
                    << std::endl;
            }
        }
    }
    output << "-----------------------------------
";
    output << "Tests passed: " << numPassed
        << "
Tests failed: " << numFailed;
    if (numMissedFailed != 0)
    {
        output << "
Tests failures missed: "                << numMissedFailed;
    }
    output << std::endl;
    return numFailed;
}

The initial statement that gets displayed shows how many test suites are being run. Why does the code look at the size of the tests instead of the size of the test suites? Well, that’s because the tests include everything, tests with a test suite and those tests without a test suite that get run under a made-up suite called Single Tests.

The primary loop in this function looks at every item in the test map. Previously, these were the pointers to the tests. Now each entry is a test suite name and a vector of test pointers. This lets us go through the tests that are already grouped by the test suite that each test belongs to. An empty test suite name represents the single tests that have no test suite.

If we find a test suite that is not empty, then we need to make sure that there exists at least one entry in the test suite with a matching suite name. If not, then this is an error in the test project and no further tests will be run.

If the test project registers a test with a suite name, then it must also register the setup and teardown code for that suite. Assuming we have the setup and teardown code for the suite, each registered setup is run and checked for an error. If there is an error setting up the test suite, then only the tests in that suite will be skipped.

Once all the test suite setup code is run, then the tests are run for that suite.

And after all the tests for the suite are run, then all the test suite teardown code is run.

There are two more parts to enable all this. The first is the TestSuiteSetupAndTeardown class, which goes into Test.h right after the existing SetupAndTeardown class. It looks like this:

template <typename T>
class TestSuiteSetupAndTeardown :
    public T,
    public TestSuite
{
public:
    TestSuiteSetupAndTeardown (
        std::string_view name,
        std::string_view suite)
    : TestSuite(name, suite)
    { }
    void suiteSetup () override
    {
        T::setup();
    }
    void suiteTeardown () override
    {
        T::teardown();
    }
};

This is the class that is used in a test .cpp file to declare a test suite setup and teardown instance with a specific policy class. This class uses multiple inheritances to bridge the policy class and the common TestSuite interface class. When the runSuite function calls either suiteSetup or suiteTeardown through a pointer to TestSuite, these virtual methods will end up calling the override methods in this class. Each one just calls the setup or teardown methods in the policy class to do the actual work.

The last change to explain is the macros. We need two additional macros to declare a test that belongs to a test suite without an expected exception and with an expected exception. The macros are called TEST_SUITE and TEST_SUITE_EX. There are minor changes needed in the existing TEST and TEST_EX macros because of the refactoring of the TestBase class. The existing macros need to be updated to use the new Test and TestEx classes instead of TestBase and TestExBase. Additionally, the existing macros need to now pass an empty string for the test suite name. I’ll show the new macros here because they are so similar except for the difference in the test suite name. The TEST_SUITE macro looks like this:

#define TEST_SUITE( testName, suiteName ) 
namespace { 
class MERETDD_CLASS : public MereTDD::Test 
{ 
public: 
    MERETDD_CLASS (std::string_view name, 
      std::string_view suite) 
    : Test(name, suite) 
    { } 
    void run () override; 
}; 
} /* end of unnamed namespace */ 
MERETDD_CLASS MERETDD_INSTANCE(testName, suiteName); 
void MERETDD_CLASS::run ()

The macro now accepts a suiteName parameter that gets passed to the instance as the suite name. And the TEST_SUITE_EX macro looks like this:

#define TEST_SUITE_EX( testName, suiteName, exceptionType ) 
namespace { 
class MERETDD_CLASS : public MereTDD::TestEx<exceptionType> 
{ 
public: 
    MERETDD_CLASS (std::string_view name, 
        std::string_view suite, 
        std::string_view exceptionName) 
    : TestEx(name, suite, exceptionName) 
    { } 
    void run () override; 
}; 
} /* end of unnamed namespace */ 
MERETDD_CLASS MERETDD_INSTANCE(testName, suiteName, #exceptionType); 
void MERETDD_CLASS::run ()

The new suite macros are so similar to the modified non-suite macros that I tried to change the non-suite macros to call the suite macros with an empty suite name. But I could not figure out how to pass an empty string to another macro. The macros are short, so I left them with similar code.

That’s all of the changes needed to enable the test suites. The summary output looks a little different now after these changes. Building and running the test project produces the following output. It’s a bit long because we have 30 tests now. So, I’m not showing the entire output. The first part looks like this:

Running 2 test suites
--------------- Suite: Single Tests
------- Test: Test will run setup and teardown code
Passed
------- Test: Test will run multiple setup and teardown code
Passed
------- Test: Test can be created
Passed
------- Test: Test that throws unexpectedly can be created
Expected failure
Unexpected exception thrown.

Here, you can see that there are two test suites. One is named Suite 1 and contains two tests plus the suite setup and teardown, and the other is unnamed and contains all the other tests that do not belong to a test suite. The first part of the output happens to be the single tests. The rest of the summary output shows the test suite and looks like this:

--------------- Suite: Suite 1
------- Setup: Test suite setup/teardown 1
Passed
------- Setup: Test suite setup/teardown 2
Passed
------- Test: Test part 1 of suite
Passed
------- Test: Test part 2 of suite
Passed
------- Teardown: Test suite setup/teardown 1
Passed
------- Teardown: Test suite setup/teardown 2
Passed
-----------------------------------
Tests passed: 30
Tests failed: 0
Tests failures missed: 1

Each test suite is introduced in the summary output with the suite name followed by all the tests in that suite. For an actual suite, you can see the setup and teardown that surround all the tests. Each setup and teardown is run as if it was a test.

And the end shows the pass and fail counts just like before.

In this section, I briefly explained some of the error handling for the setup and teardown code, but it needs more. The main purpose of this section was to get the setup and teardown working for the test suites. Part of that required a bit of error handling such as what to do when a test declares that it belongs to a suite that doesn’t exist. The next section will go deeper into this.

Handling errors in setup and teardown

Bugs can be found anywhere in code, and that includes inside the setup and teardown code. So, how should these bugs be handled? In this section, you’ll see that there is no single way to deal with bugs in setup and teardown. It’s more important that you be aware of the consequences so that you can write better tests.

Let’s start at the beginning. We’ve already sidestepped a whole class of problems related to multiple setup and teardown declarations. We decided that these would simply be allowed instead of trying to prevent them. So, a test can have as many setup and teardown declarations as it wants. Additionally, a test suite can declare as many instances of setup and teardown as needed.

However, just because multiple instances are allowed doesn’t mean that there won’t be any problems. The code that creates the test data entries is a good example. I thought about fixing the problem in the code but left it so that I can explain the problem here:

int createTestEntry ()
{
    // If this was real code, it might open a
    // connection to a database, insert a row
    // of data, and return the row identifier.
    return 100;
}

The problem is hinted at in the preceding comment. It says that real code would return the row identifier. Since this is a test helper function that has no connection to an actual database, it simply returns a constant value of 100.

You’ll want to avoid your setup code doing anything that can conflict with additional setup code. A row identity in a database will not conflict because the database will return a different ID each time data is inserted. But what about other fields that get populated in the data? You might have constraints in your table, for example, where a name must be unique. If you create a fixed test name in one setup, then you won’t be able to use the same name in another.

Even if you have different fixed names in different setup blocks so they won’t cause conflicts, you can still run into problems if the test data doesn’t get cleaned up properly. You might find that your tests run okay the first time and then fail thereafter because the fixed names already exist in the database.

I recommend that you randomize your test data. Here is the other example that creates a test table:

std::string createTestTable ()
{
    // If this was real code, it might open a
    // connection to a database, create a temp
    // table with a random name, and return the
    // table name.
    return "test_data_01";
}

The comment in the preceding code also mentions creating a random name. It’s fine to use a fixed prefix, but consider making the digits at the end random instead of fixed. This won’t completely solve the problem of colliding data. It’s always possible that random numbers will turn out to be the same. But doing this together with a good cleanup of the test data should help eliminate most cases of conflicting setups.

Another problem has already been handled in the test library code. And that is what to do when a test declares that it belongs to a test suite and that test suite does not have any setup and teardown code defined.

This is treated as a fatal error in the test application itself. The moment a required test suite setup and teardown registration cannot be found, the test application exits and does not run any more tests.

The fix is simple. Make sure you always define test suite setup and teardown code for all test suites that the tests use. It’s okay to have a test suite setup and teardown code registered that is never used by any test. But the moment a test declares that it belongs to a test suite, then that suite becomes required.

Now, let’s talk about exceptions in setup and teardown code. This includes confirmations because a failed CONFIRM macro results in an exception being thrown. It’s okay to add confirmations to set up code like this:

class TempEntry
{
public:
    void setup ()
    {
        mId = createTestEntry();
        CONFIRM(10, mId);
    }

Currently, this will cause the setup to fail because the identity is fixed to always be a value of 100. And the confirmation tries to make sure that the value is 10. Because the test setup code is called as if it was a regular function call, the result of this failed confirmation will be the same as any other failed confirmation in the test itself. The test will fail, and the summary will show where and why the failure happened. The summary looks like this:

------- Test: Test will run multiple setup and teardown code
Failed confirm on line 51
    Expected: 10
    Actual  : 100

However, putting confirmations in the teardown code is not recommended. And throwing exceptions from the teardown code is not recommended – especially for test teardown code because test teardown code is run from inside a destructor. So, moving the confirmation to the teardown code like this will not work the same way:

class TempEntry
{
public:
    void setup ()
    {
        mId = createTestEntry();
    }
    void teardown ()
    {
        deleteTestEntry(mId);
        CONFIRM(10, mId);
    }

This will result in an exception being thrown during the destruction of the SetupAndTeardown class that uses the TempEntry policy class. The entire test application will be terminated like this:

Running 2 test suites
--------------- Suite: Single Tests
------- Test: Test will run setup and teardown code
terminate called after throwing an instance of 'MereTDD::ActualConfirmException'
/tmp/codelite-exec.sh: line 3: 38155 Abort trap: 6           ${command}

The problem is not as severe in the test suite teardown code because that teardown code is run by the test library after all the tests in the suite have been completed. It is not run as part of a class destructor. It’s still good advice to follow about not throwing any exceptions in the teardown code at all.

Treat your teardown code as an opportunity to clean up the mess left behind by the setup and the tests. Normally, it should not contain anything that needs to be tested.

The test suite setup code is a little different from the test setup code. While an exception in the test setup code causes the test to stop running and fail, an exception thrown in a test suite setup will cause all the tests in that suite to be skipped. Adding this confirmation to the test suite setup will trigger an exception:

class TempTable
{
public:
    void setup ()
    {
        mName = createTestTable();
        CONFIRM("test_data_02", mName);
    }

And the output summary shows that the entire test suite has been disrupted like this:

--------------- Suite: Suite 1
------- Setup: Test suite setup/teardown 1
Failed confirm on line 73
    Expected: test_data_02
    Actual  : test_data_01
Test suite setup failed. Skipping tests in suite.

The preceding message says that the test suite will be skipped.

All the error handling that went into the test library for both test setup and teardown and test suite setup and teardown is largely untested itself. What I mean is that we added an extra feature to the test library to support any expected failures. And I did not do the same thing for expected failures in the setup and teardown code. I felt that the extra complexity needed to handle expected failures in the setup and teardown code was not worth the benefit.

We’re using TDD to help guide the design of the software and to improve the quality of the software. But TDD doesn’t completely remove the need for some manual testing of edge conditions that are too difficult to test in an automated manner or that are just not feasible to test.

So, will there be a test to make sure that the test library really does terminate when a required test suite is not registered? No. That seems like the kind of test that is best handled through manual testing. There might be situations you’ll encounter that are similar, and you’ll have to decide how much effort is needed to write a test and whether the effort is worth the cost.

Summary

This chapter completes the minimum functionality needed in a unit testing library. We’re not done developing the testing library, but it now has enough features to be useful to another project.

You learned about the issues involved with adding setup and teardown code and the benefits provided. The primary benefit is that tests can now focus on what is important to be tested. Tests are easier to understand when everything needed to run the test is no longer cluttering the test and the cleanup is handled automatically.

There are two types of setup and teardown. One is local to the test; it can be reused in other tests but local means that the setup runs at the beginning of the test and the teardown happens at the end of the test. Another test that shares the same setup and teardown will repeat the setup and teardown in that other test.

The other type of setup and teardown is actually shared by multiple tests. This is the test suite setup and teardown; its setup runs before any of the tests in the suite begin, and its teardown runs after all the tests in the suite are complete.

For the local tests, we were able to integrate them into the tests fairly easily without much impact on the test library. We used a policy-based design to make writing the setup and teardown code easy. And the design lets the test code access the resources prepared in the setup.

The test suite setup and teardown was more elaborate and needed extensive support from the test library. We had to change the way tests were registered and run. But at the same time, we simplified the code and made it better. The test suite setup and teardown design uses the same policy that the local setup and teardown uses, which makes the whole design consistent.

And you also learned a few tips about how to handle errors in the setup and teardown code.

The next chapter will continue to give you guidance and tips on how to write better tests.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
13.58.82.79