2

Test Results

So far, we have a test library that can only have a single test. You’ll see what happens in this chapter when we try to add another test and you’ll see how to enhance the test library to support multiple tests. We’ll need to use an old and rarely used capability of C++ that actually comes from its early C roots to support multiple tests.

Once we get more than one test, we’ll need a way to view the results. This will let you tell at a glance whether everything passed or not. And finally, we’ll fix the result output so that it no longer assumes std::cout.

We’ll cover the following main topics in this chapter:

  • Reporting a single test result based on exceptions
  • Enhancing the test library to support multiple tests
  • Summarizing the test results to clearly see what failed and what passed
  • Redirecting the test result so the output can go to any stream

Technical requirements

All code in this chapter uses standard C++ that builds on any modern C++ 17 or later compiler and standard library. The code is based on and continues from the previous chapter.

You can find all the code for this chapter at the following GitHub repository: https://github.com/PacktPublishing/Test-Driven-Development-with-CPP.

Reporting a single test result

So far, our single test just prints its hardcoded name when run. There was some early thinking that we might need a result in addition to the test name. This is actually a good example of adding something to the code that is not needed or used. Okay, a minor example because we will need something to keep track of whether the test passes or fails, but it’s still a good example of getting ahead of ourselves because we have actually never used the mResult data member yet. We’re going to fix that now with a better way to track the result of running a test.

We’ll assume that a test succeeds unless something happens to cause it to fail. What can happen? There will eventually be a lot of ways you can cause a test to fail. For now, we’ll just consider exceptions. This could be an exception that a test throws on purpose when it detects something is wrong or it could be an unexpected exception that gets thrown.

We don’t want any exceptions to stop the tests from running. An exception thrown from one test shouldn’t be a reason to stop running others. We still only have a single test but we can make sure that an exception doesn’t stop the entire test process.

What we want is to wrap the run function call in a try block so that any exceptions will be treated as a failure, like this:

inline void runTests ()
{
    for (auto * test: getTests())
    {
        try
        {
            test->run();
        }
        catch (...)
        {
            test->setFailed("Unexpected exception thrown.");
        }
    }
}

When an exception is caught, we want to do two things. The first is to mark the test as a failure. The second is to set a message so that the result can be reported. The problem is that we don’t have a method called setFailed on the TestInterface class. It’s actually good to first write the code as we’d like it to be.

In fact, the idea of TestInterface was for it to be a set of pure virtual methods like an interface. We could add a new method called setFailed but then the implementation would need to be written in a derived class. This seems like a basic part of a test to be able to hold the result and a message.

So, let’s refactor the design and change TestInterface into more of a base class and call it TestBase instead. We can also move the data members from the class declared inside the TEST macro and put them in the TestBase class:

class TestBase
{
public:
    TestBase (std::string_view name)
    : mName(name), mPassed(true)
    { }
    virtual ~TestBase () = default;
    virtual void run () = 0;
    std::string_view name () const
    {
        return mName;
    }
    bool passed () const
    {
        return mPassed;
    }
    std::string_view reason () const
    {
        return mReason;
    }
    void setFailed (std::string_view reason)
    {
        mPassed = false;
        mReason = reason;
    }
private:
    std::string mName;
    bool mPassed;
    std::string mReason;
};

With the new setFailed method, it no longer made sense to have an mResult data member. Instead, there’s an mPassed member, along with the mName member; both came from the TEST macro. It also seemed like a good idea to add some getter methods, especially now that there’s also an mReason data member. Altogether, each test can now store its name, remember whether it passed or not, and the reason for failure, if it failed.

Only a slight change is needed in the getTests function to refer to the TestBase class:

inline std::vector<TestBase *> & getTests ()
{
    static std::vector<TestBase *> tests;
    return tests;
}

The rest of the changes simplify the TEST macro like this to remove the data members, which are now in the base class, and to inherit from TestBase:

#define TEST 
class Test : public MereTDD::TestBase 
{ 
public: 
    Test (std::string_view name) 
    : TestBase(name) 
    { 
        MereTDD::getTests().push_back(this); 
    } 
    void run () override; 
}; 
Test test("testCanBeCreated"); 
void Test::run ()

Checking to make sure everything builds and runs again shows that we are back to a running program with the same result as before. You’ll see this technique often with a refactor. It’s good to keep any functional changes to a minimum when refactoring and focus mostly on just getting back to the same behavior as before.

Now, we can make some changes that will affect observable behavior. We want to report what is happening while the test is running. For now, we’ll just send the output to std::cout. We’ll change this later in this chapter to avoid assuming the output destination. The first change is to include iostream in Test.h:

#define MERETDD_TEST_H
#include <iostream>
#include <string_view>
#include <vector>

Then, change the runTests function to report the progress of the test being run, like this:

inline void runTests ()
{
    for (auto * test: getTests())
    {
        std::cout << "---------------
"
            << test->name()
            << std::endl;
        try
        {
            test->run();
        }
        catch (...)
        {
            test->setFailed("Unexpected exception thrown.");
        }
        if (test->passed())
        {
            std::cout << "Passed"
                << std::endl;
        }
        else
        {
            std::cout << "Failed
"
                << test->reason()
                << std::endl;
        }
    }
}

The original try/catch remains unchanged. All we do is print some dashes for a separator and the name of the test. It’s probably a good idea to flush this line to the output right away. In the case that something happens later, at least the name of the test will be recorded. After the test is run, the test is checked to see whether it passed or not, and the appropriate message is displayed.

We’ll also change the test in Creation.cpp to throw something to make sure we get a failure. We no longer need to include iostream because it’s usually not a good idea to display anything from the test itself. You can display output from the test if you want to but any output in the test itself tends to mess up the reporting of the test results. When I sometimes need to display output from within a test, it’s usually temporary.

Here is the test modified to throw an int:

#include "../Test.h"
TEST
{
    throw 1;
}

Normally, you would write code that throws something other than a simple int value, but at this point, we just want to show what happens when something does get thrown.

Building and running it now shows the expected failure due to an unexpected exception:

---------------

testCanBeCreated

Failed

Unexpected exception thrown.

Program ended with exit code: 0

We can remove the throw statement from the test so that the body is completely empty and the test will now pass:

---------------

testCanBeCreated

Passed

Program ended with exit code: 0

We don’t want to keep modifying the test for different scenarios. It’s time to add support for multiple tests.

Enhancing the test declaration to support multiple tests

While a single test works, trying to add another one does not build. This is what I tried to do in Creation.cpp by adding another test. One of the tests is empty and the second test throws an int. These are the two scenarios we were just trying to work with:

#include "../Test.h"
TEST
{
}
TEST
{
    throw 1;
}

The failure is due to the Test class being declared twice, as well as the run method. The TEST macro declares a new global instance of the Test class each time it’s used. Each instance is called test. We don’t see the classes or the instances in the code because they are hidden by the TEST macro.

We’ll need to modify the TEST macro so that it will generate unique class and instance names. And while we’re doing that, let’s also fix the name of the test itself. We don’t want all tests to have the name "testCanBeCreated", and since the name will need to come from the test declaration, we’ll need to also modify the TEST macro to accept a string. Here is how the new Creation.cpp file should look:

#include "../Test.h"
TEST("Test can be created")
{
}
TEST("Test with throw can be created")
{
    throw 1;
}

This lets us give sentence names to each test, instead of treating the name like a single-word function name. We still need to modify the TEST macro but it’s good to start with the intended usage first and then make it work.

For making unique class and instance names, we could just ask for something unique from the programmer, but the type name of the class and the instance name of that class really are details that the programmer writing tests shouldn’t need to worry about. Requiring a unique name to be supplied would only make the details visible. We could instead use a base name and add to it the line number where the test is declared to make both the class and instance names unique.

Macros have the ability to get the line number of the source code file where the macro is used. All we have to do is modify the resulting class and instance names by appending this line number.

It would be nice if this was easy.

All the macros are handled by the preprocessor. It’s actually a bit more complicated than that but thinking in terms of the preprocessor is a good simplification. The preprocessor knows how to do simple text replacement and manipulation. The compiler never sees the original code that is written with the macro. The compiler instead sees the end result after the preprocessor is done.

We will need two sets of macros declared in Test.h. One set will generate a unique class name, such as Test7 if the TEST macro was used on line 7. The other set of macros will generate a unique instance name, such as test7.

We need a set of macros because going from a line number to a concatenated result such as Test7 requires multiple steps. If this is the first time you’ve seen macros used like this, it’s normal to find them confusing. Macros use simple text replacement rules that can seem like extra work for us at first. Going from a line number to a unique name requires multiple steps of text replacement that are not obvious. The macros look like this:

#define MERETDD_CLASS_FINAL( line ) Test ## line
#define MERETDD_CLASS_RELAY( line ) MERETDD_CLASS_FINAL( line )
#define MERETDD_CLASS MERETDD_CLASS_RELAY( __LINE__ )
#define MERETDD_INSTANCE_FINAL( line ) test ## line
#define MERETDD_INSTANCE_RELAY( line ) MERETDD_INSTANCE_FINAL( line )
#define MERETDD_INSTANCE MERETDD_INSTANCE_RELAY( __LINE__ )

Each set needs three macros. The macro to use is the last in each set, MERETDD_CLASS and MERETDD_INSTANCE. Each of these will be replaced with the relay macro using the __LINE__ value. The relay macro will see the real line number instead of __LINE__ and the relay macro will then be replaced with the final macro and the line number it was given. The final macro will use the ## operator to do the concatenation. I did warn you that it would be nice if this was easy. I’m sure this is one of the reasons so many programmers avoid macros. At least you’ve already made it through the most difficult usage of macros in this book.

The end result will be, for example, Test7 for the class name and test7 for the instance name. The only real difference between these two sets of macros is that the class name uses a capital T for Test and the instance name uses a lowercase t for test.

The class and instance macros need to be added to Test.h right above the definition of the TEST macro that will need to use them. All of this works because, even though the TEST macro looks like it uses many source code lines, remember that each line is terminated with a backslash. This causes everything to end up on a single line of code. This way, all the line numbers will be the same each time the TEST macro is used and the line number will be different the next time it’s used.

The new TEST macro looks like this:

#define TEST( testName ) 
class MERETDD_CLASS : public MereTDD::TestBase 
{ 
public: 
    MERETDD_CLASS (std::string_view name) 
    : TestBase(name) 
    { 
        MereTDD::getTests().push_back(this); 
    } 
    void run () override; 
}; 
MERETDD_CLASS MERETDD_INSTANCE(testName); 
void MERETDD_CLASS::run ()

The MERETDD_CLASS macro is used to declare the class name, declare the constructor, declare the type of the global instance, and scope the run method declaration to the class. All four of these macros will use the same line number because of the backslashes at the end of each line.

The MERETDD_INSTANCE macro is used just once to declare the name of the global instance. It will also use the same line number as the class name.

Building the project and running now shows that the first test passes because it doesn’t really do anything and the second test fails because it throws the following:

---------------
Test can be created
Passed
---------------
Test with throw can be created
Failed
Unexpected exception thrown.
Program ended with exit code: 0

The output ends a bit abruptly and it’s time to fix that. We’ll add a summary next.

Summarizing the results

The summary can begin with a count of how many tests will be run. I thought about adding a running count for each test but decided against that because the tests are run in no particular order right now. I don’t mean that they will be run in a different order each time the testing application is run but they could be reordered if the code is changed and the project is rebuilt. This is because there is no fixed order when creating the final application that the linker will use between multiple .cpp compilation units. Of course, we would need tests spread across multiple files to see the reordering, and right now, all the tests are in Creation.cpp.

The point is that the tests register themselves based on how the global instances get initialized. Within a single .cpp source file, there is a defined order, but there is no guaranteed order between multiple files. Because of this, I decided not to include a number next to each test result.

We’ll keep track of how many tests passed and how many failed, and at the end of the for loop that runs all the tests, a summary can be displayed.

As an additional benefit, we can also change the runTests function to return the count of how many tests failed. This will let the main function return the failed count too so that a script can test this value to see whether the tests passed or how many failed. An application exit code of zero will mean that nothing failed. Anything other than zero will represent a failed run and will indicate how many tests have failed.

Here is the simple change to main.cpp to return the failed count:

int main ()
{
    return MereTDD::runTests();
}

Then, here is the new runTests function with the summary changes. The changes are described in three parts. All of this is a single function. Only the description is broken into three parts. The first part just displays the count of how many tests will be run:

inline int runTests ()
{
    std::cout << "Running "
        << getTests().size()
        << " tests
";

In the second part, we need to keep track of how many tests pass and how many fail, like this:

    int numPassed = 0;
    int numFailed = 0;
    for (auto * test: getTests())
    {
        std::cout << "---------------
"
            << test->name()
            << std::endl;
        try
        {
            test->run();
        }
        catch (...)
        {
            test->setFailed("Unexpected exception thrown.");
        }
        if (test->passed())
        {
            ++numPassed;
            std::cout << "Passed"
                << std::endl;
        }
        else
        {
            ++numFailed;
            std::cout << "Failed
"
                << test->reason()
                << std::endl;
        }
    }

And in the third part, after looping through all the tests and counting how many passed and how many failed, we display a summary with the counts, like this:

    std::cout << "---------------
";
    if (numFailed == 0)
    {
        std::cout << "All tests passed."
            << std::endl;
    }
    else
    {
        std::cout << "Tests passed: " << numPassed
            << "
Tests failed: " << numFailed
            << std::endl;
    }
    return numFailed;
}

Running the project now shows an initial count, the individual test results, and a final summary, and you can also see the application exit code is 1 because of the failed test:

Running 2 tests
---------------
Test can be created
Passed
---------------
Test with throw can be created
Failed
Unexpected exception thrown.
---------------
Tests passed: 1
Tests failed: 1
Program ended with exit code: 1

The final line that displays the exit code is not actually part of the testing application. This is normally not shown when the application is run. It’s part of the development environment that I am using to write this code. You would normally be interested in the exit code if you were running the testing application from a script such as Python as part of an automated build script.

We have one bit of cleanup still to do with the results. You see, right now, everything gets sent to std::cout and this assumption should be fixed so that the results can be sent to any output stream. The next section will do this cleanup.

Redirecting the output results

This is a simple edit that should not cause any real change to the application so far. Right now, the runTests function uses std::cout directly when displaying the results. We’re going to change this so that the main function will pass std::cout as an argument to runTests. Nothing will actually change because we’ll still be using std::cout for the results but this is a better design because it lets the testing application decide where to send the results, instead of the testing library.

By the testing library, I mean the Test.h file. This is the file that other applications will include in order to create and run tests. With the project we have so far, it’s a bit different because we’re writing tests to test the library itself. So, the whole application is just the Test.h file and the tests folder containing the testing application.

We first need to change main.cpp to include iostream and then pass std::cout to runTests, like this:

#include "../Test.h"
#include <iostream>
int main ()
{
    return MereTDD::runTests(std::cout);
}

Then, we no longer need to include iostream in Test.h, because it really doesn’t need any input and it doesn’t need to refer to std::cout directly. All it needs is to include ostream for the output stream. This could be the standard output, a file, or some other stream:

#ifndef MERETDD_TEST_H
#define MERETDD_TEST_H
#include <ostream>
#include <string_view>
#include <vector>

Most of the changes are to replace std::cout with a new parameter called output, like this in the runTests function:

inline int runTests (std::ostream & output)
{
    output << "Running "
        << getTests().size()
        << " tests
";
    int numPassed = 0;
    int numFailed = 0;
    for (auto * test: getTests())
    {
        output << "---------------
"
            << test->name()
            << std::endl;

Not all of the changes are shown in the previous code. All you need to do is replace every use of std::cout with output.

This was a simple change and does not affect the output of the application at all. In fact, it’s good to make changes like this that are isolated from other changes, just so the new results can be compared with previous results to make sure nothing unexpected has changed.

Summary

This chapter introduced macros and their ability to generate code based on the line number as a way to enable multiple tests. Each test is its own class with its own uniquely named global object instance.

Once multiple tests were supported, then you saw how to track and report the results of each test.

The next chapter will use the build failures in this chapter to show you the first step in the TDD process. We’ve been following these process steps already without specifically mentioning them. You’ll learn more about the TDD process in the next chapter, and the way that the test library has been developed so far should start making more sense as you understand the reasons.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.119.248.149