13
How to Test Floating-Point and Custom Values
We first encountered the need to test floating-point values in Chapter 5, Adding More Confirm Types, and created a simple solution that would let us compare floating-point values within a margin of error. We need the small margin because floating-point values that are close and might even look identical when displayed are almost always not exactly equal. These small differences make it hard to verify test results.
The main topics in this chapter are as follows:
- More precise floating-point comparisons
- Adding floating-point Hamcrest matchers
- Writing custom Hamcrest matchers
We’re going to improve the simple solution developed earlier into a much better way to compare floating-point values that is more precise and works for both small and big values. We’ll use the better comparison for both the earlier classical-style confirmations and the new Hamcrest-style confirmations.
You’ll also learn how to create your own Hamcrest matchers in this chapter. We’ll be creating a new matcher to test for inequality instead of always testing for equality, and you’ll see how to contain one matcher inside another so that you can better reuse a matcher without needing to duplicate all the matcher template specializations.
Finally, you’ll learn how to create another custom simple matcher that will be slightly different than the other matchers so far because the new matcher will not need an expected value.
Technical requirements
All code in this chapter uses standard C++ that builds on any modern C++ 20 or later compiler and standard library. The code is based on and continues enhancing the testing library from Part 1 of this book, Testing MVP.
You can find all the code for this chapter in the following GitHub repository:
https://github.com/PacktPublishing/Test-Driven-Development-with-CPP
More precise floating-point comparisons
Whenever improvements are needed, one of the first things to look for is a way to measure the current design. Back in Chapter 5, Adding More Confirm Types, we examined floating-point numbers and I explained that comparing any floating-point type value—float, double, or long double—directly with another floating-point value is a bad idea. The comparison is too sensitive to small rounding errors and will usually result in the two values comparing not equal.
In Chapter 5, I showed you how to add a small margin to the comparison so that an accumulation of errors would not throw off the comparison as long as the two numbers being compared were close enough in value to each other. In other words, two values can compare equal as long as they are close enough to each other.
But what margin should be used? We simply picked some small numbers, and that solution worked. We’re going to improve that solution. And now that you’re becoming familiar with test-driven development (TDD), we’re going to create some functions to help test our solution. All these functions will go at the top of Hamcrest.cpp.
The first function will convert a floating-point value into a fraction by dividing by a constant. We’ll divide by 10, like this:
template <typename T>T calculateFraction (T input){ T denominator {10}; return input / denominator;}This is a template, so it will work for float, double, and long double types. The intent is for the input to be a whole number, and this function will turn the number into tenths. Remember from Chapter 5 that tenths don’t have exact representations in binary. There will be a slight error introduced but not much because we only do one division calculation.
We’ll need another function that will generate bigger margins of error by doing more work, like this function:
template <typename T>T accumulateError (T input){ // First add many small amounts. T partialAmount {0.1}; for (int i = 0; i < 10; ++i) { input += partialAmount; } // Then subtract to get back to the original. T wholeAmount {1}; input -= wholeAmount; return input;}This function adds 1 and then subtracts 1, so the input should remain unchanged. But because we add many small amounts that will all equal 1, the function introduces many errors during the calculations. The result that gets returned should be close to the original input but not the same.
The last helper function will call the first two functions many times for many different values and count how many times the results are equal. The function looks like this:
template <typename T>int performComparisons (int totalCount){ int passCount {0}; for (int i = 0; i < totalCount; ++i) { T expected = static_cast<T>(i); expected = calculateFraction(expected); T actual = accumulateError(expected); if (actual == expected) { ++passCount; } } return passCount;}The function uses the fraction as the expected value since it should have the fewest errors. The expected value is compared with the actual value, which we get from accumulating many small errors. The two values should be close but not exactly equal. They should be close enough to be counted as equal, though.
Who defines what is close enough? That’s really up to you to decide. The tests we’re creating in this book might be allowing more errors than your application can tolerate. You’ll understand after reading this section how to modify your code if you need more or less tolerance. There is no right answer for how to compare floating-point values that will work for all applications. The best you can do is to be aware of your own needs and adapt the code to fit those needs.
The performComparisons function also uses the == operator without any type of margin. The results should have a lot of unequal results. But how many? Let’s write a test to find out!
Add this test to the end of Hamcrest.cpp:
TEST("Test many float comparisons"){ int totalCount {1'000}; int passCount = performComparisons<float>(totalCount); CONFIRM_THAT(passCount, Equals(totalCount));}The test will cycle through 1,000 values, turning each one into tenths, introducing errors, and counting how many compared equal. The results are really bad:
------- Test: Test many float comparisonsFailed confirm on line 125 Expected: 1000 Actual : 4Only four values were close enough to be considered equal with the standard equality operator. You might get slightly different results depending on your computer and your compiler. If you do get different results, then that should be more evidence about how unreliable floating-point comparisons are. How about double and long double types? Add these two tests to find out:
TEST("Test many double comparisons"){ int totalCount {1'000}; int passCount = performComparisons<double>(totalCount); CONFIRM_THAT(passCount, Equals(totalCount));}TEST("Test many long double comparisons"){ int totalCount {1'000}; int passCount = performComparisons<long double>(totalCount); CONFIRM_THAT(passCount, Equals(totalCount));}The results are just as bad and look like this:
------- Test: Test many double comparisonsFailed confirm on line 132 Expected: 1000 Actual : 4------- Test: Test many long double comparisonsFailed confirm on line 139 Expected: 1000 Actual : 0Let’s add a margin to the equality comparison to see how much better the comparisons become. We’ll start with the values used in the existing confirm overloads in Test.h. One of the overloads looks like this:
inline void confirm ( float expected, float actual, int line){ if (actual < (expected - 0.0001f) || actual > (expected + 0.0001f)) { throw ActualConfirmException( std::to_string(expected), std::to_string(actual), line); }}The value we are interested in is the hardcoded literal floating-point value. In this case, it’s 0.0001f. All we need to do is create three more helper functions that return these values. Note that the double and long double overloads have a different value than the float type. Place these three helper functions in Hamcrest.cpp, right before the performComparisons function, like this:
constexpr float getMargin (float){ return 0.0001f;}constexpr double getMargin (double){ return 0.000001;}constexpr long double getMargin (long double){ return 0.000001L;}These three helper functions will let us customize the margin for each type. They each take a floating-point type parameter that is only used to determine which function to call. We don’t actually need or use the parameter value passed to the function. We’ll call these helper functions from within the performComparisons template, which will know the type to be used based on how the template was constructed.
We’re also going to slightly change how we compare with a margin. Here’s an example of how the confirm functions compare:
if (actual < (expected - 0.0001f) || actual > (expected + 0.0001f))Instead of this, we’re going to subtract the expected value from the actual value and then compare the absolute value of that subtraction result with the margin. We need to include cmath at the top of Hamcrest.cpp for the abs function, and we’re going to need limits soon also, like this:
#include "../Test.h"#include <cmath>#include <limits>And now, we can change the performComparisons function to use the margin, like this:
template <typename T>int performComparisons (int totalCount){ int passCount {0}; for (int i = 0; i < totalCount; ++i) { T expected = static_cast<T>(i); expected = calculateFraction(expected); T actual = accumulateError(expected); if (std::abs(actual - expected) < getMargin(actual)) { ++passCount; } } return passCount;}After making these changes, all the tests pass, like this:
------- Test: Test many float comparisonsPassed------- Test: Test many double comparisonsPassed------- Test: Test many long double comparisonsPassedThis means that all 1,000 values are now matching within a small margin of error. This is the same solution explained in Chapter 5. We should be good, right? Not quite.
The problem is that the margin value is quite big for small numbers and too small for big numbers. All the tests are passing, but that’s just because we have a margin that’s big enough to let a lot of comparisons be treated as equal.
To see this, let’s refactor the comparison out of the performComparisons function so that the check is in its own function, like this:
template <typename T>bool compareEq (T lhs, T rhs){ return std::abs(lhs - rhs) < getMargin(lhs);}template <typename T>int performComparisons (int totalCount){ int passCount {0}; for (int i = 0; i < totalCount; ++i) { T expected = static_cast<T>(i); expected = calculateFraction(expected); T actual = accumulateError(expected); if (compareEq(actual, expected)) { ++passCount; } } return passCount;}And then we can write a couple tests to call compareEq directly, like this:
TEST("Test small float values"){ // Based on float epsilon = 1.1920928955078125e-07 bool result = compareEq(0.000001f, 0.000002f); CONFIRM_FALSE(result);}TEST("Test large float values"){ // Based on float epsilon = 1.1920928955078125e-07 bool result = compareEq(9'999.0f, 9'999.001f); CONFIRM_TRUE(result);}The test for small float values compares two numbers that are obviously different, yet the comparison function will consider them equal and the test fails. The fixed margin considers any float values within 0.0001f to be equal. We want the two values to compare not equal, but our margin is big enough that they are considered to be equal.
What is the epsilon value that the comment refers to? We’ll start using the actual epsilon values in just a moment, and this is why I suggested that you include limits. Floating-point numbers have a concept called epsilon, which is a value defined in limits for each floating-point type. The epsilon value represents the smallest distance between adjacent floating-point values for values between 1.0 and 2.0. Remember that floating-point values can’t represent every possible fractional number, so there are gaps between the numbers that can be represented.
You can see the same thing yourself if you write down numbers with only a fixed number of decimal places on paper. Let’s say that you limit yourself to only using two digits after the decimal point. You could write 1.00 and 1.01 and 1.02. Those are adjacent values. In fact, 1.00 and 1.02 are the closest numbers you can represent to 1.01 by using only two digits after the decimal place. What about a number such as 1.011? It’s definitely closer to 1.01 than 1.02 but we can’t write 1.011 because it needs three digits after the decimal point. The epsilon value for our experiment is 0.01. Floating-point numbers have a similar problem except that the value of epsilon is smaller and not a simple value such as 0.01.
Another complication is that the distance between adjacent floating-point numbers increases as the numbers get larger, and the distance decreases as the numbers get smaller. The test for small float values uses small values, but the values are much bigger than epsilon. Because the numbers are much bigger than epsilon, we want the test to fail. The test passes because our fixed margin is even bigger than epsilon.
The test for large float values also fails. It uses two values that are different by 0.001f, which would be a really big difference if we were comparing 1.0f with 1.001f. At small values, a difference of 0.001f would be enough to cause the values to compare not equal. But we’re not dealing with small values—we’re dealing with values that are almost 10,000! We now want the larger values to be considered equal because the fractional part makes up a smaller percentage of the larger numbers. The test fails because our fixed margin doesn’t consider that the values are larger and only looks at the difference, which is greater than the fixed margin allows.
We can also test the other floating-point types. Add these two similar tests for double right after the two tests added for small and large float values, like this:
TEST("Test small double values"){ // Based on double epsilon = 2.2204460492503130808e-16 bool result = compareEq(0.000000000000001, 0.000000000000002); CONFIRM_FALSE(result);}TEST("Test large double values"){ // Based on double epsilon = 2.2204460492503130808e-16 bool result = compareEq(1'500'000'000'000.0, 1'500'000'000'000.0003); CONFIRM_TRUE(result);}For the double type, we have a different epsilon value that’s much smaller than the epsilon for floats, and we also have many more significant digits to work with, so we can use numbers with more digits. We were limited to only about 7 digits when working with floats. With doubles, we can use numbers with about 16 digits. Notice that with doubles, we need a large value in the trillions in order to see a difference of 0.0003 that should be compared as equal.
If you are wondering how I arrived at these test numbers, I just picked small numbers just one decimal place bigger than epsilon for the small value tests. And for the large values, I choose a bigger number that I multiplied by (1 + epsilon) to arrive at the other number to be compared with. I then rounded the other number a bit so that it would be a bit closer. I had to choose a big number to start with that would stay within the number of digits allowed for each type.
Since we’re using the double epsilon value for long doubles, the tests for small and large long doubles look similar to the tests for double. The long double tests look like this:
TEST("Test small long double values"){ // Based on double epsilon = 2.2204460492503130808e-16 bool result = compareEq(0.000000000000001L, 0.000000000000002L); CONFIRM_FALSE(result);}TEST("Test large long double values"){ // Based on double epsilon = 2.2204460492503130808e-16 bool result = compareEq(1'500'000'000'000.0L, 1'500'000'000'000.0003L); CONFIRM_TRUE(result);}The only difference between the double tests and the long double tests is the long double suffix L at the end of the long double literal values.
After adding all six tests for small and large floating-point type tests, they all fail when run.
The reason for the failures is the same for each type. The small value tests all fail because the fixed margin considers the values to be equal when they should not be equal, and the large value tests consider the values to be not equal when they really are close, considering the large value. In fact, the large values are all within a single epsilon value from each other. The large values are as close as they can possibly get without being exactly equal. Sure—the long double large values could have been closer, but we’re simplifying long doubles a bit by using the bigger epsilon from the double type.
We need to enhance the compareEq function so that the margin can be smaller for small values and bigger for big values. The moment we take on the responsibilities of comparing floating-point values, there are a lot of details that need to be handled. We skipped the extra details in Chapter 5. We’re also going to skip some of the details even here. If you haven’t realized it yet, dealing with floating-point values is really hard. The moment you think everything is working is when another detail comes along that changes everything.
Let’s first fix the getMargin functions to return the real epsilon values modified slightly for each type, like this:
constexpr float getMargin (float){ // 4 is chosen to pass a reasonable amount of error. return std::numeric_limits<float>::epsilon() * 4;}constexpr double getMargin (double){ // 4 is chosen to pass a reasonable amount of error. return std::numeric_limits<double>::epsilon() * 4;}constexpr long double getMargin (long double){ // Use double epsilon instead of long double epsilon. // Double epsilon is already much bigger than // long double epsilon so we don't need to multiply it. return std::numeric_limits<double>::epsilon();}The getMargin functions now use the epsilon values for the types as defined in numeric_limits. The margins are tuned for our needs. You might want to multiply by different numbers, and you might want to use the real epsilon value for long doubles. The reason we want bigger margins than epsilon itself is that we want to consider values to be equal that are more than just one epsilon value greater or lesser away from each other. We want a little more room for the accumulation of at least a few calculation errors. We multiply epsilon by 4 to give that extra room, and we use the double epsilon for long doubles, which might really be too much already. But these margins work for what we need.
We’ll use the more accurate margin values in the new compareEq function, which looks like this:
template <typename T>bool compareEq (T lhs, T rhs){ // Check for an exact match with operator == first. if (lhs == rhs) { return true; } // Subnormal diffs near zero are treated as equal. T diff = std::abs(lhs - rhs); if (diff <= std::numeric_limits<T>::min()) { return true; } // The margin should get bigger with bigger absolute values. // We scale the margin up by the larger value or // leave the margin unchanged if larger is less than 1. lhs = std::abs(lhs); rhs = std::abs(rhs); T larger = (lhs > rhs) ? lhs : rhs; larger = (larger < 1.0) ? 1.0 : larger; return diff <= getMargin(lhs) * larger;}I like to use parameter names lhs and rhs for operator-type functions such as this. The abbreviations stand for left-hand side and right-hand side, respectively.
Consider these two numbers:
3 == 4When making a comparison of these, 3 is on the left side of the operator and would be the lhs argument, while 4 is on the right and would be the rhs argument.
There’s always a chance that the two numbers being compared are exactly equal to each other. So, the first thing we check is for an exact match using the == operator.
The compareEq function goes on to check the difference between the two numbers for a subnormal result. Remember I said that floating-point numbers are complicated? There could be an entire book written about floating-point math, and there are probably several books already written. I won’t go into much explanation about subnormal values except to say that this is how floating-point values are represented when they are extremely close to zero. We’ll consider any two subnormal values to be equal.
Subnormal values are also a good reason to compare your numbers with each other instead of comparing their difference with zero. You might wonder what the problem is. Doesn’t the code in the compareEq function subtract one value from the other to arrive at the difference? Yes, it does. But our compareEq function doesn’t try to compare the difference with zero directly. We figure out which of the two values is bigger and then scale the margin by multiplying the margin with the bigger value. We also avoid scaling the margin down when we are comparing values less than 1.0.
If you have two values to compare and instead of passing them to compareEq, you pass their difference and compare the difference with zero, then you remove the ability of the compareEq function to do the scaling because the compareEq function would then only see a small difference and zero being compared.
The lesson here is to always pass your numbers to be compared directly to the compareEq function and let it figure out how much the two numbers are different by taking into account how big the numbers are. You’ll get more accurate comparisons.
We could make the compareEq function even more elaborate. Maybe we could consider the sign of subnormal values instead of considering all of them to be equal, or maybe we could scale the margin down more so that we get very precise when dealing with subnormal values. This is not a book about math, so we’re going to stop adding more to the compareEq function.
After making the changes to compareEq, all the tests pass. We now have a solution that allows small amounts of accumulated errors and lets two numbers compare equal when they are close enough. The solution works for both really small numbers and really big numbers. The next section will turn the code we explored here into a better Hamcrest equality matcher.
Adding floating-point Hamcrest matchers
We explored better floating-point comparisons in the previous section, and now it’s time to use the comparison code in the unit test library. Some of the code should be moved into Test.h where it fits better and can then be used by the test library. The rest of the code that was written should stay in Hamcrest.cpp because it’s code that supports the tests.
The code that needs to be moved is the compareEq function and the three getMargin functions that compareEq calls to get the margins. We also need to move the includes of cmath and limits into Test.h, like this:
#include <cmath>#include <cstring>#include <limits>#include <map>#include <ostream>#include <string_view>#include <type_traits>#include <vector>The three getMargin functions and the compareEq function can be moved into Test.h, right before the first override of the confirm function that accepts Booleans. None of the code in the moved functions needs to change. Just cut the includes and the functions out of Hamcrest.cpp and paste the code into Test.h.
We might as well fix the existing floating-point classic confirm functions. This is why I had you move the compareEq function into Test.h immediately before the first confirm function. The change to the existing floating-point confirm functions is simple. They need to call compareEq instead of using hardcoded margins that don’t scale. The confirm function for float types looks like this after the change:
inline void confirm ( float expected, float actual, int line){ if (not compareEq(actual, expected)) { throw ActualConfirmException( std::to_string(expected), std::to_string(actual), line); }}The other two confirm functions that accept double and long double types should be changed to look similar. All three confirm functions will create the correct template of compareEq based on the expected and actual parameter types.
We should build and run the test application to make sure nothing broke with this small refactoring. And all the tests pass. We now have updated classic style confirm functions that will work better with floating-point comparisons.
We can make the code slightly better, though. We have three almost identical functions that are different only by their parameter types. The only reason for the three functions is that we want to override the confirm function for floating-point types. But since we’re using C++20, let’s use concepts instead! Concepts are a new feature that we’ve already started using when we specialized the Equals matcher to work with char arrays and char pointers in the previous chapter. Concepts allow us to tell the compiler which types are acceptable for template parameters and even function parameters. In the previous chapter, we were only using the requires keyword to place some restrictions on the template parameters. We’ll be using more well-known concepts here in this chapter.
We need to include concepts like this in Test.h:
#include <concepts>#include <cmath>#include <cstring>#include <limits>#include <map>#include <ostream>#include <string_view>#include <type_traits>#include <vector>And then, we can replace the three confirm functions that accept float, double, and long double types with a single template that uses the floating_point concept, like this:
template <std::floating_point T>void confirm ( T expected, T actual, int line){ if (not compareEq(actual, expected)) { throw ActualConfirmException( std::to_string(expected), std::to_string(actual), line); }}This new template will only accept floating-point types, and by making both expected and actual share the same type T, then both types must be the same. The definition of floating_point is one of the well-known concepts defined in the concepts header.
Now that we have the classic style confirmations working, let’s get the Hamcrest Equals matcher working for floating-point values. We can first change the three large floating-point tests in Hamcrest.cpp to stop calling compareEq directly and instead use the CONFIRM_THAT macro so that they look like this:
TEST("Test large float values"){ // Based on float epsilon = 1.1920928955078125e-07 CONFIRM_THAT(9'999.0f, Equals(9'999.001f));}TEST("Test large double values"){ // Based on double epsilon = 2.2204460492503130808e-16 CONFIRM_THAT(1'500'000'000'000.0, Equals(1'500'000'000'000.0003));}TEST("Test large long double values"){ // Based on double epsilon = 2.2204460492503130808e-16 CONFIRM_THAT(1'500'000'000'000.0L, Equals(1'500'000'000'000.0003L));}We’re not going to change the tests for the small floating-point values yet because we don’t have a matcher that does inequality comparisons. The solution might be as simple as putting the not keyword in front of Equals, but let’s hold off on that for just a moment because we’ll be exploring our options in the next section.
With the change to the tests, they should fail because we haven’t yet specialized the Equals matcher to do anything different for floating-point types. Building and running the test application shows that the three tests do fail, like this:
------- Test: Test large float valuesFailed confirm on line 152 Expected: 9999.000977 Actual : 9999.000000------- Test: Test small double valuesPassed------- Test: Test large double valuesFailed confirm on line 165 Expected: 1500000000000.000244 Actual : 1500000000000.000000------- Test: Test small long double valuesPassed------- Test: Test large long double valuesFailed confirm on line 178 Expected: 1500000000000.000300 Actual : 1500000000000.000000Notice how the expected values printed in the summary report don’t exactly match the literal values given in the tests for the float and the double types. The long double does display a value that matches the value given in the test. The discrepancy is because floating-point variables are unable to always match exact values. The differences become more visible with floats, a little less visible with doubles, and closer to the desired values with long doubles.
The steps we just went through follow TDD. We modified existing tests instead of creating new tests because we don’t expect callers to use compareEq directly. The tests were initially written to call compareEq directly to show that we had a solution for floating-point types that worked. Modifying the tests to the desired usage is the right thing to do, and then, by running the tests, we can see the failures. This is good because we expected the tests to fail. Had the tests passed instead, then we would need to find the reason for the unexpected success.
Let’s get the tests to pass again! We need a version of Equals that knows how to work with floating-point types. We’ll use the concept floating_point that we just used for the classic style confirmations to create another version of Equals that will call compareEq for floating-point types. Place this new Equals specialization in Test.h right after Equals that works with char pointers, like this:
template <std::floating_point T>class Equals<T> : public Matcher{public: Equals (T const & expected) : mExpected(expected) { } bool pass (T const & actual) const { return compareEq(actual, mExpected); } std::string to_string () const override { return std::to_string(mExpected); }private: T mExpected;};That’s all we need to change to get the tests passing again. The new Equals specialization accepts any floating-point type and will be preferred by the compiler for floating-point types instead of the general-purpose Equals template. The floating-point version of Equals calls compareEq to do the comparison. We also don’t need to worry about which types will be passed to to_string and can simply call std::to_string since we know that we will have one of the built-in floating-point types. The to_string assumption could fail if the user passes in some other type that has been created to be a floating_point concept type, but let’s keep the code as simple as it can be for now and not worry about custom floating-point types.
The next section will start by creating a solution to test for inequality instead. We’ll use the solution we create in the next section to modify the small floating-point Hamcrest tests.
Writing custom Hamcrest matchers
The previous section ended with the Equals matcher specialized to call compareEq for floating-point types. We also modified the tests for large floating-point values because they could use the Hamcrest style and the Equals matcher.
We left the tests for small floating-point values unchanged because those tests need to make sure that the actual and expected values are not equal.
We want to update the small floating-point value tests and need a way to test for not equal values. Maybe we could create a new matcher called NotEquals, or we could put the not keyword in front of the Equals matcher.
I’d like to avoid the need for a new matcher if possible. We don’t really need any new behavior—we just need to flip the results of the existing Equals matcher. Let’s try modifying the small floating-point value tests to look like this in Hamcrest.cpp:
TEST("Test small float values"){ // Based on float epsilon = 1.1920928955078125e-07 CONFIRM_THAT(0.000001f, not Equals(0.000002f));}TEST("Test small double values"){ // Based on double epsilon = 2.2204460492503130808e-16 CONFIRM_THAT(0.000000000000001, not Equals(0.000000000000002));}TEST("Test small long double values"){ // Based on double epsilon = 2.2204460492503130808e-16 CONFIRM_THAT(0.000000000000001L, not Equals(0.000000000000002L));}The only changes are to stop calling compareEq directly and to use the CONFIRM_THAT macro with the Equals matcher. Notice that we flip the results of the Equals matcher by placing the not keyword in front.
Does it build? No. We get compile errors similar to this:
MereTDD/tests/Hamcrest.cpp:145:29: error: no match for 'operator!' (operand type is 'MereTDD::Equals<float>') 145 | CONFIRM_THAT(0.000001f, not Equals(0.000002f)); | ^~~~~~~~~~~~~~~~~~~~~The not keyword in C++ is a shortcut for operator !. Normally with TDD, the next step would be to modify the code so that the tests can build. But we have a problem. The not keyword expects the class to have an operator ! method or some way to cast the class to a Boolean. Either option requires that the class be able to generate a bool value, and that’s not how the matchers work. In order for a matcher to know whether the result should pass or not, it needs to know the actual value. The confirm_that function passes the matcher the needed actual value as an argument to the pass method. We can’t turn a matcher by itself into a bool result.
We’re going to have to create a NotEquals matcher. While not my first preference, a new matcher is acceptable from a test perspective. Let’s change the tests to look like this instead:
TEST("Test small float values"){ // Based on float epsilon = 1.1920928955078125e-07 CONFIRM_THAT(0.000001f, NotEquals(0.000002f));}TEST("Test small double values"){ // Based on double epsilon = 2.2204460492503130808e-16 CONFIRM_THAT(0.000000000000001, NotEquals(0.000000000000002));}TEST("Test small long double values"){ // Based on double epsilon = 2.2204460492503130808e-16 CONFIRM_THAT(0.000000000000001L, NotEquals(0.000000000000002L));}Another reason that I wanted to avoid a new matcher is to avoid the need to specialize the new matcher as we did for the Equals matcher, but there is a way to create a matcher called NotEquals and base its implementation on all the work we did for the Equals matcher. All we need to do is contain the Equals matcher and flip the pass result, like this:
template <typename T>class NotEquals : public Matcher{public: NotEquals (T const & expected) : mExpected(expected) { } template <typename U> bool pass (U const & actual) const { return not mExpected.pass(actual); } std::string to_string () const override { return "not " + mExpected.to_string(); }private: Equals<T> mExpected;};Add the NotEquals matcher right after all the template specializations of the Equals matcher in Test.h.
The NotEquals matcher is a new matcher type that contains an Equals matcher for its mExpected data member. This will give us all the specialization we did for the Equals matcher. Whenever the NotEquals::pass method is called, we just call the mExpected.pass method and flip the result. And whenever the to_string method is called, we just add "not " to whichever string mExpected provides.
One interesting thing to notice is that the pass method is itself a template based on a type U. This will let us construct a NotEquals matcher given a string literal and then call the pass method with std::string.
We should add a test for using the NotEquals matcher with a string literal and std::string or, even better, to extend an existing test. We have two tests already that work with strings, string literals, and char pointers. Both tests are in Hamcrest.cpp. The first test should look like this:
TEST("Test hamcrest style string confirms"){ std::string s1 = "abc"; std::string s2 = "abc"; CONFIRM_THAT(s1, Equals(s2)); // string vs. string CONFIRM_THAT(s1, Equals("abc")); // string vs. literal CONFIRM_THAT("abc", Equals(s1)); // literal vs. string // Probably not needed, but this works too. CONFIRM_THAT("abc", Equals("abc")); // literal vs. literal std::string s3 = "def"; CONFIRM_THAT(s1, NotEquals(s3)); // string vs. string CONFIRM_THAT(s1, NotEquals("def")); // string vs. literal CONFIRM_THAT("def", NotEquals(s1)); // literal vs. string}And the second test should be modified to look like this:
TEST("Test hamcrest style string pointer confirms"){ char const * sp1 = "abc"; std::string s1 = "abc"; char const * sp2 = s1.c_str(); // avoid sp1 and sp2 being same CONFIRM_THAT(sp1, Equals(sp2)); // pointer vs. pointer CONFIRM_THAT(sp2, Equals("abc")); // pointer vs. literal CONFIRM_THAT("abc", Equals(sp2)); // literal vs. pointer CONFIRM_THAT(sp1, Equals(s1)); // pointer vs. string CONFIRM_THAT(s1, Equals(sp1)); // string vs. pointer char const * sp3 = "def"; CONFIRM_THAT(sp1, NotEquals(sp3)); // pointer vs. pointer CONFIRM_THAT(sp1, NotEquals("def")); // pointer vs. literal CONFIRM_THAT("def", NotEquals(sp1)); // literal vs. pointer CONFIRM_THAT(sp3, NotEquals(s1)); // pointer vs. string CONFIRM_THAT(s1, NotEquals(sp3)); // string vs. pointer}Building and running the test application shows that all the tests pass. Instead of adding new tests, we were able to modify the existing tests because the two existing tests were focused on the string and char pointer types. The NotEquals matcher fit right into the existing tests.
Having both Equals and NotEquals matchers gives us more than we had with the classic style confirmations, and we can go further by creating another matcher. You can also create matchers to do whatever you want in your test projects. We’re going to create a new matcher in the MereTDD namespace but you can put yours in your own namespace. The matcher we’ll be creating will test to make sure that an integral number is even. We’ll call the matcher IsEven, and we can write a couple of tests in Hamcrest.cpp to look like this:
TEST("Test even integral value"){ CONFIRM_THAT(10, IsEven<int>());}TEST("Test even integral value confirm failure"){ CONFIRM_THAT(11, IsEven<int>());}You’ll notice something different about the IsEven matcher: it doesn’t require an expected value. The matcher only needs the actual value passed to it in order to confirm whether the actual value is even or not. Because there’s nothing to pass to the constructor when creating an IsEven matcher in the test, we need to specify the type, like this:
IsEven<int>()The second test should fail, and we’ll use the failure to get the exact error message so that we can turn the test into an expected failure. But we first need to create an IsEven matcher. The IsEven class can go in Test.h immediately after the NotEquals matcher, like this:
template <std::integral T>class IsEven : public Matcher{public: IsEven () { } bool pass (T const & actual) const { return actual % 2 == 0; } std::string to_string () const override { return "is even"; }};I wanted to show you an example of a really simple custom matcher so that you’ll know that they don’t all need to be complicated or have multiple template specializations. The IsEven matcher just tests the actual value in the pass method to make sure it’s even, and the to_string method returns a fixed string.
Building and running shows the even value test passes while the intended failure test fails, like this:
------- Test: Test even integral valuePassed------- Test: Test even integral value confirm failureFailed confirm on line 185 Expected: is even Actual : 11With the error message, we can modify the even confirm failure test so that it will pass with an expected failure, like this:
TEST("Test even integral value confirm failure"){ std::string reason = " Expected: is even
"; reason += " Actual : 11"; setExpectedFailureReason(reason); CONFIRM_THAT(11, IsEven<int>());}Building and running now shows that both tests pass. One passes successfully and the other passes with an expected failure, like this:
------- Test: Test even integral valuePassed------- Test: Test even integral value confirm failureExpected failure Expected: is even Actual : 11That’s all there is to making custom matchers! You can make matchers for your own classes or add custom matchers for new behaviors. Maybe you want to verify that a number has only a certain number of digits, that a string begins with some given text prefix, or that a log message contains a certain tag. Remember how in Chapter 10, The TDD Process in Depth, we had to verify tags by writing to a file and then scanning the file to make sure the line just written contained a tag? We could have a custom matcher that looks for a tag instead.
Summary
One of the main benefits of the Hamcrest style confirmations is their ability to be extended through custom matchers. What better way to explore this ability than through floating-point confirmations? Because there is no single best way to compare floating-point values, you might need a solution that’s tuned to your specific needs. You learned about a good general-purpose floating-point comparison technique in this chapter that scales a small margin of error so that bigger floating-point values are allowed to differ by greater amounts as the values get bigger and still be considered to be equal.
If this general solution doesn’t meet your needs, you now know how to create your own matcher that will do exactly what you need.
And the ability to extend matchers doesn’t stop at floating-point values. You might have your own custom behavior that you need to confirm, and after reading this chapter, you now know how to create a custom matcher to do what you need.
Not all matchers need to be big and complicated and have multiple template specializations. You saw an example of a very simple custom matcher that confirms whether a number is even or not.
We also made good use of the concepts feature, new in C++20, which allows you to easily specify requirements on your template types. We made good use of concepts in this chapter to make sure that the floating-point matcher only works for floating-point types and that the IsEven matcher only works for integral types. You can use concepts in your matchers too, which will help you control how your matchers can be used.
The next chapter will explore how to test services and will introduce a new service project that uses all the code developed so far in this book.