Verification by Containment
Let’s consider the code in Listing 7-1.
Listing 7-1: A simple string composition function
public String describeAddition(int left, int right) {
return "The sum of " + left + " and " + right
+ " is " + (left + right);
}
While the example is trivial and a bit contrived, it is a fair representation of the way strings are commonly handled. How might we test such a method? Well, what do we know about the method? We know some fragments of the text, but those should not change from run to run. What we are probably most concerned with is that the correct values make it into the output. We might write a test like that in Listing 7-2.
Listing 7-2: A test for the code in Listing 7-1
public void testDescribeAddition() {
int left = 3;
int right = 5;
int expectedSum = left + right;
String expectedResult = "The sum of " + left
+ " and " + right
+ " is " + expectedSum;
String actualResult = describeAddition(left, right);
assertEquals(expectedSum, actualResult);
}
An ideal test will not break when trivial changes are made. What if we wanted to change the initial fragment to “The sum of the numbers” instead? We would have to make the change in two places, doubling the maintenance cost for that change. If there were multiple tests for the method—testing negative numbers, zero values, or special sums, for example—then the maintenance cost would increase severalfold.
I like to call the fact that the string in the test is the same as the string in the code “coincidental equality” or “coincidental similarity.” We know the values are the same by construction. However, we have applied copy–paste reuse in a manner that increases the maintenance burden and fragility of our tests. There must be a better way.
Let’s change our equality assertion to something more relaxed but at the same time less fragile (see Listing 7-3).
Listing 7-3: Less fragile assertions for the test in Listing 7-2
assertTrue(actualResult.contains(left));
assertTrue(actualResult.contains(right));
assertTrue(actualResult.contains(expectedSum));
Now we are resistant to changes in the initial string—or for that matter any of the string literals—but we have weakened the verification. We are only verifying part of the result. We have traded strength of verification for an improvement in the maintainability.
Additionally, what if one of our operands was 0? The expected sum would be the same as one of our operands. Or what if our operands were equal or the resulting sum had one of our operands as a substring, such as adding 9 and 1? The result of 10 would give us an ambiguous match against the operand of 1. In these cases, we would not know if the correct occurrence was verified by the assertion, leaving us with tests of unreliable value.
This technique can be useful and is certainly better than nothing. When using it, we should understand its limitations and use it judiciously. It can be particularly appropriate when the result is complex enough that a match gives us a high likelihood of correctness. If that is the case, we should be careful that we’re not overspecifying the correctness by, for example, testing for containment of a string in which white space doesn’t matter but our exemplar has a specific layout. Let’s look at another approach.