Although it is impossible to test code without concrete knowledge of what a particular program does, and how, there are nevertheless some general principles of testing that are useful to follow. Correctly designed and implemented code must produce the right answer when given correct inputs. Furthermore, when given incorrect ones, the program should not silently die, crash, or get stuck, but should diagnose the problem—where, why, and if necessary, when the error happened—and then either gracefully terminate or return to the initial state from which it can process the next input. Testing must include everything from unit tests of each single class, to unit tests of groups of classes working together, to a test of the whole application.
To the extent possible, you should try to create a reproducible test that leads to the same results when repeated. This can be a challenge when dealing with multi-threaded applications, when the timing of events between different threads is an issue, but even in cases like that it is usually possible to convert tests of some parts of the code to a single-threaded mode where the results should be totally deterministic.
In order to test multiple classes, organize them in a hierarchy such that some classes are considered more “basic” than others. In other words, the classes on one level of the hierarchy can make calls only to the classes on the same level or below, not above. Then the sequence of testing is clear. Otherwise, you’ll face a chicken-and-egg problem when deciding what to test first. An even better design is when a class at each level uses only classes below it, as shown in Figure 14-1.
Each piece of code that expects some input must be tested with both correct and incorrect inputs. Try to “push” the code and see how it behaves not only under normal but also abnormal circumstances. For instance, if the code expects a pointer (or pointers) to some inputs, what would happen if you provide NULL(s) instead? If an algorithm expects integers, test whether there could be an integer overflow. If an algorithm expects doubles, test what happens if they are very small or very large. See how code behaves when different inputs differ by several orders of magnitude. Will the algorithm lose its accuracy?
If the algorithm works with input of a variable size (e.g., an array, vector, or matrix, or if the code reads several numbers from a file), see what happens when the size of input grows by an order of magnitude. You must have an understanding of the complexity of your algorithm, e.g., if the input contains N units of information, how much does the time of processing increase as a function of N when N increases? Then test it whether this is true in practice.
If the algorithm does some calculation numerically but in specific cases it has an analytical solution, compare them. If there is asymptotic behavior when some parameter becomes small or large, test it.
If the algorithm does something in a very smart and efficient way, consider writing a brute-force version of the same algorithm. Although this will be much slower, it will also be much simpler and therefore less error-prone. Then compare the results, at least for small input size.
If an algorithm takes as an input an arbitrary set of numbers,
such as in the case of sorting, it is usually a good idea to generate
test inputs in a pseudo-random manner—e.g., using the function rand()—so
that you can create a lot of different test sets easily. This technique
still allows the tests to be repeatable, because you can recreate the same
set by specifying the same seed for the random number generator.
Always look for special cases. If the algorithm takes an array, what happens if it is empty or contains just one element? What if all elements of an array are the same? If it takes a matrix, what happens if the determinant of that matrix is zero?
If you use hash sets or hash maps, test them for collisions with a realistic set of inputs. Try to look for worst-case scenarios.
If your inputs depend on a calendar date, make sure to include the February 29th in a leap year. I have found that in algorithms generating sets of dates starting from some initial date, this is usually a very special case that can sometimes lead to the discovery of rare but interesting bugs. Therefore, if you are testing data that includes a range of dates, make sure that it is at least five years long so that it includes at least one leap year. (Strictly speaking, not every five-year interval includes a leap year, because the years 1900, 2100, 2200, and 2300 are not leap years, so you might need about nine years of data instead, depending on the century in which you are reading this book.
Automate your testing as much as possible. The best set of tests is one that runs with one push of a button and tests everything there is to test about your code. There are many frameworks and utilities that make it easy to achieve this automation.
Plan your work so that you spend between 30% to 50% of your time testing. This is the part of planning that is very easy to underestimate and where things tend to go wrong, thus ruining delivery schedules. Remember: the more effort you spend on testing, the easier your life will be when your code goes into production.