Debugger APIs
Think back for a moment to some of your first programming experiences, the ones right after you discovered or were shown how to use the debugger. It changed how you interacted with the software. Instead of inserting print statements only to rip or comment them out later, you could step through, skipping over the parts irrelevant to your investigation and dropping to statement-level or even assembly-level detail to know intimately how your software really operated.
Have you used a debugger that handles multiple threads well? It vastly simplifies the task of controlling the interactions between the threads. You can pause a thread at a break point and walk it and the threads it interacts with through their executions in whatever sequence you wish.
Hmmm. That sounds a lot like what we’ve been doing with all of the previous techniques. Every instruction then becomes a seam, a place we can insert some level of control to reproduce race conditions, constrained only by the fundamental rules of the underlying machine or virtual machine execution model.
But how do debuggers do that? In short, every execution environment that pretends to multitask—and a few that don’t—has APIs for programmatic control of execution. Debuggers are just programs that use an esoteric set of interfaces. In the form in which we typically encounter them, they exercise their exquisite control based on our manually issued commands. Set a break point. Run to a particular line. Watch for a variable to change. With sufficient ingenuity, these same APIs can be used to exploit atomic seams.
Such power comes with a cost. Some debuggers require special privileges to execute, privileges that you may not want to grant to your automated test suite that any developer or tester could modify. Using these APIs often requires a detailed understanding of the underlying execution model, which at a minimum requires a steep and long learning curve on a technology that probably will not be used frequently unless you write compilers or debuggers.
If we jump to a level outside of code, we have more opportunity to leverage this technique. Although it may seem quaint in the age of highly capable, graphical IDEs, the lowly command-line debugger can mitigate the costs of this approach. Many command-line debuggers, including gdb, jdb, and the Perl debugger, are scriptable to varying degrees, allowing their use in automation frameworks.
One application of this technique uses the command-line debugger to set a variable watch that triggers a macro during a Monte Carlo simulation. The variable watch establishes the deviant condition that characterizes the sporadic failure. The commands associated with triggering the watch constitute the output you wish to receive when you’ve detected the deviant condition. The gdb commands13 in Listing 13-16 might display the stack trace, then continue execution when the reference count is nonzero at the end of a hypothetical garbage collection in a memory pool.
13. The gdb documentation can be found at http://sourceware.org/gdb/onlinedocs/gdb/Break-Commands.html#Break-Commands.
Listing 13-16: gdb commands to verify a reference count at the end of a run
break finalize_gc if refs > 0
commands
bt # Stack trace
cont
end
While this is certainly not a complete test-automation or race-reproduction solution, it can be a valuable tool in isolating race conditions and even in characterizing their rates of occurrence. Take, for example, the variation shown in Listing 13-17 on the garbage collection scenario previously covered. The sole purpose of this set of commands is to compute the failure percentage.
Listing 13-17: Computing failure percentage using the debugger
break gc
commands
set num_gcs = num_gcs + 1
cont
end
break finalize_gc if refs > 0
commands
set num_failures = num_failures + 1
refs = 0
cont
end
break exit
commands
printf "Garbage collection failure rate: %f%%
",
num_failures/num_gcs
cont
end