For anything but very small benchmarks and small pieces of proof of concept code, it is usually a good idea to measure "outside the system". By that, we mean running the benchmark with some kind of external driver. The driver is an application independent of the actual benchmark code on which the performance evaluation is to take place.

Working with a driver usually means that the driver injects a workload into the benchmark, for example over the network. The entire response time, including network traffic, for the benchmark to run its payload code and respond is then measured.

Using a driver has the benefit that response times can be accurately measured without mixing up the measurements with the data generation or load generation. The driver can also be moved to another machine, and can be run under lower total load, to make sure that neither data generation nor load generation is a bottleneck in the benchmark.

The need for externalizing certain measurements can perhaps be more simply illustrated by a smaller example. Consider the following benchmark that measures how fast an implementation of the MD5 message digest algorithm runs on random data:

import java.util.Random;
import java.security.*;
public class Md5ThruPut {
static MessageDigest algorithm;
static Random r = new Random();
static int ops;
public static void main(String args[]) throws Exception { algorithm = MessageDigest.getInstance("MD5");
algorithm.reset();
long t0 = System.currentTimeMillis();
test(100000);
long t1 = System.currentTimeMillis();
System.out.println((long)ops / (t1 - t0) + " ops/ms");
}
public static void test(int size) {
for (int i = 0; i < size; i++) {
byte b[] = new byte[1024];
r.nextBytes(b);
digest(b);
}
}
public static void digest(byte [] data) {
algorithm.update(data);
algorithm.digest();
ops++;
}
}

If our goal was to only measure the performance of the MD5 algorithm, the previous benchmark is less than optimal, as the generation of the random input data is part of the time of an operation being measured. So, the total runtime will not only reflect the performance of the MD5 algorithm, but it will also reflect the performance of the random number generator. This was probably not intended. A better version of the MD5 benchmark would look like the following program:

import java.util.Random;
import java.security.*;
public class Md5ThruPutBetter {
static MessageDigest algorithm;
static Random r = new Random();
static int ops;
static byte[][] input;
public static void main(String args[]) throws Exception { algorithm = MessageDigest.getInstance("MD5");
algorithm.reset();
generateInput(100000);
long t0 = System.currentTimeMillis();
test();
long t1 = System.currentTimeMillis();
System.out.println((long)ops / (t1 - t0) + " ops/ms");
}
public static void generateInput(int size) {
input = new byte[size];
for (int i = 0; i < size; i++) {
input[i] = new byte[1024];
r.nextBytes(input[i]);
}
}
public static void test() {
for (int i = 0; i < input.length; i++) {
digest(input[i]);
}
}
public static void digest(byte [] data) {
algorithm.update(data);
algorithm.digest();
ops++;
}
}

It is also of the utmost importance to collect a large amount of statistics before drawing any conclusions from a benchmark. The simplest way to do this is to repeat the measurements many times—do multiple benchmark runs. This way, a better grasp of the standard deviations in benchmark results can be obtained. Relevant benchmark scores can only be recorded if the size of deviation for the score in your benchmarking setup is known.

If possible, multiple runs should also be spread over multiple equivalent machines. That way, configuration errors can be discovered and removed from the data. For example, a forgotten load generator may be running on one of the benchmarking machines, contributing to lower scores. If all runs take place on that machine, erroneous measurements will be recorded.

Micro benchmarks are benchmarks that contain a very small amount of code and only measure a small piece of functionality, for example, how quickly the JVM multiplies instances of java.math.BigInteger or how quickly it does AES encryption. Micro benchmarks are simple to write and often require just a single or a few function calls that contain the algorithmic payload.

If any large application could be reduced to a number of micro benchmarks, or at least "mini benchmarks", life would be easier. Sadly, for modern complex applications this is rarely the case. However, almost always, a fair (but for industrial purposes, incomplete) amount of micro benchmarks can be created from the understanding gained by profiling an application and figuring out what it does. The following code is a function from a real world micro benchmark that is used as a performance regression test for the JRockit JVM:

public Result testArrayClear(Loop loop, boolean validate) {
long count = 0;
OpsPerMillis t = new OpsPerMillis("testArrayClear");
t.start();
loop.start();
while (!loop.done()) {
int[] intArray = new int[ARRAYSIZE];
System.arraycopy(this.sourceIntArray, 0, intArray, 0, ARRAYSIZE);
//Introduce side effects:
//This call prevents dead code elimination
//from removing the entire allocation.
escape(intArray);
count++;
}
t.end();
return new OpsPerMillis(count, t.elapsed());
}

Java requires that objects are cleared on allocation, so that all their fields are initiated to their default values. The code optimizer in JRockit should be able to detect that the newly allocated array intArray is immediately and completely overwritten by data, and so, as it is non-volatile, need not be cleared. If, for some reason, this code optimization starts to fail, this benchmark will take longer to complete, and the QA infrastructure will trigger a warning while examining the result database.

Micro benchmarks can (and should) also be created from scratch for problems that are known to be performance-critical in an application. For example for an XML parser, it makes sense to create a small benchmark that operates on a set of auto-generated XML files of different sizes. For a math package that wants to use java.math.BigDecimal, it makes sense to write a couple of small self-contained applications that operate on BigDecimal instances.

Creating micro benchmarks that aren't valid or that don't produce useful results for the problem set is not just a waste of time and effort, but it is also potentially harmful if the benchmark is believed to accurately measure all important aspects of a problem. For example, testing an implementation of java.util.HashMap by just creating a HashMap and filling it up with data might not be good enough. How long does rehashing take? Extracting elements? What about collisions in HashMaps of different sizes?

Similarly, testing a java.math.BigDecimal implementation by just performing a large number of additions is almost certainly not good enough. What if there is a fatal performance flaw in the division algorithm?

While the previous two examples might seem somewhat artificial, they are still examples of the kind of thinking that can lead you astray when creating benchmarks. A somewhat more relevant example might be the case of benchmarking an important synchronized operation in a class library. If the lock in the synchronized operation is contended in an application, this obviously won't show up in a single threaded micro benchmark. It may seem trivial that reducing the load from many to fewer threads fundamentally changes the lock behavior, but this is one of the main reasons that benchmarks fail to represent the real workload. Make sure that any lock in a benchmark is actually stressed by many threads, if this is the case in the real application.

Finally, it might make sense to try to eliminate parts of the runtime that aren't relevant to the problem from the benchmark. If you want to measure pure code performance for some algorithm, it probably makes little sense to create large numbers of objects and stress the garbage collector at the same time. Or at least, pick a good garbage collection strategy that won't interfere too much with the execution of the algorithm (large heap, no nursery, and optimized for throughput).

public class BadMicro {
public static void main(String args[]) {
long t0 = System.currentTimeMillis();
for (int i = 0; i < 1000000; i++) {
// complex benchmarking operation
}
long t1 = System.currentTimeMillis();
System.out.println("Time: " + (t1 - t0) + " ms");
}
}

import java.util.Random;
public class AnotherBadMicro {
static Random r = new Random();
static int sum;
public static void main(String args[]) {
long t0 = System.currentTimeMillis();
int s = 0;
for (int i = 0; i < 1000; i++) {
s += r.nextInt();
}
sum = s;
long t1 = System.currentTimeMillis();
System.out.println("Time: " + (t1 - t0) + " ms");
}
}

Different VMs use different optimization heuristics that trigger at different times. So, it might be possible to improve the quality of measurements and decrease deviation by doing a small amount of "dry runs" or warm-up rounds before starting the actual measurements. The warm-up gives the VM a chance to retrieve runtime feedback from the executing code and perform optimizations in relevant places, before measurements start. This ensures that the measurements are done in a steady optimized state.

Many industry standard benchmarks, such as the SPECjvm2008 suite mentioned later in this chapter, come with warm-up rounds built into the benchmark.