Chapter 7
Measure

So far I’ve avoided talking too much about measurements, despite their obvious importance to optimization. Why? Because in most cases people get it right. Your intuition nudges you to run the code several times, measure results, and pick the most commonly seen measurement.

This is a viable approach, and it should yield meaningful results most of the time. But if you get your measurements wrong, your optimization will go wrong too. You may either miss the small speedup, or falsely believe in ten times optimization where in fact there is none.

So let’s think about what can go wrong with measurements and how to deal with it.

External factors are the first issue that comes to mind. These might be processes running in parallel on the same machine, or unexpected swapping, or even CPU frequency management capping the system performance.

But your application internals can also affect measurements. Earlier we saw that to profile execution time you must turn off GC. Otherwise you’ll see random slowdowns in unexpected places—in the same way disk operations, or other system calls in your code, may lead to nondeterministic results.

Finally, it is impossible to exclude all external and internal factors. So your measurements will vary when repeated. It can be hard in this case to compare sets of before and after optimization numbers because they will be just slightly different or even overlap.

So, let’s talk about all these issues and see what we can do to obtain valid measurements and compare them correctly.