Takeaways

There’s only one way to prove that the optimization worked. You measure the performance before and after, and you compare. But the devil is in the details. Here’s what you need to take care of to get the measurements right.

1. Minimize external factors to increase measurement accuracy.

2. Make sure that GC behaves as predictably as possible to decrease variability in measurements.

3. Take as many measurements as practical to make statistical analysis possible. A good default is 30.

4. Compare before and after numbers by calculating the confidence interval of the optimization effect. Conclude that optimization worked only when the lower bound of the confidence interval is higher than 0.

5. Try to reduce dispersion in measurements as much as possible. Otherwise even with statistical tools you won’t be able to tell whether or not you optimized the code.

Now we know how to do measurements, and how to compare them. But the goal of optimization is not to measure it, nor even to make sure that the optimized code indeed runs faster.

The real goal is to optimize and to make sure the slowdown never happens again. How can you do that? After optimization you’ll need to measure the performance after every change, and detect even the smallest regressions from the achieved performance level.

If that smells like testing, you’re right: it is testing. Performance testing. And that’s exactly what we’ll talk about in the next chapter.