Benchmarking Your Code

No discussion about performance would be complete (or to my way of thinking, even worth beginning) without that famous quote from Donald Knuth [Knu74]:

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.”

If we apply our time-limited efforts randomly around the codebase, it may not improve any bottlenecks that exist. My experience tends to align with Knuth’s advice here: it saves us time, and a lot of wasted effort, to first find problem areas in our programs. If our program doesn’t need to be fast, any performance work is unnecessary, or at least, premature. If our web application spends the majority of its time on I/O tasks, such as transferring data from the database to the application itself and pushing it out to the user’s web browser, any speed improvements we make will not noticeably impact users. When we make performance improvements, it’s critical we have metrics showing that we made an impact.

Happily, there’s a wonderful benchmarking tool that can help us make decisions about what parts of our code to optimize and how to optimize them. Hugo Duncan’s Criterium^[16] is the benchmarking library of choice for many Clojure developers, and with good reason. Getting started with Criterium is pretty easy, since it’s just a matter of adding it to your Leiningen dependencies. Criterium runs your code many times in order to take JIT and garbage collection into account. It also collects statistics on the running times to provide more detailed information, such as standard deviation and outliers. (Don’t worry about the cryptic :jvm-opts ^:replace [] line—it just enables some JVM JIT optimizations.)

As an aside, the optimizations result in a slightly longer startup time, but it’s worth taking that extra time in performance testing and other real-world production scenarios.

Criterium actually uses macros internally for some of the same reasons we discussed in Chapter 4, ​Evaluate Code in Context​, but here we just want to measure performance improvements. Let’s pop open a REPL inside a project with project.clj and see how to use Criterium to measure the code’s speed before optimizing:

The bench macro executes the given expression several times, so it may take a while before we see the output. We can wrap it with (with-progress-reporting ...) to see more feedback while the benchmarking system is running. From here on out, we’ll only show the mean and standard deviation times, but you can try them out in your REPL if you want statistics on outliers and other details. If you have some logs to analyze, why not run that through Criterium and see how it goes?

A few microseconds might not sound like a lot, but when you’re in the middle of a graph search algorithm or some other CPU-intensive task, it may be a significant bottleneck. We can pretty easily knock off some time by avoiding reflection (a Java mechanism that determines dynamically how a method will dispatch at runtime). Let’s see what happens when we add a type hint (^String) to the argument to length-1, or even just call .length on the literal string directly, since these cases don’t require reflection:

Figuring out where to put type hints to avoid reflection can be a little tricky, and since avoiding reflection is necessary for high-performance Clojure code, it’s nice to have a hook to warn us about it: (set! *warn-on-reflection* true). If we set that var to true, we’ll get a helpful warning letting us know that the code we just wrote will need to use reflection:

There are all kinds of other tools to see how your Clojure code is performing, from profilers like JVisualVM^[17] (included in the Oracle JDK distribution) to disassemblers like no.disassemble.^[18] These tools are there to help you pick the right problems to solve, so be sure to use them! You don’t want to waste your valuable time optimizing code that’s already fast enough.