Minimize External Factors

It’s obvious that before measuring performance you need to free up memory, ensure that nothing else runs in parallel, and that no disk I/O happens. But there are a few things you might want to do beyond that.

Disable Dynamic CPU Frequency Scaling

The first thing to consider is disabling dynamic CPU frequency scaling, not because scaling makes your program run slower, but because two separate program runs may be incomparable because they’re executed with the CPU capped at different frequencies. That’s why it’s a good idea to disable CPU frequency scaling if you can.

On Linux, the governor defines the current CPU frequency. By default the ondemand governor is used, meaning that CPU speed will change dynamically depending on the load. You can check which governor your system runs with the cpupower tool.

These are the frequency stats for my laptop. It doesn’t have much to do while I’m just typing this text, so the ondemand governor scales my CPU down to a leisurely 1.20 GHz.

If you’re on Linux, you can ask the cpupower tool to set the performance governor that will force CPU to its maximum frequency.

Windows has an analog to Linux’s governor called the power plan. The default power plan is Balanced, similar to ondemand on Linux. You can change it to High performance in the Windows Control Panel.^[21]

There’s no easy way to disable CPU power management features on Mac OS. You might have to Google. There are solutions that work for some people.

Disabling dynamic CPU frequency will make your measurements more deterministic, but don’t spend too much time on it if none of the tools and tricks I’ve described work for you. A bit of statistical analysis will help you make sense out of your measurements anyway.

Warm It Up

Another thing you should be aware of is a “cold” state. Alternative Ruby implementations running on a virtual machine (VM), like JRuby and Rubinius, require warm-up before they can produce their best results. If you run one of these, be sure to estimate how many times you need to execute the same code before the VM warms up. This number depends on code complexity, and can be as small as 10 times and as large as 10,000 times.

Even if you’re running MRI, you can still run into the cold state problems with third-party software. Let’s take, for example, the PostgreSQL database server.

PostgreSQL relies on the filesystem cache to keep data in memory after first access. Given enough memory, all data you ever touch will eventually end up being cached. We all know that disk I/O is slow. This means that your Ruby application performance will be radically different depending on whether or not your data is cached.

Unlike in the previous case with alternative Rubys, here you should be really interested in both cold and warm state performance. There’s no guarantee your data will always remain cached, so you must measure both states.

To get to the warm state you can just execute the same code twice. But getting a predictable cold state requires cleaning the filesystem cache. Here’s the trick that works on Linux: