Make Internals Behave Predictably

Two things in our applications can randomly slow it down: GC and system calls. (I/O is the most common case of a system call.)

I can imaging you sighing, “Oh no, not GC again.” But in this case it’s easy to neutralize its effect on measurements, in one of two ways.

First, it’s sometimes OK to disable GC completely before measurement. If raw performance of our code is what we’re interested in, then it’s absolutely fine to do that.

If we want our measurement to be closer to the real world, we can try to make GC as predictable as possible by forcing it before executing the code, like this:

This way our code will trigger GC in the same manner every time we run the Benchmark.realtime block.

But beware of the pitfall here. We’ll have to run that code several times in order to obtain the proper measurement. If we do that by restarting the application, we’ll be fine. But in a loop, as in the following code snippet, this will result in an incorrect measurement.

Why would the measurements be incorrect even though we run GC.start before each benchmark block? Because our code will allocate memory for either Ruby objects or their data. So GC will have to deal with more objects on the Ruby heap and more data on the system heap. This way, after each iteration GC time will tend to increase, thus increasing the execution time that we measure.

On Linux, Mac OS X, or any other Unix-based system, we can fix it by forking the code before the measurement:

Forking ensures that any memory allocations happen in child processes only and don’t affect the parent application. This way, every child process will start in the same state, and any GC in the child will happen at (more or less) the same time.

To minimize the effect of I/O and other system calls, simply make sure the operating system is not busy with anything else and run the code several times to cache the files that you read.