Let's first start by thanking the great work of Lloyd Chan and Sunjay Varma, who were the first to suggest this approach. You can find the code we will be using in Sunjay's blog (http://sunjay.ca/2017/04/27/rust-benchmark-comparison-travis). Nevertheless, it makes sense to check it, understand it, and see how it works.

The idea is simple: on Travis-CI builds, you can build against multiple Rust channels. When a pull request is received when building against the nightly channel, let's run all the benchmarks and then compare them to benchmarks we will run on the pull request target branch. Finally, output the comparison results in Travis-CI's build logs.

Let's start by configuring our Travis-CI build script. For that, we will need a .travis.yml file similar to the following one in our repository:

language: rust
dist: trusty # Use a little more updated system
os:
  - linux # Build for Linux
  - osx # Build also for MacOS X

# Run builds for all the supported trains
rust:
  - nightly
  - beta
  - stable
  - 1.16.0 # Minimum supported version

# Load travis-cargo
before_script:
  - export PATH=$PATH:~/.cargo/bin

# The main build
script:
  - cargo build
  - cargo package
  - cargo test

after_success:
  # Benchmarks
  - ./travis-after-success.sh

Let's see what this code does. First of all, if you never used Travis-CI for your continuous integration, you should know that the .travis.yml YAML file contains the build configuration. In this case, we tell Travis-CI that we want to build a Rust project (so that it sets up the compiler by itself) and we tell it that we want to build against nightly, beta, and stable release channels. I usually like to add the minimum supported Rust version, mostly to know when it breaks, so that we can advertise the minimum Rust compiler version in our documentation.

We then export the cargo binary path so that we can add cargo binaries by installing them in the build. This will be needed for the benchmark comparison script. Then, we tell Travis-CI to build the library/binary crate, we tell it to package it to check that a valid package will be generated, and we finally run all the unit tests. So far, nothing too different from a normal Travis-CI Rust build.

Things change once we get to the after-success section. We call a shell script that we haven't defined yet. This script will contain the logic of the benchmark comparison.

Before writing all the code, let's first learn about a library that will make things much easier for us. I'm talking about the cargo-benchcmp, cargo binary. This executable can read outputs from Rust benchmarks and compare them. To install it, you only need to run cargo install cargo-benchcmp. It also has some great command-line arguments that can help us get the output we want.

To get the results of a benchmark in a file, it's as simple as doing cargo bench > file. In this case, we will have two benchmarks, the control benchmark, a benchmark that we decide will be the reference; and a variable benchmark, the one we want to compare. Usually, a pull request will have the target branch as a control benchmark and the pull request branch as a variable benchmark.

Using the executable is as easy as running cargo benchcmp control variable. This will show a great output with a side-by-side comparison. You can ask the tool to filter the output a bit, since you probably don't want to see tens of benchmarks with really similar values, and you are probably interested in big improvements or regressions.

To see the improvements, add the --improvements flag to the command line and, to see the regressions, add the --regressions flag. You can also set up a threshold as a percentage, and benchmarks that change below that threshold won't show, to avoid non-changing benchmarks. For that, use the --threshold {th} syntax, where {th} is a number higher than 0 representing the percentage change that should be taken into account.

Now we understand this, let's see the code that will be in the travis-after-success.sh file:

#!/usr/bin/env bash

set -e
set -x

if [ "${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}" != "master" ] && [ "$TRAVIS_RUST_VERSION" == "nightly" ]; then
    REMOTE_URL="$(git config --get remote.origin.url)"

    # Clone the repository fresh...
    cd ${TRAVIS_BUILD_DIR}/..
    git clone ${REMOTE_URL} "${TRAVIS_REPO_SLUG}-bench"
    cd "${TRAVIS_REPO_SLUG}-bench"

    # Bench the pull request base or master
    if [ -n "$TRAVIS_PULL_REQUEST_BRANCH" ]; then
      git checkout -f "$TRAVIS_BRANCH"
    else # this is a push build
      git checkout -f master
    fi
    cargo bench --verbose | tee previous-benchmark
    # Bench the current commit that was pushed
    git checkout -f "${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}"
    cargo bench --verbose | tee current-benchmark

    cargo install --force cargo-benchcmp
    cargo benchcmp previous-benchmark current-benchmark
   fi

Let's see what this script is doing. The set -e and set -x commands will simply improve how the commands are shown in Travis-CI build logs. Then, only for nightly, it will clone the repository in a new location. If it's a pull request, it will clone the base branch; if not, it will clone the master branch. Then, it will run benchmarks in both places and compare them using cargo-benchcmp. This will show the results in the build logs.

This script can, of course, be modified to suit any needs and, for example, use a different branch to the master branch as a default branch, or filter the output of the comparison, as we saw earlier.