The purpose of benchmarking is to measure a code's performance. The Go test command-line tool comes with support for the automated generation and measurement of benchmark metrics. Similar to unit tests, the test tool uses benchmark functions to specify what portion of the code to measure. The benchmark function uses the following function naming pattern and signature:
func Benchmark<Name>(*testing.B)
Benchmark functions are expected to have names that start with benchmark and accept a pointer value of type *testing.B. The following shows a function that benchmarks the Add method for type SimpleVector (introduced earlier):
import (
"math/rand"
"testing"
"time"
)
...
func BenchmarkVectorAdd(b *testing.B) {
r := rand.New(rand.NewSource(time.Now().UnixNano()))
for i := 0; i < b.N; i++ {
v1 := New(r.Float64(), r.Float64())
v2 := New(r.Float64(), r.Float64())
v1.Add(v2)
}
}
golang.fyi/ch12/vector/vec_bench_test.go
Go's test runtime invokes the benchmark functions by injecting pointer *testing.B as a parameter. That value defines methods for interacting with the benchmark framework such as logging, failure-signaling, and other functionalities similar to type testing.T. Type testing.B also offers additional benchmark-specific elements, including an integer field N. It is intended to be the number of iterations that the benchmark function should use for effective measurements.
The code being benchmarked should be placed within a for loop bounded by N, as illustrated in the previous example. For the benchmark to be effective, there should be no variances in the size of the input for each iteration of the loop. For instance, in the preceding benchmark, each iteration always uses a vector of size 2 (while the actual values of the vectors are randomized).
Benchmark functions are not executed unless the test command-line tool receives the flag -bench. The following command runs all the benchmarks functions in the current package:
$> go test -bench=. PASS BenchmarkVectorAdd-2 2000000 761 ns/op BenchmarkVectorSub-2 2000000 788 ns/op BenchmarkVectorScale-2 5000000 269 ns/op BenchmarkVectorMag-2 5000000 243 ns/op BenchmarkVectorUnit-2 3000000 507 ns/op BenchmarkVectorDotProd-2 3000000 549 ns/op BenchmarkVectorAngle-2 2000000 659 ns/op ok github.com/vladimirvivien/learning-go/ch12/vector 14.123s
Before dissecting the benchmark result, let us understand the previously issued command. The go test -bench=. command first executes all the test functions in the package followed by all the benchmark functions (you can verify this by adding the verbose flag -v to the command).
Similar to the -run flag, the -bench flag specifies a regular expression used to select the benchmark functions that get executed. The -bench=. flag matches the name of all benchmark functions, as shown in the previous example. The following, however, only runs benchmark functions that contain the pattern "VectorA" in their names. This includes the BenchmarkVectroAngle() and BenchmarkVectorAngle() functions:
$> go test -bench="VectorA" PASS BenchmarkVectorAdd-2 2000000 764 ns/op BenchmarkVectorAngle-2 2000000 665 ns/op ok github.com/vladimirvivien/learning-go/ch12/vector 4.396s
As mentioned previously, when benchmarks are executed, the test tool will also run all test functions. This may be undesirable, especially if you have a large number of tests in your package. A simple way to skip the test functions during benchmark execution is to set the -run flag to a value that matches no test functions, as shown in the following:
> go test -bench=. -run=NONE -v PASS BenchmarkVectorAdd-2 2000000 791 ns/op BenchmarkVectorSub-2 2000000 777 ns/op ... BenchmarkVectorAngle-2 2000000 653 ns/op ok github.com/vladimirvivien/learning-go/ch12/vector 14.069s
The previous command only executes benchmark functions, as shown by the partial verbose output. The value of the -run flag is completely arbitrary and can be set to any value that will cause it to skip the execution of test functions.
Unlike tests, a benchmark report is always verbose and displays several columns of metrics, as shown in the following:
$> go test -run=NONE -bench="Add|Sub|Scale" PASS BenchmarkVectorAdd-2 2000000 800 ns/op BenchmarkVectorSub-2 2000000 798 ns/op BenchmarkVectorScale-2 5000000 266 ns/op ok github.com/vladimirvivien/learning-go/ch12/vector 6.473s
The first column contains the names of the benchmark functions, with each name suffixed with a number that reflects the value of GOMAXPROCS, which can be set at test time using the -cpu flag (relevant for running benchmarks in parallel).
The next column displays the number of iterations for each benchmark loop. For instance, in the previous report, the first two benchmark functions looped 2 million times, while the final benchmark function iterated 5 million times. The last column of the report shows the average time it takes to execute the tested function. For instance, the 5 million calls to the Scale method executed in benchmark function BenchmarkVectorScale took on average 266 nanoseconds to complete.
By default, the test framework gradually adjusts N to be large enough to arrive at stable and meaningful metrics over a period of one second. You cannot change N directly. However, you can use flag -benchtime to specify a benchmark run time and thus influence the number of iterations during a benchmark. For instance, the following runs the benchmark for a period of 5 seconds:
> go test -run=Bench -bench="Add|Sub|Scale" -benchtime 5s PASS BenchmarkVectorAdd-2 10000000 784 ns/op BenchmarkVectorSub-2 10000000 810 ns/op BenchmarkVectorScale-2 30000000 265 ns/op ok github.com/vladimirvivien/learning-go/ch12/vector 25.877s
Notice that even though there is a drastic jump in the number iterations (factor of five or more) for each benchmark, the average performance time for each benchmark function remains reasonably consistent. This information provides valuable insight into the performance of your code. It is a great way to observe the impact of code or load changes on performance, as discussed in the following section.
Another useful aspect of benchmarking code is to compare the performance of different algorithms that implement similar functionalities. Exercising the algorithms using performance benchmarks will indicate which of the implementations may be more compute- and memory-efficient.
For instance, two vectors are said to be equal if they have the same magnitude and same direction (or have an angle value of zero between them). We can implement this definition using the following source snippet:
const zero = 1.0e-7
...
func (v SimpleVector) Eq(other Vector) bool {
ang := v.Angle(other)
if math.IsNaN(ang) {
return v.Mag() == other.Mag()
}
return v.Mag() == other.Mag() && ang <= zero
}
golang.fyi/ch12/vector/vec.go
When the preceding method is benchmarked, it yields to the following result. Each of its 3 million iterations takes an average of half a millisecond to run:
$> go test -run=Bench -bench=Equal1 PASS BenchmarkVectorEqual1-2 3000000 454 ns/op ok github.com/vladimirvivien/learning-go/ch12/vector 1.849s
The benchmark result is not bad, especially when compared to the other benchmarked methods that we saw earlier. However, suppose we want to improve on the performance of the Eq method (maybe because it is a critical part of a program). We can use the -benchmem flag to get additional information about the benchmarked test:
$> go test -run=bench -bench=Equal1 -benchmem PASS BenchmarkVectorEqual1-2 3000000 474 ns/op 48 B/op 2 allocs/op
The -benchmem flag causes the test tool to reveal two additional columns, which provide memory allocation metrics, as shown in the previous output. We see that the Eq method allocates a total of 48 bytes, with two allocations calls per operation.
This does not tell us much until we have something else to compare it to. Fortunately, there is another equality algorithm that we can try. It is based on the fact that two vectors are also equal if they have the same number of elements and each element is equal. This definition can be implemented by traversing the vector and comparing its elements, as is done in the following code:
func (v SimpleVector) Eq2(other Vector) bool {
v.assertLenMatch(other)
otherVec := other.(SimpleVector)
for i, val := range v {
if val != otherVec[i] {
return false
}
}
return true
}
golang.fyi/ch12/vector/vec.go
Now let us benchmark the Eq and Eq2 equality methods to see which is more performant, as done in the following:
$> go test -run=bench -bench=Equal -benchmem PASS BenchmarkVectorEqual1-2 3000000 447 ns/op 48 B/op 2 allocs/op BenchmarkVectorEqual2-2 5000000 265 ns/op 32 B/op 1 allocs/op
According to the benchmark report, method Eq2 is more performant of the two equality methods. It runs in about half the time of the original method, with considerably less memory allocated. Since both benchmarks run with similar input data, we can confidently say the second method is a better choice than the first.
This discussion only scratches the surface of comparative benchmarks. For instance, the previous benchmark tests use the same size input. Sometimes it is useful to observe the change in performance as the input size changes. We could have compared the performance profile of the equality method as we change the size of the input, say, from 3, 10, 20, or 30 elements. If the algorithm is sensitive size, expanding the benchmark using such attributes will reveal any bottlenecks.