Introducing Randomness and Property-Based Testing

We can solve some of the problems that example-based tests suffer from by introducing a bit of chaos in our tests. Using randomness to generate inputs will allow us to test a wider range of inputs against our code and potentially create inputs that trigger edge cases.

In our sorting example, what we really want to test is that the output of Enum.sort/1 is a sorted version of the input. For any random input, we can think of a few properties that the output will always retain. For example, the output list always has the same length as the input list. Another property is that the output list is always sorted, which is something that we can check in a pretty straightforward way by checking that each element is smaller than or equal to the following one. Now that we’ve thought of these properties, we could change our test so that we generate random lists and test these properties on the output of our code instead of checking what the output is. Let’s see how to do that:

The random_list/0 function creates an infinite stream of random numbers between -100 and 100 and then picks a random number of elements from the stream using Enum.take/2. The number of elements we pick from the stream is the length of the random list, which we keep between 0 and 10 elements. The sorted?/1 function checks that the first two elements of the list are sorted and then recursively checks the rest of the list until it arrives at an empty or one-element list, which is always sorted. On line 9, we check our first property, that the sorted list has the same number of elements as the input list. On line 10, we check the second property, that the sorted list is sorted.

This approach to testing has a few benefits. One of the most obvious is that it can potentially test on a lot more inputs than example-based testing can. In our example, if we want to change the number of tested lists to a hundred or a thousand, we can just change the right end of the range on line 5. However, the usefulness of testing on many inputs is limited unless the inputs vary.

The Role of Randomness

This is where randomness comes into play. By having a lot of inputs generated at random, our hope is to cover a decent part of the possible inputs to our code and at the same time cover a good variety of inputs. Essentially, we want a good sample of inputs that represents the input space, which is the set of all possible inputs. In our example, we’re still covering a tiny part of our input space (all lists of numbers), but covering the whole input space is often unfeasible. Random generation gives us a nice compromise, especially considering that every time the tests are run, possibly different lists are generated. Generating random elements also helps us to uncover potential corner cases that we didn’t anticipate.

You might be asking yourself how much randomness is enough, that is, how many inputs you need to generate or how many times you need to run these tests to have confidence that they cover enough of the input space. In many cases, the input space is infinite or too vast to cover, but only you will know how far to push it based on the specific use case.

The test we wrote for Enum.sort/1 is an example of a kind of test called property-based tests. They are called that because of the method we used to come up with this kind of test: we think of properties that our code holds regardless of the input we feed to it, provided the input is valid.

The benefits of property-based testing don’t end with what we’ve just discussed. Coming up with valid inputs and properties is a huge part of property-based testing, but it’s also a helpful design tool. If you have to write down in clear terms what the valid inputs of your code are, you could end up expanding or shrinking the space of valid inputs. Coming up with properties, instead, forces you to think about what your code should do regardless of the specific input you feed to it, which might help with the design or implementation of the code. In the list-sorting example, the functionality is trivial, so it’s hard to see the design benefits of property-based testing; but in more complex contexts, it can be useful to think about these things.

Property-based testing is rarely done in a hand-rolled way like we did in our example, as there’s a plethora of frameworks (for all kinds of programming languages) that facilitate the implementation of property-based tests. Usually, property-based testing frameworks provide powerful ways of generating data and an infrastructure for verifying properties against that generated data. There’s also one important feature that makes using a property-based testing framework a clear advantage over rolling out your own randomness-based tests: frameworks simplify the randomly generated inputs when a failure occurs, and they present error messages that tend to be significantly easier to understand and address than if you handwrite tests with random data like we did.

For Elixir, the property-based testing framework we’re going to use from now on is called stream_data.^[49]

Introducing stream_data

stream_data is a property-based testing framework for Elixir. It provides two main functionalities, data generation and a framework for writing and running properties. The data generation aspect of the library is usable outside of property-based testing as a standalone feature, but it’s the backbone of the whole framework and is also used extensively when writing properties.

To follow along in the next few sections, create a new Mix project with $ mix new sorting and then add :stream_data as a dependency in your mix.exs file:

Now, run $ mix deps.get to fetch the dependency. As you can see in the code, we’ve only added :stream_data in the :test environment since we’ll only be using the library when testing.

Before diving into the framework, let’s rewrite the RandomizedSortTest test we hand-rolled earlier to use the tools that stream_data provides:

Don’t worry about the new things you see in this test. We’ll cover all of them in this chapter. The goal here is to show you what stream_data looks like. For now, run mix test in the project where you added this file and see the beautiful green dots.

As it turns out, the underlying shape of the test is quite similar to RandomizedSortTest. Instead of using the test macro to define a test, we use property (on line 5). Then we use a new construct, check all, on line 6. This replaces the for comprehension we had. On the same line, we have list <- list_of(integer()). That’s exactly one of the most important features of a property-based framework: data generators. Here stream_data takes care of generating random data (with cool characteristics we’ll see later) for you. Now that we have an idea of what a stream_data test looks like, let’s move on to dissecting its components in a more detailed way.

In the next sections, we’re going to start exploring from the data generation aspect of stream_data and then move on to designing and running properties. To follow along, run iex -S mix to fire up an IEx session from the root of the project that includes stream_data as a dependency.

You might be wondering why we won’t illustrate these concepts on one of the applications we developed in the previous chapters (such as Soggy Waffle). Well, the reason is that we would have to bend those applications in weird ways to be able to show these ideas effectively. Instead, we decided to use simple, small, and self-contained examples so that we can focus on property-based testing concepts and tools.