Chapter 1
ScalaCheck: Property-based Testing
What sets ScalaCheck apart from numerous mainstream testing tools is its core idea, the concept of property-based testing. This chapter explains that concept and highlights its advantages over traditional testing.
To make the concept as clear as possible, ScalaCheck is compared directly to JUnit, one of the most commonly used unit-testing tool for Java. This chapter uses small examples and code snippets to demonstrate this comparison, and the next chapter gives you a complete testing example, with implementations both in ScalaCheck and in JUnit.
1.1 Traditional versus property-based unit testing
The difference between property-based testing in ScalaCheck and the more traditional testing in the vein of JUnit is closely related to the difference between specifications and tests. Hence, let's start by sorting these two concepts out.
A specification is a definition of a program's behavior in the general case. Preferably, a specification should not deal with concrete examples of program functionality. Instead it should provide an abstract description of the behavior, that you can use both when implementing the program and when verifying it. You commonly verify a program by testing it, which means you give the program various inputs and make sure the program behavior follows the specification. Such verification can never be complete since the tests only cover specific examples of the program's operation. There are also methods for formally proving that a program adheres to a specification, but that is hard to do for all types of programs and all types of specifications. The fact that most specifications are written in an informal way (usually it is a document written in plain English) makes it even harder to form any sort of formal proofs around them.
Tests are concrete examples of how a program should behave in particular situations. A unit test runs a small part of a program (often a class method) with a predefined input, and asserts that the method gives the correct output and manipulates the program state correctly for that given input. The JUnit framework makes it easy to organize many unit tests and to run the tests automatically and repeatedly. Most unit tests can be considered formal due to the fact that you can let a computer execute your tests and decide whether they pass or not. Of course, there are also informal tests—for example, usability tests or performance tests. Such tests require human assistance for deciding whether the program behaves correctly or not.
We can see that specifications and tests are related, but there is a significant difference in generality between the two concepts. Often there is also a formality difference. Tests are mostly formal and easy to verify, but they don't give us assurance about the overall program behavior, just about small concrete cases here and there. Specifications, on the other hand, give us a more complete picture of the program but they are often informal and difficult to verify formally. Instead, tests are used to verify specific cases of a specification.
Tests as specification is a concept that is gaining popularity in many test-driven software development processes. The idea is that if you are rigid enough about testing and write tests for covering most parts of your implementation. You should then be able to put less focus on your informal specification documents and instead start considering the tests themselves a specification for your program. This has the obvious advantage of making the specification easy to verify: you just run all the tests and make sure they pass. This is sometimes called an executable specification. The downside is that by the very nature of tests, such a specification will not be complete. The specification completeness will solely depend on the number of tests that you are willing to define and maintain.
Property-based testing, as implemented in ScalaCheck, is similar to the concept of tests as specification, in the sense that it results in a specification that you can verify cheaply. However, while the idea of tests as specification is to make your specification more test-centered, property-based testing goes in the opposite direction by making your tests more specification-like. It does so by generalizing tests to properties. Properties are just as central in ScalaCheck as tests are in JUnit. I will spend the next section investigating the concept of properties.
Tests versus properties
A test is a concrete example of how a program should behave in a particular situation, while a property is an abstract, general specification. Let us look at examples to try to make the situation clear. Imagine you want to make sure that the max method of the java.lang.Math class is implemented correctly. If you were to write a test for that method using JUnit 3, it could look something like Listing 1.1.
import junit.framework.TestCase;
public class MathTest extends TestCase {
public void testMax() { int z = Math.max(1,2) assertEquals(2, z) }
}
Listing 1.1 - Testing the java.lang.Math.max method with JUnit.
You would probably add more tests to cover more situations; perhaps you settle for something like Listing 1.2, covering a reasonable number of input combinations.
import junit.framework.TestCase;
public class MathTest extends TestCase {
public void testMax2() { int z = Math.max(1,0) assertEquals(1, z) }
public void testMax3() { int z = Math.max(10,10) assertEquals(10, z) }
public void testMax4() { int z = Math.max(-2,0) assertEquals(0, z) }
}
Listing 1.2 - A more complete JUnit test for java.lang.Math.max.
This is a straightforward approach for verifying the Math.max method. Even if you didn't know how the method behaved, you could look at these tests and figure that out quickly. Still, the tests don't really describe the method's behavior; they merely give us a number of usage examples. A property, on the other hand, will give us a more general description of the behavior. Listing 1.3 shows an example of how a ScalaCheck property that specifies the max method might look.
import org.scalacheck.Properties import org.scalacheck.Prop.forAll
object MathProps extends Properties("Math") {
property("max") = forAll { (x: Int, y: Int) => val z = java.lang.Math.max(x, y) (z == x || z == y) && (z >= x && z >= y) }
}
Listing 1.3 - A ScalaCheck property that verifies java.lang.Math.max.
You should be able to understand the example in Listing 1.3 even if you don't have much experience with Scala. In the next chapter, I will go into more details, but for now, focus on the contents of the call to forAll. As you can see, we don't deal with any concrete integer values here as we did in the tests. Instead, we have two integers, x and y, ignoring their concrete values. We then use the two integers in a call to the Math.max method, and finally we verify the result with a boolean expression. The boolean expression states that the result of the call to Math.max must be equal to one of the input parameters (the max method is not allowed to make up an integer value), and that it must be larger than or equal to both of the input values. Those conditions define the method completely.
What we feed into the forAll method is an anonymous function, which takes two integer values as parameters and returns a boolean value as a result. Anonymous functions are a powerful feature of the Scala language.
Properties are sometimes called parameterized tests, which you can see if you picture the abstract parameters x and y replacing the concrete integer values. When we do that, we also replace the concrete equality check used in the tests with a general logic expression that should hold for all values of x and y.
You can see that this property is a much better description of what the max method does than the collection of test cases I defined earlier. You can read the property and get a complete definition of the behavior, rather than a set of examples that only indicates the method's behavior.
There is not much more to property-based testing than this. Simply replace a set of concrete test cases with one abstract property that describes a code unit's behavior. ScalaCheck will then take your abstract property and turn it into possibly thousands of concrete test cases by generating randomized data for the property's parameters. It will run each test in an attempt to falsify the property, and only if each test for a given property passes will ScalaCheck regard the property as true.
This is a brief introduction and explanation of property-based testing, and I think the next chapter illustrates the concept more clearly, with a complete example of how you can use ScalaCheck to test a small library of string routines. Before moving along, the remainder of this chapter enumerates some of the selling points of property-based testing in general and ScalaCheck in particular.
Remember also that everything that is offered by traditional, example-based tests still is viable in a property-based setting. Since properties just are a generalization of tests, you can write properties that make use of concrete tests instead of, or in combination with, ones that are generated by ScalaCheck. Property-based testing simply gives you new ways of expressing expectations on your code, it doesn't remove anything you are used to from example-based testing.
1.2 Benefits from property-based testing
There are several benefits you gain from property-based testing over traditional unit testing, and an overall goal of this book is to demonstrate that. What follows here is a condensed overview of some of the benefits.
Test coverage can increase since test cases are generated in a random fashion, and the code is tested with many more cases. Of course, if you write manual JUnit tests carefully you can get good coverage too, but ScalaCheck makes it harder to miss out on many cases. Since you can control the distribution of generated test cases in ScalaCheck and collect statistics on the kind of data that has been used, you can get reliable test coverage if you write your properties with care. Also, ScalaCheck's testing is not completely randomized: by default, when coming up with integers or lists or other basic inputs, it makes sure to always include common edge cases. It includes zero, one, maximum and minimum integers, and empty lists. Such cases can trigger unexpected behavior in code but are easy to miss if you write your test cases manually.
Specification completeness is easier to accomplish with ScalaCheck than with JUnit. Manually writing test cases almost always results in an approximation of code behavior. In ScalaCheck, you often have a chance to define exactly how your code should behave. This can be useful not only because it enables better testing, but also because it forces you to reflect about your code's exact behavior. It makes it harder to skip over edge cases in your tests, in your implementation, and in your thinking. Of course, you can write properties in ScalaCheck that only cover part of your implementation, too. Sometimes it's not worthwhile to specify a method completely, because it would mean the property would be just as complex as the implementation.
Maintenance is another point where ScalaCheck has an edge over traditional unit testing. Since each property represents a whole set of tests, code size and repetition decreases dramatically in many cases. This has great impact on code maintainability and the cost of refactoring. One of the benefits of unit testing in general is that it gives you confidence to refactor your code often. However, unit tests can be an obstruction when you need to make changes that affect many test cases. Cutting down on testing code means less code to maintain and refactor. Additionally, testing code is sometimes low quality compared to other parts of a project. You often copy and paste tests, making small modifications for varying cases. Eliminating duplicated code in tests improves overall code quality.
Test readability is a matter of personal taste, but I often find it easier to read and understand one concise property rather than go through several similar test cases to find out the intended behavior of a given code unit. A property can become complex and hard to grasp if it contains many conditions and long-winded logic. However, ScalaCheck has functions for combining properties and lots of high-level helper methods that make it easier to write clear properties.
Test case simplification is a powerful feature of ScalaCheck. It is enabled by the fact that properties are abstract, and ScalaCheck therefore has control over the test data that is used. As soon as ScalaCheck finds a set of arguments that makes a property false, it tries to simplify those arguments. For example, if a property takes a list of integers as its parameter, then ScalaCheck will first generate many different integer lists and feed them to the property. If it stumbles across a list that makes the property false, ScalaCheck will test the property with smaller and smaller variants of that list, as long as the property still fails. Then ScalaCheck prints both the smallest list that still causes property failure, and the original list it stumbled across. By default, the first generated parameter is called ARG_0. A typical scenario could look like this:
scala> p.check ! Falsified after 23 passed tests. > ARG_0: List("-1", "-1") > ARG_0_ORIGINAL: List("935472295", "1", "-1", "-1", "2147483647", "2113129492", "2147483647")
scala> p.check ! Falsified after 3 passed tests. > ARG_0: List("1", "1") > ARG_0_ORIGINAL: List("0", "1", "1", "1", "2147483647")
Here, we can see the output from two runs of ScalaCheck for some property p. We can see the simplified and original property arguments for each run, and we can come to the conclusion that lists with duplicate integers cause the property to fail. If ScalaCheck hadn't simplified the arguments then this would not have been as obvious. You will see more examples of test case simplification in Chapter 3 and Chapter 6.
1.3 Conclusion
This chapter has given you a brief theoretical background to property-based testing, as well as an overview of its main benefits. The next chapter will show a larger example, comparing ScalaCheck directly to JUnit to make the difference between property-based testing and traditional, example-based testing clearer. The rest of the book will then focus on ScalaCheck and how to use it to do property-based testing.