Chapter 5
Properties in Detail
This chapter aims to deepen your knowledge about ScalaCheck's property API. It is made up of two parts. The first part will show you various ways to analyze property failures and investigate what test cases ScalaCheck has generated for your properties. The second part is a systematic overview of ScalaCheck's property combinators.
With these pieces, you can specify detailed expectations of the code under test, provide labels that narrow down problem areas, and gain insight into how your code was tested.
5.1 Investigating property failures
A testing tool can provide great confidence and satisfaction when it reports passed tests. Conversely, when tests fail a good tool pinpoints the cause as quickly as possible. Ideally, your testing tool should not only tell you that a test failed but also why it failed. This section demonstrates through examples several ScalaCheck features that can help you find out why a property failed.
Labeling generated test data
As you've seen in this book's examples, in case of failure ScalaCheck always presents the values it has generated for a property's parameters. This works fine even when you nest calls to forAll like below:
import org.scalacheck.Prop.{forAll, BooleanOperators}
val p = forAll { xs: List[Int] => forAll { n: Int => (n >= 0 && n < xs.length) ==> xs(n) >= 0 } }
scala> p.check ! Falsified after 0 passed tests. > ARG_0: List("-14386367") > ARG_0_ORIGINAL: List("1800647208", "-493891772") > ARG_1: 0
By default, ScalaCheck names the parameters ARG_0, ARG_1, ARG_2, and so forth. The suffix _ORIGINAL is added whenever ScalaCheck has simplified an argument; both variants falsify the property.
You can set a specific label of an argument if you use an explicit generator, and use the |: or :| operator. The pipe should point to the label; the colon should point to the generator. You can either use a String or a Symbol as the label. Here are two examples:
import org.scalacheck.Prop.forAll import org.scalacheck.Gen.{oneOf, choose}
val p = forAll(choose(0,100) :| "pos", choose(-10,0) :| "neg")(_ * _ < 0)
val q = forAll('prime |: oneOf(2,3,5,7)) { prime => prime % 2 != 0 }
scala> p.check ! Falsified after 19 passed tests. > pos: 0 > neg: 0 > neg_ORIGINAL: -84
scala> q.check ! Falsified after 7 passed tests. > prime: 2
Smart equivalence checks
Boolean expressions are common in properties. They can be a bit tricky to debug if you get a property failure, however, since that just means the boolean expression was false. You can do better by replacing the operator == with ?= or =?. If you do that, ScalaCheck will record both the left-hand side and the right-hand side of the boolean expression when evaluating the property. If the property fails, then both values are presented. The value closest to the question mark will be presented as the actual value and the value closest to the equal sign will be presented as the expected one. You have to import Prop.AnyOperators to access these equality operators.
In the example below, a method that interleaves two lists is defined, and then a property states that the length of the resulting list should be the sum of the lengths of the two interleaved lists. To show the effect of the ?= operator, I have introduced a bug in the interleave method:
import org.scalacheck.Prop.{AnyOperators, forAll}
def interleave[T](xs: List[T], ys: List[T]): List[T] = { if(xs.isEmpty) ys else if(ys.isEmpty) xs else xs.head :: ys.head :: interleave(xs.tail,xs.tail) } val propInterleave = forAll { (xs: List[Int], ys: List[Int]) => val res = interleave(xs,ys) res.length ?= (xs.length + ys.length) }
When checking the propInterleave property, we get a nice report on what differs in the equality comparison:
scala> propInterleave.check ! Falsified after 5 passed tests. > Labels of failing property: Expected 3 but got 2 > ARG_0: List("0") > ARG_0_ORIGINAL: List("2061986129") > ARG_1: List("0", "0") > ARG_1_ORIGINAL: List("-2045643334", "-635574522")
ScalaCheck informs us why the property failed: the length of the list was 2 and not 3, as one would expect from the generated parameters. This is because the faulty interleave implementation causes the xs list to be interleaved with itself (except for the first list item) instead of with ys.
Labeling properties
Large and complex properties often contain several smaller parts, making it difficult to know which part is false when the whole property fails. In such cases, ScalaCheck allows you to label the individual parts. Just as for generators, you use the :| or |: operator to achieve this. Let's continue the interleave example, correcting the error we found. We also extend the property so that it defines the interleave method completely, not only the length condition.
import org.scalacheck.Prop.{AnyOperators, forAll, all}
def interleave[T](xs: List[T], ys: List[T]): List[T] = { if(xs.isEmpty) ys else if(ys.isEmpty) xs else xs.head :: ys.head :: interleave(ys.tail, xs.tail) }
val propInterleave = forAll { (xs: List[Int], ys: List[Int]) => val res = interleave(xs,ys) val is = (0 to math.min(xs.length, ys.length)-1).toList all( "length" |: xs.length+ys.length =? res.length, "zip xs" |: xs =? is.map(i => res(2*i)) ++ res.drop(2*ys.length), "zip ys" |: ys =? is.map(i => res(2*i+1)) ++ res.drop(2*xs.length) ) }
Three conditions are combined using the Prop.all method. We could just as well have put && operators between the conditions; the effect is the same. Each condition is also labeled to make it easier to see which caused the property failure. Let's check the property:
scala> propInterleave.check ! Falsified after 6 passed tests. > Labels of failing property: Expected List("0", "0") but got List("0", "1") zip xs > ARG_0: List("0", "0") > ARG_0_ORIGINAL: List("1", "423082184") > ARG_1: List("0", "1") > ARG_1_ORIGINAL: List("1730012397", "941647908")
You might have spotted the error in interleave already; the lists are not interleaved in the correct order. The condition marked zip xs says that every other item (starting with the first one) in res should come from xs, but in fact the third item comes from ys. This is clear from ScalaCheck's test report, where both the condition label and the expected and actual values are presented. Notice also how ScalaCheck has simplified the lists as much as possible to still be able to manifest the error. Lists of length zero or one wouldn't have triggered the defect, but the lists generated above are just right for demonstrating it.
You can tell ScalaCheck to print out interesting intermediate values to make the test report even more informative. For example, it could be valuable to see for yourself the result of the call to interleave. To do this, we add another label to the whole property:
val propInterleave = forAll { (xs: List[Int], ys: List[Int]) => val res = interleave(xs,ys) val is = (0 to Math.min(xs.length, ys.length)-1).toList all( "length" |: xs.length+ys.length =? res.length, "zip xs" |: xs =? is.map(i => res(2*i)) ++ res.drop(2*ys.length), "zip ys" |: ys =? is.map(i => res(2*i+1)) ++ res.drop(2*xs.length) ) :| ("res: "+res) }
scala> propInterleave.check ! Falsified after 7 passed tests. > Labels of failing property: Expected List("0", "-1") but got List("0", "0") zip xs res: List(0, 0, 0, -1) > ARG_0: List("0", "-1") > ARG_0_ORIGINAL: List("-2044148153", "2147483647", "-1") > ARG_1: List("0", "0") > ARG_1_ORIGINAL: List("-2147483648", "1073458288", "3876387")
By now, there shouldn't be any doubt about the error. The second elements of xs and ys are in the wrong order in res. By using property labels in this way, you can make it easier to find out exactly why a property fails. You can put as many labels as you like on a property. Each time ScalaCheck finds a property that fails, it will list all the associated labels.
Using test framework error messages
One of the reasons you may wish to use ScalaCheck with a traditional test framework is that you can take advantage of its error messages to avoid cluttering your properties with too many labels. For example, here's how you might write the previous example using ScalaTest's PropertyChecks trait, which provides integration with ScalaCheck:
import org.scalatest._ import Matchers._ import prop.PropertyChecks._
forAll { (l1: List[Int], l2: List[Int]) => val res = interleave(l1,l2) val is = (0 to Math.min(l1.length, l2.length)-1).toList l1.length + l2.length shouldEqual res.length l1 shouldEqual is.map(i => res(2*i)) ++ res.drop(2*l2.length) l2 shouldEqual is.map(i => res(2*i+1)) ++ res.drop(2*l1.length) }
Note that instead of labels, you just write matcher expressions like you would in a traditional example-based test. If you paste this code into a REPL session, it would fail similarly as before:
TestFailedException was thrown during property evaluation. Message: List(0, 0) did not equal List(0, 1) Location: (<console>:23) Occurred when passed generated values ( arg0 = List(0, 0), // 3 shrinks arg1 = List(0, 1) // 33 shrinks )
You get a good error message, List(0, 0) did not equal List(0, 1), as well as the filename and line number containing the offending assertion, <console>:23, as demonstrated here. This location refers to the same line labeled l1 in previous example. In an IDE, you can click on this and hop right to that line of code. Thus, you do not need the labels.
Collecting test statistics
Even if a property passes, you might want to know a little about what kind of data ScalaCheck used when testing it. For example, if you have non-trivial preconditions for a method, and have not taken enough care when writing the property for it, you could find that ScalaCheck only tests the method with a very restricted data set. Or maybe you have written a custom generator but missed out on large parts of the possible instance variants. Or you might be curious as to what test cases ScalaCheck has come up with. If so, you can use some of ScalaCheck's statistics collectors to analyze the input data generated for a tested property.
You can use the method Prop.classify to classify generated test data and make ScalaCheck present the data distribution after the property has been tested. The following property demonstrates its use:
import org.scalacheck.Prop.{forAll, classify}
val p = forAll { n:Int => classify(n % 2 == 0, "even", "odd") { classify(n < 0, "neg", "pos") { classify(math.abs(n) > 50, "large") { n+n == 2*n } } } }
ScalaCheck produces the following output when testing this property:
scala> p.check + OK, passed 100 tests. > Collected test data: 26% pos, even 20% neg, odd 18% pos, odd 12% neg, even 12% large, pos, odd 7% large, neg, even 3% large, neg, odd 2% large, pos, even
As you can see, classify accepts one or two labels after the classification condition, depending on how explicit you want the distribution report to be. The labels can be of any type; ScalaCheck will use their toString methods in order to print them out in the report. You can add as many classifications as you like; ScalaCheck will just add them together and present the distribution as above, whether the property test passes or fails.
In addition to the classify method, there is also a more general method for collecting distribution statistics. Prop.collect takes one (calculated) label object without any condition. Look at this example:
import org.scalacheck.Prop.{forAll, collect} import org.scalacheck.Gen.choose
val propSlice = forAll { xs: List[Int] => forAll(choose(0,xs.length-1)) { n => forAll(choose(n,xs.length)) { m => val slice = xs.slice(n,m) val label = slice.length match { case 0 => "none" case 1 => "one" case n if n == xs.length => "whole" case _ => "part" } collect(label) { xs.containsSlice(slice) } } } }
This property checks that the list returned from List.slice is always considered part of the original list, using the List.containsSlice method. Nested forAll expressions generate reasonable indices, n and m. Then a label is set according to how big the slice was. Finally, this label is passed to the collect method. ScalaCheck outputs the following when the property is tested:
scala> propSlice.check + OK, passed 100 tests. > Collected test data: 69% part 21% one 10% none
Now we can see that containsSlice was never tested with the whole list as input. xs.containsSlice(xs) should be trivially true, but this is just such an edge case that the implementation might have gone wrong. A simple fix is to add that condition to the property too:
l.containsSlice(slice) && l.containsSlice(l)
5.2 ScalaCheck's property combinators
Until now, you have been introduced to several of ScalaCheck's property-creating methods in a somewhat random fashion. This section will present the available property combinators in a more systematic way.
Prop.forAll
You have already seen much use of the Prop.forAll property combinator. In logic, a for-all construction is called a universal quantifier, and is simply a way to say that a certain condition holds for all members of a specific set.
forAll in ScalaCheck works in exactly the same way: it specifies that a certain condition should hold for a given set of values. The condition that the forAll method expects is a function that returns either a Boolean or a Prop instance. The data sets that the condition should hold for is either inferred implicitly from the function's input type, or specified explicitly by one or more generators (Gen instances). Here are some trivial examples:
import org.scalacheck.Prop.forAll import org.scalacheck.Gen.{choose, numChar, alphaChar}
val p1 = forAll { n:Int => 2*n == n+n }
val p2 = forAll { (s1:String, s2:String) => (s1+s2).endsWith(s2) }
val p3 = forAll(choose(0,10)) { n => n >= 0 && n <= 10 }
val p4 = forAll(numChar, alphaChar) { (cn,ca) => cn != ca }
For the first two properties, p1 and p2, ScalaCheck uses implicit generators to come up with input values to the property functions. For p3, it uses an explicit generator (Gen.choose), and for p4 it uses two explicit generators. In both cases, the input to the property function matches the values produced by the generators.
When ScalaCheck tests a for-all property, it can't check the condition for every possible member of the data set, since there might be an infinite number of them. Instead, ScalaCheck is satisfied as soon as it has found a certain number of cases for which the condition holds. By default this number is 100, but you can configure it when checking the property, which Chapter 7 demonstrates.
Prop.throws
Prop.throws is a boolean method that returns true only if a certain exception is thrown when a given expression is evaluated. You can use it in properties in the following way:
import org.scalacheck.Prop.{forAll, throws, classify}
val propDivByZero = forAll { n:Int => throws(classOf[ArithmeticException]) (n/0) }
val propListBounds = forAll { (xs: List[String], i: Int) => val inside = i >= 0 && i < xs.length classify(inside, "inside", "outside") { inside || throws(classOf[IndexOutOfBoundsException])(xs(i)) } }
The propListBounds property checks that the list throws an exception if out-of-bounds indices are accessed. The variable inside tells us if the index that ScalaCheck generated lies within the bounds of the generated list. If inside is true, the property condition is trivially true. If it is false, the throws method asserts that the list throws the correct exception.
As you can see, we add a Prop.classify call to the propListBounds property. This way we can see if the property test covers both cases in which an IndexOutOfBoundsException exception is and is not thrown:
scala> propDivByZero.check + OK, passed 100 tests.
scala> propListBounds.check + OK, passed 100 tests. > Collected test data: 84% outside 16% inside
Prop.exists
Apart from the universal quantifier (Prop.forAll in ScalaCheck), in logic there is also an existential quantifier. This specifies that a certain condition should hold for at least one member of a specific set. In other words, one member must exist in the set for which the condition is true.
The existential quantifier can be found in ScalaCheck too, in the form of Prop.exists. This method works in much the same way as Prop.forAll: it takes a function literal as its condition and either infers the data set itself or accepts an explicit generator. However, when ScalaCheck tests a property produced by the exists method, it will pass the property as soon as it finds one case for which the condition holds.
In practice, the Prop.exists combinator is not so useful, since it can be difficult for ScalaCheck to find a case that satisfies the condition, at least if the condition is non-trivial. You might find use for it though, and it makes ScalaCheck's similarity to logic apparent. Here are two basic examples:
import org.scalacheck.Prop.exists import org.scalacheck.Gen.choose
val p1 = exists { n:Int => (n % 2 == 0) && (n % 3 == 0) } val p2 = exists(choose(0,10)) { _ == 3 }
When ScalaCheck finds a case that fulfills an existential property, it reports the property as proved instead of passed as it does for universally quantified properties. We can let ScalaCheck test the above properties to see this:
scala> p1.check + OK, proved property. > ARG_0: 0
scala> p2.check + OK, proved property. > ARG_0: 3
As you can see, ScalaCheck also presents the arguments it found that prove the properties.
You can nest forAll and exists expressions as you like, and you can of course use any other property combinators inside the conditions you supply.
Constant properties
There are several constant properties defined in org.scalacheck.Prop that always give a certain result. The constant properties are Prop.undecided, Prop.falsified, Prop.proved, Prop.passed, and Prop.exception. We can check out their behavior below:
import org.scalacheck.Properties import org.scalacheck.Prop.{ undecided, proved, passed, exception, falsified }
object ConstantProps extends Properties("Const") { property("p1") = undecided property("p2") = falsified property("p3") = proved property("p4") = passed property("p5") = exception(new Throwable("My fault")) }
scala> ConstantProps.check ! Const.p1: Gave up after only 0 passed tests. 101 tests were discarded. ! Const.p2: Falsified after 0 passed tests. + Const.p3: OK, proved property. + Const.p4: OK, passed 100 tests. ! Const.p5: Exception raised on property evaluation. > Exception: java.lang.Throwable: My fault $line3.$read$$iw$$iw$ConstantProps$.<init>(<c... $line3.$read$$iw$$iw$ConstantProps$.<clinit>(...
The constant properties are not overly useful in real-world situations, but they can be used as placeholders for other properties, or in property comparisons, for example.
Property operators
The common boolean operators, && and ||, are defined for Prop instances. This means that you can build complex properties by combining simpler ones. This can help you to put labels on different parts of a property:
import org.scalacheck.Prop.{propBoolean, forAll}
val propSorted = forAll { xs: List[Int] => val r = xs.sorted
val isSorted = r.indices.tail.forall(i => r(i) >= r(i-1)) val containsAll = xs.forall(r.contains) val correctSize = xs.size == r.size
isSorted :| "sorted" && containsAll :| "all elements" && correctSize :| "size" }
If this property failed, we would immediately see which part of the property condition caused the failure. Notice that we import Prop.propBoolean above, since this is one of the few cases when the Scala compiler can't handle the automatic conversion of boolean values to Prop instances completely by itself.
In addition to the && and || operators, there are two property-grouping combinators called Prop.all and Prop.atLeastOne.
Prop.all takes a list of properties and returns a property that holds if all the provided properties hold. It's like using the && operator repeatedly; Prop.all(p1, p2, p3) is the same as p1 && p2 && p3.
Prop.atLeastOne is similar to Prop.all; it just requires one of the provided properties to hold. In other words, Prop.atLeastOne(p1, p2, p3) is the same as p1 || p2 || p3.
Grouping properties like this lets ScalaCheck test several conditions with each generated input. It also allows you to fully specify a function with a single property.
5.3 Conclusion
This chapter presented most of the property combinators that are available in ScalaCheck. When using ScalaCheck, you will find that the most useful combinators are the ones used for labeling and collecting statistics, and of course the Prop.forAll combinator. The Prop.throws method is very valuable when you want to make sure your code handles exceptional cases correctly.
These few pieces combine to let you fully specify and describe your expectations of the code under test. The next chapter will detail the ways you can control how ScalaCheck generates test cases for your properties.