ScalaTest is a popular testing framework created by programmer Bill Venners specifically for Scala. ScalaTest is an extensive behavior-driven development (BDD) suite with numerous built-in specs, but it also intergrates with some of the classic testing frameworks like JUnit and TestNG. ScalaTest also has two assertion dialects to choose from, depending how you want your test to read. ScalaTest’s learning curve is fast, and it runs automatically in the test plug-in installed with SBT.
ScalaTest offers several different flavors of tests. The most basic one is the FunSpec, which we used to add an Artist to an Album in our introduction. It contains a standard storyboard that describes the reason for the existence of the test using a describe clause and subsequent tests that fulfill that description. As we saw in our introductory chapter, AlbumTest.scala took the form:
package com.oreilly.testingscala
import org.scalatest.FunSpec
import org.scalatest.matchers.ShouldMatchers
class AlbumTest extends FunSpec with ShouldMatchers {
describe("An Album") {
it ("can add a Artist object to the album") {
val album = new Album("Thriller", 1981,
new Artist("Michael", "Jackson"))
album.artist.firstName should be ("Michael")
}
}
}AlbumTest in this example extends FunSpec. This is a standard BDD specification. FunSpec is a trait, which means that is mixed into a class. This mixed-in trait provides us with a few methods to run our test: describe and it. describe is the subject of the test. In our case we are testing an Album, that is subject under specification. Each test is specified using an it method, which is mixed in from FunSpec.
it is used to describe the purpose of the test.
In our example, the it method verifies that we can add an Artist to an Album at construction time.
AlbumTest also mixed in the trait ShouldMatchers. The ShouldMatchers trait provides a DSL used to make the assertions. In AlbumTest, the ShouldMatchers trait was used to form the assertion:
album.artist.firstName should be ("Michael")Note the should verb in the statement. The lefthand side of the assertion will typically be the object whose state is being investigated, and the righthand side will typically be the value that is expected. If any matcher fails, the matcher will throw a TestFailedException. At that point, ScalaTest will trap that error and report it as a failed test. If the matcher succeeds, nothing happens, and the test continues.
ScalaTest can be run on the command line or through a build tool like SBT, as we have already prepared.
We can run our earlier AlbumTest with a command issued within the project folder:
$ scala -cp scalatest-1.8.jar org.scalatest.tools.Runner -p . -o -s AlbumTest
ScalaTest is meant to be readable at the output, giving a clean, storylike output. If you have terminal coloring, tests that pass will be formatted with the color green, giving you intuitive feedback about the success of your code.
To use ScalaTest in SBT, include the dependency vector in build.sbt as described in Chapter 2. For a refresher, either one of these lines will work.
libraryDependencies += "org.scalatest" %% "scalatest" % "1.8" % "test"
libraryDependencies += "org.scalatest" % "scalatest_2.9.2" % "1.8" % "test"
For information about what each of these settings mean, please refer to Chapter 2.
The next section will discuss the many ways a developer can write assertions using ScalaTest.
In the previous example, we made an assertion that the first name of the Thriller artist’s was indeed Michael. These assertions check that the code results in a state that we are expecting.
ScalaTest’s assertions come in two flavors: MustMatchers and ShouldMatchers. The only difference between them is the language that shows up in the testing report.
ScalaTest provides a range of matchers for many situations. In the following subsections we’ll illustrate them with Should Matchers, because Must Matchers work the same way.
Simple matchers perform the simple act of asserting one value with another.
src/test/scala/com/oreilly/testingscala/ShouldMatcherSpec.scala.
val list = 2 :: 4 :: 5 :: Nil list.size should be (3)
The right hand value that is being evaluated must be enclosed in parentheses. While there is an urge to not use parentheses in the last part of an assertion, it is required. If you miss the parentheses there will be a compilation error. For example list.size should be 3 is incorrect and will not compile. The assertion should be changed to list.size should be (3) in order for it to compile.
The same assertion can be made using the equal method:
list.size should equal(3)
There rarely is any value in asserting a condition using == and != in ScalaTest, because the line will be evaluated but no actual testing assertion will be made. For example, merely stating that list.size == 4 in the previous example will be evaluated to false, but the test will still continue to run and possibly report a successful completion since a TestFailedException is not thrown.
ScalaTest includes matchers that aid in making assertions about strings. You can determine whether one string contains another, starts with a particular string, ends with a particular string, or matches a regular expression. For more information on regular expressions and how to sculpt them effectively, see Mastering Regular Expressions by Jeffrey E.F. Friedl (O’Reilly).
val string = """I fell into a burning ring of fire.
I went down, down, down and the flames went higher"""
string should startWith("I fell")
string should endWith("higher")
string should not endWith "My favorite friend, the end"
string should include("down, down, down")
string should not include ("Great balls of fire")
string should startWith regex ("I.fel+")
string should endWith regex ("h.{4}r")
string should not endWith regex("\d{5}")
string should include regex ("flames?")
string should fullyMatch regex ("""I(.|
|S)*higher""")These are examples of ScalaTest’s string matchers. Using a Johnny Cash lyric, the first assertion checks that this particular lyric starts with “I fell,” while the second assertion checks that the lyric ends with the String “higher.” The third assertion uses not to assert that Jim Morrison and Johnny Cash lyrics are not mixed. The fourth assertion asserts that indeed the lyrics “down, down, down” are included. The fifth assertion makes sure that Jerry Lee Lewis’s lyrics are not included in the Ring of Fire, because having Great Balls of Fire in Ring of Fire might violate a some fire codes in some counties.
The sixth through ninth assertions use regular expressions. The last assertion uses fullyMatch as a modifier to regex to assert that the entire lyric must match the regular expression.
Note
A keen eye will have noticed that the last assertion uses triple quotes instead of single quotes for the regular expression. This is preferable because in Scala a triple quote, or raw string, saves you from having to escaping each backslash () and make it two backslashes (\). For more information on escaping backslashes and raw strings, refer to Programming Scala by Dean Wampler and Alex Payne (O’Reilly).
ScalaTest supports relational operators. These examples should be self-explanatory.
val answerToLife = 42 answerToLife should be < (50) answerToLife should not be > (50) answerToLife should be > (3) answerToLife should be <= (100) answerToLife should be >= (0) answerToLife should be === (42) answerToLife should not be === (400)
Perhaps the only ScalaTest that requires explanation is the triple-equal operator (===). The operator is used to evaluate whether the right hand side is equal to the left. As mentioned previously, using Scala’s equals operator (==) will only evaluate equality, but never assert equality, so it’s best to stick with should be, should equal, or ===.
Floating-point arithmetic in the Java virtual machine (JVM) is a nasty business—worthy of a Discovery Channel reality show.
Consider the operation 0.9 - 0.8. A seasoned Java developer knows that this is not an innocent call. Running that operation in a Scala REPL will result in 0.09999999999999998. ScalaTest provides a buffer to account for some of these inaccuracies using a plusOrMinus method.
(0.9 - 0.8) should be (0.1 plusOrMinus .01) (0.4 + 0.1) should not be (40.00 plusOrMinus .30)
In the first line, the righthand side operation asserts that the answer is 0.1 plus or minus a discrepancy of 0.1. The second line in the example is not based much in reality, but just shows that plusOrMinus can be used in any kind of circumstance that you can invent.
In Scala, a very important point to consider is that the == operator evaluates the natural equality for value types and object identity for reference types, not reference equality. In ScalaTest, the === will assert object equality. But to test object references, ScalaTest offers theSameInstanceAs:
val garthBrooks = new Artist("Garth", "Brooks")
val chrisGaines = garthBrooks
garthBrooks should be theSameInstanceAs (chrisGaines)
val debbieHarry = new Artist("Debbie", "Harry")
garthBrooks should not be theSameInstanceAs(debbieHarry)This example will instantiate a new Artist, Garth Brooks—perhaps you’d recognize him as one of your friends in low places. The Garth Brooks object is referenced by the garthBrooks variable, and also by the chrisGaines variable. The third line asserts that the object the garthBrooks variable is referencing is the same object. The last two lines of the example assert that a Debbie Harry object is not the same object as Garth Brooks. If he is, well, congratulations go out to Garth Brooks for pulling a really nifty trick.
For Iterable, one of many types in Scala that make up a collection, ScalaTest provides a couple of methods to help you make assertions.
List() should be('empty)
8 :: 6 :: 7 :: 5 :: 3 :: 0 :: 9 :: Nil should contain(7)The first line uses an 'empty Scala symbol to assert that an Iterable is empty.
Symbols in Scala are defined as object scala.Symbol and are typically used as identifiers and keys. Symbols are immutable placeholders, and unlike strings that are used as a definition or a setting than a String to represent a name, account number, etc.
The second line in the above example asserts that Jenny’s number actually contains the number 7 in the List.
ScalaTest has length and size matchers to determine the size of a Seq or Traversable. According to the ScalaDoc, length and size are equivalent, therefore their use depends on your preference.
(1 to 9) should have length (9) (20 to 60 by 2) should have size (21)
ScalaTest has a special matcher syntax for Map. Assertions with Map include the ability to ask whether a key or value is in the Map.
val map = Map("Jimmy Page" -> "Led Zeppelin", "Sting" -> "The Police",
"Aimee Mann" -> "Til' Tuesday")
map should contain key ("Sting")
map should contain value ("Led Zeppelin")
map should not contain key("Brian May")The example is self-explanatory. Given a map of artist names with their associated band names, assertions are made that Sting is associated with the Police and that Jimmy Page is associated with Led Zeppelin. The last line, an assertion is made that Brian May, guitarist for Queen, is not in the map.
ScalaTest’s and and or methods can be used to create compound assertions in a test.
val redHotChiliPeppers = List("Anthony Kiedis", "Flea", "Chad Smith", "Josh
Klinghoffer")
redHotChiliPeppers should (contain("Anthony Kiedis") and
(not contain ("John Frusciante")
or contain("Dave Navarro")))This example is a list of the current members of the Red Hot Chili Peppers as a List of String objects. The assertion is made that the List contains singer Anthony Kiedis but does not contain former guitarists, John Frusciante and Dave Navarro.
In practice, using compound matchers will pose some difficulty with the parentheses. Here are some rules to keep in mind when engineering compound assertions. First, parentheses must be included around and and or assertions. Secondly, remember that the righthand assertion must also be wrapped in parentheses. The following line would not compile.
redHotChiliPeppers should not contain "The Edge" or contain "Kenny G"
To fix the first issue, we apply parentheses around the or assertion.
redHotChiliPeppers should not (contain "The Edge" or contain "Kenny G")
This still will not compile, because we don’t have parentheses around the righthand side of the assertions The Edge or Kenny G. After we repair this, the example should look as follows and should compile.
redHotChiliPeppers should not (contain ("The Edge") or contain ("Kenny G"))Another rule to keep in mind with compound matchers is that and and or are not short-circuited. In other words, all clauses are evaluated even when other languages would decide they are unnecessary and would skip their evaluation. The following example illustrates that rule.
var total = 3
redHotChiliPeppers should not (contain ("The Edge") or contain {total += 6;
"Kenny G"})
total should be (9)The above contrived example will set a var variable to 3. If you are unfamiliar with var, var is a non-final variable with the ability to change the reference[3]. Under short-circuiting in another language such as Lisp or Perl, total would never be increased by 6, because the Red Hot Chili Peppers don’t contain The Edge and therefore the first assertion is true. A short-circuiting language would stop and not evaluate the second clause. But since ScalaTest does not short circuit, both clauses are evaluated.
Lastly, Scala rarely deals with null because Scala has Some(...) and None to avoid these cases. But Java uses null, a lot, and since ScalaTest is Java-friendly, there will be circumstances where you care about null when architecting test cases.
gorillaz should (not be (null) and contain ("Damon Albarn"))The previous example will generate the ever-so-hated NullPointerException when a developer if gorillaz is referencing null. To avoid this prickly message, undo compound assertions and place each assertion on a line of its own:
gorillaz should not be (null)
gorillaz should contain ("Damon Albarn")If gorrillaz is null, the test still won’t pass, which is correct. But this time the other test won’t throw a NullPointerException.
ScalaTest has also a clever way to assert that an object’s properties (found through its getter methods) are valid in one cohesive check.
import scala.collection.mutable.WrappedArray
val album = new Album("Blizzard of Ozz", 1980, new Artist("Ozzy", "Osbourne"))
album should have (
'title ("Blizzard of Ozz"),
'year (1980),
'artist (new Artist("Ozzy", "Osbourne"))
)Property matchers can be used to reflect on the object’s properties and make assertions on those properties. This example checks that the album Blizzard of Ozz was created in 1980 by some chap named Ozzy Osbourne. Assertions can be made to be sure that the title, year, and artist are indeed the ones given at instantiation. It is worth remembering that Artist is a class whose parameters are val, therefore getters are created automatically by Scala and using property assertions will work. It would also work to make all the properties of Artist a var, since var creates both a getter and a setter implicitly.
Since ScalaTest is Java friendly, it can be used to make a assertions about basic java.util collections in the same way that it can test Scala collections. The following example shows many of the methods used previously, like should have length, should contain, etc.
import java.util.{List => JList, ArrayList => JArrayList, Map => JMap,
HashMap => JHashMap}
val jList: JList[Int] = new JArrayList[Int](20)
jList.add(3); jList.add(6); jList.add(9)
val emptyJList: JList[Int] = new JArrayList[Int]()
emptyJList should be('empty)
jList should have length (3)
jList should have size (3)
jList should contain(6)
jList should not contain (10)
val backupBands: JMap[String, String] = new JHashMap()
backupBands.put("Joan Jett", "Blackhearts")
backupBands.put("Tom Petty", "Heartbreakers")
backupBands should contain key ("Joan Jett")
backupBands should contain value ("Heartbreakers")
backupBands should not contain key("John Lydon")The top line may look esoteric to beginners. Since there is a List container in Scala as well as in Java, Scala provides an import alias that can be used to rename classes used within the class. In the previous example, whenever a java.util.List is required, it will be referred to in the test class as JList. ArrayList will be referred to as JList, Map as JMap, and HashMap as JHashMap.
The first half of the example creates a JList of Int referenced by jList and an empty JList of Int called emptyJList. The following five lines just make an assertion about java.util.List using the same ScalaTest language as for Scala collections. backupBands is a java.util.Map construct that maps a headliner with the backup band using the same ScalaTest map assertions used for Scala maps.
There is no extra setup or hassle to get ScalaTest to work with “Plain Old Java.”
What is the difference between ShouldMatchers and MustMatchers? Nothing except how you would like the assertion to display on the output. Do not expect that should should pass through, while a must matcher will throw a TestFailedException. If an assertion is not met using either matcher, a TestFailedException will be thrown regardless of whether you use should or must. The following is a small cross-section of all the examples seen in the previous section, except the word should is replace with the word must.
val list = 2 :: 4 :: 5 :: Nil list.size must be(3)
val string = """I fell into a burning ring of fire.
I went down, down, down and the flames went higher"""
string must startWith regex ("I.fel+")
string must endWith regex ("h.{4}r")val answerToLife = 42 answerToLife must be < (50) answerToLife must not be >(50)
val garthBrooks = new Artist("Garth", "Brooks")
val chrisGaines = garthBrooks
val debbieHarry = new Artist("Debbie", "Harry")
garthBrooks must be theSameInstanceAs (chrisGaines)(0.9 - 0.8) must be(0.1 plusOrMinus .01)
List() must be('empty)
1 :: 2 :: 3 :: Nil must contain(3)
(1 to 9).toList must have length (9)
(20 to 60 by 2).toList must have size (21)val map = Map("Jimmy Page" -> "Led Zeppelin", "Sting" -> "The Police",
"Aimee Mann" -> "Til' Tuesday")
map must contain key ("Sting")
map must contain value ("Led Zeppelin")
map must not contain key("Brian May")val redHotChiliPeppers = List("Anthony Kiedis", "Flea", "Chad Smith",
"Josh Klinghoffer")
redHotChiliPeppers must (contain("Anthony Kiedis") and
(not contain ("John Frusciante")
or contain("Dave Navarro")))There are a couple of ways in ScalaTest to verify that an expected exception is made and trapped. The first way is by placing the volatile code in an intercept block. The intercept block is analogous to the barrels used by bomb sqauds to defuse bombs: any code expected to throw an exception is placed in the block. If the code does not throw the expected exception, the test will fail.
"An album" should {
"throw an IllegalArgumentException if there are no acts when created" in {
intercept[IllegalArgumentException] {
new Album("The Joy of Listening to Nothing", 1980, List())
}
}
}This example is a standard spec that expects an IllegalArgumentException when an Album is created with no Artist. If the instantiation of Album does not throw an exception, the test itself will fail.
Another way to assert that an exception should be thrown is to use either a ShouldMatcher or MustMatcher to assert that the call indeed throws the necessary Exception. This is done using an evaluating block and either a must or should clause to check that it produces the expected Exception, as seen in the following example.
val thrownException = evaluating {new Album("The Joy of Listening to Nothing",
1980, List())} must produce [IllegalArgumentException]
thrownException.getMessage() must be ("An Artist is required")The two examples do nearly the same thing, but the second one using the evaluating clause allows the developer to introspect the Exception and assert information about the exception after it has been thrown. One of the benefits of using this method is that a call often throws different exceptions, and it is good practice to find out exactly which one was thrown. If by chance the class Album does has two different cases where an IllegalArgumentException is thrown (say, one when there is no Artist placed onto the album, and one where the album year is less than 1900) it would be wise to introspect the exception and make sure that the right one was captured. If it isn’t, the test should fail. The evaluating example just shown does extra analysis to ensure that the IllegalArgumentException that is thrown indeed the one that was expected.
Before continuing further into the different specifications, an introduction to some of the tools available in ScalaTest is in order. We’ll start with informers—not the kind that’ll rat you out. Informers in ScalaTest are spices, analogous to debug statements, that can be applied anywhere in a test to display information about the test. To apply an informer, merely add an info(String) method anywhere within your test.
Informers provide enhanced feedback on the test and give any stakeholder on the project a clear picture of the purpose of your test.
class AlbumSpec extends FunSpec with ShouldMatchers {
describe("An Album") {
it("can add an Artist to the album at construction time") {
val album = new Album("Thriller", 1981, new Artist("Michael", "Jackson"))
info("Making sure that Michael Jackson is indeed the artist of Thriller")
album.acts.head.asInstanceOf[Artist].firstName should be("Michael")
album.acts.head.asInstanceOf[Artist].lastName should be("Jackson")
}
}
}The testing results will give us the informer’s results, prepended with + to denote that the printout comes from an informer. Running test-only com.oreilly.testingscala.AlbumSpec will render the following response in SBT. As a reminder test-only takes a test class argument so that SBT will only test one class. In our case that’s com.oreilly.testingscala.AlbumSpec.
> test-only com.oreilly.testingscala.AlbumSpec
[info] Compiling 1 Scala source to /home/danno/testing_scala_book.git
/testingscala/target/scala-2.9.2/test-classes...
[info] AlbumSpec:
[info] An Album
[info] - can add an Artist to the album at construction time
[info] + Making sure that Michael Jackson is indeed the artist of Thriller
[info] Passed: : Total 1, Failed 0, Errors 0, Passed 1, Skipped 0
[success] Total time: 4 s, completed Dec 19, 2011 11:16:12 AMThese are three words any test-driven developer will remember. ScalaTest, as well as Specs2, which will be covered later in the book, uses the three phases to document the scenario and outcome of a test. Just about every process and recipe in the world can be described with GivenWhenThen.
Given we have eggs, milk, flour, sugar, baking powder, and baking soda. When mixed together and placed in an oven of 350°F, then we partake in an opulent cake!
Testing is the same. Each test has a Given that is the initial state of a test. In this initial state we typically gather any nouns or ingredients to the test. Following is a When: which actions or verbs are to be performed on the nouns provided in the Given clause. Finally, Then specifies the results of the test, where all post-action analysis takes place.
The silent partner in the GivenWhenThen trait is the and method. It serves to break apart any given, when, and then if any of those clauses get too unwieldy. It should be visibly evident that a given, when, or then clause has perhaps gotten big if the string describing the test is littered with and+s. For example, if a and the artist’s first and last names should be then clause states that "the act should be an instance of +ArtistMichael Jackson" it would be better to break the then clause apart, making it both more readable and more logically categorized. For instance: “the act should be an instance of Artist" and “the Artist’s first and last names should be Michael Jackson.”
GivenWhenThen in the back end are just informers that help the test-driven developer organize her thoughts in a familiar structure. GivenWhenThen is a trait that can be mixed into any test. This is particularly useful for a Spec that doesn’t have a strict structure.
GivenWhenThen can therefore be applied anywhere where needed. The technique goes particularly well with FunSpec, [FunSpec] and FeatureSpec, [FeatureSpec].
The following example retrofits the Album test, mixing in GivenWhenThen methods.
class AlbumSpec extends FunSpec with ShouldMatchers with GivenWhenThen {
describe("An Album") {
it("can add an Artist to the album at construction time") {
given("The album Thriller by Michael Jackson")
val album = new Album("Thriller", 1981, new Artist("Michael", "Jackson"))
when("the album's artist is obtained")
val artist = album.artist
then("the artist obtained should be an instance of Artist")
artist.isInstanceOf[Artist] should be (true)
and("the artist's first name and last name should be Michael Jackson")
artist.firstName should be("Michael")
artist.lastName should be("Jackson")
}
}
}Each test trait lets you mark a test as pending. pending is a placeholder for tests that have not been defined. The benefit of pending is that it lets you quickly jot down an idea, perhaps something that popped into your mind while you were focused on something else. pending also is a great way to map out a course of tests before actual implementation, possibly eliminating some ideas that you thought would make sense at first but didn’t after delineating the purpose of all the tests.
In a Spec, after each it clause the corresponding block returns pending for any test that is not implemented or just not ready.
When I first had the idea for AlbumTest and was jotting down possible tests to include, pending might have made the test look like the following:
class AlbumSpec extends FunSpec with ShouldMatchers with GivenWhenThen {
describe("An Album") {
it("can add an Artist to the album at construction time") {pending}
it("can add opt to not have any artists at construction time") {pending}
}
}The tests marked as pending will generate output marked pending when run either at the command prompt or through SBT.
[info] AlbumSpec: [info] An Album [info] - can add an Artist to the album at construction time (pending) [info] - can add opt to not have any artists at construction time (pending)
An interesting use of pending is to keep that keyword on the bottom of the test while implementing it, so that the test is still considered under construction. When the test is ready, let it run by removing the pending statement.
class AlbumSpec extends FunSpec with ShouldMatchers {
describe("An Album") {
it("can add an Artist to the album at construction time") {
val album = new Album("Thriller", 1981, new Artist("Michael", "Jackson"))
info("Making sure that Michael Jackson is indeed the artist of Thriller")
pending
}
it("can add opt to not have any artists at construction time") {pending}
}
}This example is a test still in progress. As long as the test compiles, you can run it through ScalaTest and the last pending guarantees that the test runner will still treat this as a pending test. ScalaTest will not cruelly discipline the test-driven developer with failures for an incomplete test. Note that the previous example also contains an informer that is rendered in the test results, as seen below, even though the test is pending. If the test is marked as pending, ScalaTest will still honor any informers, including GivenWhenThen methods.
[info] AlbumSpec: [info] An Album [info] - can add an Artist to the album at construction time (pending) [info] + Making sure that Michael Jackson is indeed the artist of Thriller [info] - can add opt to not have any artists at construction time (pending)
A developer is often unsure of the validity of a test, whether it’s because the production code has been phased out, the test or production code is too complex, or something is just plum broken. So each of the ScalaTest traits lets you temporarily disable certain tests. How a test is ignored depends on the trait, and the various ways will be covered with their respective traits.
To ignore any poor or broken test, replace the it keyword with ignore. The following example adds another test to the FunSpec. This test is ignored though, because it uses ignore instead of it.
package com.oreilly.testingscala
import org.scalatest.FunSpec
import org.scalatest.matchers.ShouldMatchers
class AlbumSpec extends FunSpec with ShouldMatchers {
describe("An Album") {
//Code removed for brevity
ignore("can add a Producer to an album at construction time") {
new Album("Breezin'", 1976, new Artist("George", "Benson"))
//TODO: Figure out the implementation of an album producer
}
}
}In this example, which perhaps is oversimplified, the developer may be stalled while waiting for more information about the data set or the application. If you view the response from the SBT console or command prompt, the earlier tests are still valid and run, while the ignored test is displayed with a message stating that it has been ignored.
[info] AlbumSpec: [info] An Album [info] - can add an Artist to the album at construction time [info] - can add a Producer to an album at construction time !!! IGNORED !!!
Simply changing ignore to it will make the test available for testing.
Tagging categorizes tests so that they can run as part of a group. Tagging is very much like tagging a blog entry, where the subject of the blog entry is categorized with keywords so that not only can search engines find the entry, but users can click a word cloud to see similar entries. In testing, it is very useful to categorize tests for several reasons:
- Some tests are slow and you might want to skip them at times.
- Some tests check related functionality and should be run together.
- You may want to categorize tests as unit tests, integration tests, acceptance tests, or any other type.
While this book focuses mostly on unit testing using test-driven development there are other type of testing. Integration testing involves tests whose objects work with another, or examines how objects work with outside systems or the Internet. Integration testing takes place after unit testing, therefore after test-driven development.
Another level of testing is acceptance testing, which takes the program out for a test drive to see whether it is ready for deployment. Acceptance testing deals with usability and how the stakeholders of the product react to the developer’s masterpiece.
Outside of these main levels of testing, there are other categories to consider. Security testing: how carefully contained and safe is the code? Performance testing: How fast does the application respond in production? Load and stress testing: In a server environment, how many users can hit the server, and can the server respond adequately?
Though it is difficult, many projects do have unit, integration, and acceptance testing for all code that is developed.
All testing traits in ScalaTest can be tagged with strings describing the test. Each testing trait has its own methodology to tag the test, but when the tagging is done, tests can be run either from the command prompt or SBT.
To specify which tests should be included when invoking ScalaTest using the Runner, add the -n option followed either by the name of the tag or by a list of names of tags surrounded by double quotes. To exclude any test by tag, use the -l option. Examples will appear in the sections covering each individual Spec.
SBT cannot currrently invoke the test task with tags, but tags can be used with the test-only task. To run a specific tag for a particular test class, append -- to the name of the class and apply either the -n or -l options. As with the command line, the option is followed by an individual tag name or by a list tag names in double quotes. Each spec has its own methodology of tagging and will be covered appropriately in each individual Spec.
Note
The main reason the test task does not recognize tags is that SBT is still its infancy (ScalaTest and Specs2 are mere toddlers) and each of the built-in testing frameworks supported by SBT must support tagging. At the time of writing, ScalaTest and Specs2 have tagging ability, but ScalaCheck does not.
The following FunSpec is a full example mixing in some Informer, GivenWhenThen, adding pending and ignore and a tag.
src/test/scala/com/oreilly/testingscala/AlbumSpecAll.scala.
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.{Tag, GivenWhenThen, FunSpec}
class AlbumSpecAll extends FunSpec with ShouldMatchers with GivenWhenThen {
describe("An Album") {
it("can add an Artist to the album at construction time", Tag("construction")) {
given("The album Thriller by Michael Jackson")
val album = new Album("Thriller", 1981, new Artist("Michael", "Jackson"))
when("the artist of the album is obtained")
val artist = album.artist
then("the artist should be an instance of Artist")
artist.isInstanceOf[Artist] should be(true)
and("the artist's first name and last name should be Michael Jackson")
artist.firstName should be("Michael")
artist.lastName should be("Jackson")
info("This is still pending, since there may be more to accomplish in this test")
pending
}
ignore("can add a Producer to an album at construction time") {
//TODO: Add some logic to add a producer.
}
}
}
The above is a complete test that contains most of the features in a regular FunSpec. The first test is tagged as a construction test. The first test also mixes in GivenWhenThen informer methods to provide some testing structure. It also contains a regular Informer, and is finally marked as pending. The second test is ignored. Following is the end result of this particular FunSpec.
> ~test-only com.oreilly.testingscala.AlbumSpecAll [info] AlbumSpecAll: [info] An Album [info] - can add an Artist to the album at construction time (pending) [info] + Given The album Thriller by Michael Jackson [info] + When Artist of the album is obtained [info] + Then the Artist should be an instance of Artist [info] + And the artist's first name and last name should be Michael Jackson [info] + This is still pending, since there may be more to accomplish in this test [info] - can add a Producer to an album at construction time !!! IGNORED !!! [info] Passed: : Total 2, Failed 0, Errors 0, Passed 0, Skipped 2
This is a sample of running a test, to only invoke tests with the tag of construction. Note that the ignored test of the FunSpec did not run since it wasn’t tagged with the “construction” tag.
> ~test-only com.oreilly.testingscala.AlbumSpecAll -- -n construction [info] AlbumSpecAll: [info] An Album [info] - can add an Artist to the album at construction time (pending) [info] + Given The album Thriller by Michael Jackson [info] + When Artist of the album is obtained [info] + Then the Artist should be an instance of Artist [info] + And the artist's first name and last name should be Michael Jackson [info] + This is still pending, since there may be more to accomplish in this test [info] Passed: : Total 1, Failed 0, Errors 0, Passed 0, Skipped 1
A WordSpec is another type of Spec available in ScalaTest. WordSpec makes heavy use of the items when, should, and can with the ability to combine these words with any means possible. when, should, and can are methods belonging to String by use of implicit wrapper. Implicit wrappers are Scala’s way of adding functionality to a class. Let’s continue on our musical journey, this time to the Hotel California.
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.WordSpec
class AlbumWordSpec extends WordSpec with ShouldMatchers {
"An Album" when {
"created" should {
"accept the title, the year, and a Band as a parameter, and be able to read
those parameters back" in {
new Album("Hotel California", 1977,
new Band("The Eagles", new Artist("Don", "Henley"),
new Artist("Glenn", "Frey"),
new Artist("Joe", "Walsh"),
new Artist("Randy", "Meisner"),
new Artist("Don", "Felder")))
}
}
}
}Of course, the existing code must have some changes to make this new test work. First off, an Act class is created. Act will be a superclass for both an Artist and a new class, Band. Album will be refactored to include multiple acts instead of just one artist. Artist will also be refactored to extend an Act.
src/main/scala/com/oreilly/testingscala/Act.scala.
package com.oreilly.testingscala class Act
src/main/scala/com/oreilly/testingscala/Album.scala.
package com.oreilly.testingscala class Album (val title:String, val year:Int, val acts:List[Act])
src/main/scala/com/oreilly/testingscala/Band.scala.
package com.oreilly.testingscala class Band(name:String, members:List[Artist]) extends Act
WordSpec also gives the developer a different perspective on testing by forcing him to consider many when cases. In the AlbumWordSpec example, conditions are defined before any assertions. Each specification is a sentence in its own right. With the when we declare the subject of the test, followed by a block. The should block can be used with a subject or a condition of the test. For example, we can create another test that declares the subject within a should clause.
src/main/scala/com/oreilly/testingscala/AlbumWordSpec.scala.
"An album" should {
"throw an IllegalArgumentException if there are no acts when created" in {
intercept[IllegalArgumentException] {
new Album("The Joy of Listening to Nothing", 1980, List())
}
}
}
The last example introduces some new concepts. The intercept[IllegalArgumentException] is a method that takes a type parameter and has a block that will trap an exception defined in the type parameter of intercept, in this case, an IllegalArgumentException. If an IllegalArgumentException is not caught within the intercept block the test will fail, stating that the Exception that was expected was not thrown.
Another concept from the last example that may be unfamiliar is List(): _*. This has nothing to do with ScalaTest: it is a Scala workaround that converts any scala.Seq to fit into a varargs declaration. Since Album’s third parameter in its main constructor, acts, is a vararg declaration that accepts one or more Acts we must use the _* construct.
Naturally, the IllegalArgumentException has not been handled. Inserting a Scala require method will prevent any bad data from being applied during object construction.
package com.oreilly.testingscala
class Album (val title:String, val year:Int, val acts:Act*) {
require(acts.size > 0)
}The last example uses informers to verify that the intercept successfully traps the IllegalArgumentException. We can continue to add more lines to the lines, possibly to add more assertions.
FeatureSpec is a test that categorizes a test in a set of features. A feature is simply a single feature of the software that is to be delivered. Each feature will have various scenarios of that feature. Each scenario represents a successful or failed test that defines what the object under the test can or cannot do. The more scenarios per feature, the less doubt remains that an object is unstable.
Each feature must have a unique string to describe the desired feature of the software that is being tested. Each scenario string must also be unique.
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.FeatureSpec
class AlbumFeatureSpec extends FeatureSpec with ShouldMatchers {
feature("An album's default constructor should support a parameter that accepts
Option(List(Tracks)) ") { ... }
feature("An album should have an addTrack method that takes a track and returns
an immutable copy of the Album with the added track") { ... }
}In the above example, AlbumFeatureSpec of course will extend the FeatureSpec trait and the ShouldMatchers trait for the should assertion language. The example FeatureSpec+ is divided up into features. These features are essentially a list of deliverables for the object that is under test.
In this example, an Album should have another constructor that handles an option of a list of new objects called Tracks. The final feature is that an Album should have a method called addTrack that accepts a new Track object and returns another immutable Album instance.
Note
Options in Scala are a near replacement for null. Instead of expressing no result in a return value as null, which was standard in Java and C++, a Scala developer would return None. If there is a value that can be returned it would be wrapped in Some. For example, returning the years the Cleveland Browns won the Super Bowl would return None, and for the San Francisco 49ers, a choice would be Some(1981, 1984, 1988, 1989, 1994). Learning Scala is not in the scope of this book: please refer to Programming Scala for more information on the Option[T] type.
Implementing the first feature, we wish to create a couple of scenarios.
-
An
Albumis given a list of 3 tracks exactly -
An
Albumis given an empty list -
An
Albumis given null
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.FeatureSpec
class AlbumFeatureSpec extends FeatureSpec with ShouldMatchers {
feature("An album's default constructor should support a parameter that
accepts Option(List(Tracks))") {
scenario ("Album's default constructor is given a list of the 3 tracks
exactly for the tracks parameter") {pending}
scenario ("Album's default constructor is given an empty List for the
tracks parameter") {pending}
scenario ("Album's default constructor is given null for the tracks
parameter") {pending}
}
feature("An album should have an addTrack method that takes a track and returns an immutable copy of the Album with the added track") { }
}These examples are drawn out to reenact how test-driven development would look as you’re working with ScalaTest. They also show how pending can be used just to hold a space for the developer to fill in the test later. AlbumFeatureSpec in the above example is intended to add a few scenarios and list them as pending. The next phase will be to start implementing some of these tests.
src/test/scala/com/oreilly/testingscala/AlbumFeatureSpec.scala.
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.FeatureSpec
class AlbumFeatureSpec extends FeatureSpec with ShouldMatchers {
feature("An album's default constructor should support a parameter that accepts
Option(List(Tracks))") {
scenario("Album's default constructor is given a list of the 3 tracks exactly
for the tracks parameter") {
val depecheModeCirca1990 = new Band("Depeche Mode",
new Artist("Dave", "Gahan"),
new Artist("Martin", "Gore"),
new Artist("Andrew", "Fletcher"),
new Artist("Alan", "Wilder"))
val blackCelebration = new Album("Black Celebration", 1990,
List(new Track("Black Celebration"),
new Track("Fly on the Windscreen"),
new Track("A Question of Lust")), depecheModeCirca1990: _*)
album.tracks should have size(3)
}
scenario("Album's default constructor is given an empty List for the
tracks parameter") {pending}
scenario("Album's default constructor is given null for the tracks
parameter") {pending}
}
feature("An album should have an addTrack method that takes a track and
returns an immutable copy of the Album with the added track") {pending}
}
The first scenario is filled in the previous example. The test creates a band depecheModeCirca1990+, and attempts to create an album, blackCelebration, with a list of Tracks. This moment is a good time to look back and judge the API—is it clear and understandable? Is it a clean API?
At this time, you may want to create some feature or scenario with pending after thinking through the design of the Album object—before the brain becomes occupied with other thoughts.
After the design introspection, and jotting down some ideas with feature and scenario, create the Track class (just enough to satisfy the test—no more, no less), and modify the parameters of the Album constructor to satisfy the test and make it pass.
/src/main/scala/com/oreilly/testingscala/Album.scala.
class Album (val title:String, val year:Int, val tracks:Option[List[Track]], val acts:Act*) {
require(acts.size > 0)
def this(title:String, year:Int, acts:Act*) = this (title, year, None, acts:_*)
}
/src/main/scala/com/oreilly/testingscala/Track.scala.
package com.oreilly.testingscala class Track(name:String)
The constructor for Album has been updated to include the parameter for track with a type of Option[List[Track]]. The acts parameter has been moved to the last parameter, since according to the Scala specification, any repeated parameter must be the last of a function or method. In the last example, the Track class was created.
The next example is the implementation for the next scenario: “Album’s default constructor is given an empty List for the tracks parameter.” This time, though, it will use the GivenWhenThen trait to structure the test.
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.{GivenWhenThen, FeatureSpec}
class AlbumFeatureSpec extends FeatureSpec with ShouldMatchers with GivenWhenThen {
feature("An album's default constructor should support a parameter that accepts
Option(List(Tracks))") {
//Lines removed for brevity
scenario("Album's default constructor is given a None for the tracks
parameter") {
given("the band, the Doobie Brothers from 1973")
val theDoobieBrothersCirca1973 = new Band("The Doobie Brothers",
new Artist("Tom", "Johnston"),
new Artist("Patrick", "Simmons"),
new Artist("Tiran", "Porter"),
new Artist("Keith", "Knudsen"),
new Artist("John", "Hartman"))
when("the album is instantiated with the title, the year, none tracks,
and the Doobie Brothers")
val album = new Album("The Captain and Me", 1973, None,
theDoobieBrothersCirca1973)
then("calling the albums's title, year, tracks, acts property should yield
the same results")
album.title should be("The Captain and Me")
album.year should be(1973)
album.tracks should be(None)
album.acts(0) should be(theDoobieBrothersCirca1973)
}
//Lines removed for brevity
}
}In the above example, what gets generated in the output in SBT is a very fluid statement of how the test performed.
An album's default constructor should support a parameter that accepts
Option(List(Tracks))
[info] Scenario: Album's default constructor is given a list of the 3 tracks
exactly for the tracks parameter
[info] Scenario: Album's default constructor is given a None for the tracks
parameter
[info] Given the band, the Doobie Brothers from 1973
[info] When the album is instantiated with the title, the year, none tracks,
and the Doobie Brothers
[info] Then calling the albums's title, year, tracks, acts property should
yield the same resultsFreeSpec is a test that is free of any restraint; the developer can craft it however she sees fit. Each element is of the story line is a string followed by -{. FreeSpec is engineered for the testing developer who wishes not to adhere to any pre-fabricated structure. FreeSpec tests are also attractive for test-driven developers who don’t use or prefer English as a primary testing language.
GivenWhenThen can be useful in FreeSpec just to bring some structure into the test if needed.
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.FreeSpec
class JukeboxFreeSpec extends FreeSpec with ShouldMatchers {
"given 3 albums" - {
val badmotorfinger = new Album("Badmotorfinger", 1991, None,
new Band("Soundgarden"))
val thaDoggFather = new Album("The Dogg Father", 1996, None,
new Artist("Snoop Doggy", "Dogg"))
val satchmoAtPasadena = new Album("Satchmo At Pasadena", 1951, None,
new Artist("Louis", "Armstrong"))
"when a juke box is instantiated it should accept some albums" -{
val jukebox = new JukeBox(Some(List(badmotorfinger, thaDoggFather,
satchmoAtPasadena)))
"then a jukebox's album catalog size should be 3" in {
jukebox.albums.get should have size (3)
}
}
}
"El constructor de Jukebox puedo aceptar la palabra clave de 'None'" - {
val jukebox = new JukeBox(None)
"y regresas 'None' cuando llamado" in {
jukebox.albums should be(None)
}
}
}In FreeSpec, the developer has free reign with the structure. Each statement that doesn’t contain any tests within the block terminates with -{. If the statement will contain the assertions required for the test, then an in keyword is required. In the above example, within the "given some albums" block, three albums of varying genres are instantiated. Within the next block, "when a juke box is intantiated it should accept some albums" a Jukebox is instantiated with the Some list of Albums. Again, each of these statements end in -{ since these blocks do not contain any of the assertions within. The block then a jukebox’s album catalog size should be 3 on the other hand does contain an assertion there that block ends with in { instead of -{.
In the last block of JukeboxFreeSpec the test is written in Spanish, since a FreeSpec and with the exception of the keyword in. It doesn’t force any rules like should, when, etc. in a test. To translate, the first statement of the second test says “The constructor of jukebox should accept the keyword None. The last statement of the second test says “and return None when called” after the test assertion is made.
Just to finish out this section, the production code resulting from these tests has changed. For readers who are keeping track, the following is what JukeBox could look like.
package com.oreilly.testingscala
class JukeBox (val albums:Option[List[Album]]) {
def readyToPlay = albums.isDefined
}For the developer with simple tastes, there is the FlatSpec, a no-nonsense, flat behavior-driven design spec meant to just declare the purpose of the test and implement it. FlatSpec is so named since the test is flat and lined up against the left side of the test. FlatSpec can either be written in a long or short form. First, here is the long form.
package com.oreilly.testingscala
import org.scalatest.matchers.MustMatchers
import org.scalatest.FlatSpec
import org.joda.time.Period
class TrackFlatSpec extends FlatSpec with MustMatchers {
behavior of "A Track"
it should """have a constructor that accepts the name and the length of the
track
in min:sec and returns a joda.time.Period when track.period is
called""" in {
val track = new Track("Last Dance", "5:00")
track.period must be(new Period(0, 5, 0, 0))
}
it must """throw an IllegalArgumentException with the message "Track name
cannot be blank"
if the name of the track is blank.""" in {
val exception = evaluating(new Track("")) must produce
[IllegalArgumentException]
exception.getMessage must be ("requirement failed: Track name cannot be
blank")
}
}The TrackFlatSpec example above extends the FlatSpec trait, and is using MustMatchers for must assertion grammar. In a FlatSpec the declaration of the class under a test occurs first, followed by one or more sentence specifications.
In the above example, the subject is Track. Each sentence specification supports the subject. Each supporting sentence specification starts with the word it followed by should, must, or can. The use of should, must, or can has nothing to do with the MustMatchers or ShouldMatchers trait. Those keywords belong to the FlatSpec trait. Looking at the above example, the first sentence specification uses it should, while the second sentence specification uses it must. At the end of each of sentence specification the word it is used as it is in other ScalaTest traits—to encapsulate the test logic.
Given the above test example, the Track class has changes that need to be made in order to satisfy the test.
package com.oreilly.testingscala
import org.joda.time.format.PeriodFormatterBuilder
class Track(val name: String, val length: String) {
require(name.trim().length() != 0, "Track name cannot be blank")
def this(name: String) = this (name, "0:00")
def period = {
val fmt = new PeriodFormatterBuilder()
.printZeroAlways()
.appendMinutes()
.appendSeparator(":")
.printZeroAlways()
.appendSeconds()
.toFormatter()
fmt.parsePeriod(length)
}
}This implementation uses the idea of a Period in Joda Time. A Period is a period of time that is not tied to any chronological calendar or any time zone. It specifies a length of time without an exact start or end time, and without measuring milliseconds. This is perfect for mesauring and storing a music track’s length of time. The above example also uses a PeriodFormatterBuilder to create a Parser and Printer of the Period. A Parser will convert a String representation like “05:00” to a 5-minute Period. A Printer will do the opposite and convert a Period object into a String represenation. PeriodFormatterBuilder uses a Builder design pattern. The Builder pattern typically starts with an object, and allows the developer to add certain ingredients into that builder to create a custom object. In the above example, PeriodFormatterBuilder is instantiated, and certain elements are added to create the perfect Formatter for our needs. For more information on Period, visit the Joda-Time website. For more information on the Builder design pattern, please refer to Head First Design Patterns by Eric Freeman, Elisabeth Robson, Bert Bates, and Kathy Sierra and published by O’Reilly.
For the developer who fancies classic testing structures. ScalaTest supports JUnit testing using a JUnitSuite trait. The test class must extend JUnitTestSuite to mark the test as a JUnit-style test. The rest is tried-and-true JUnit. To include JUnit in the project, modify the build.sbt file to include the repository location of the latest JUnit library. At the time of this writing, JUnit is at 4.10.
libraryDependencies ++= Seq("org.scalatest" % "scalatest_2.9.2" % "1.8" % "test"
withSources() withJavadoc(),
"joda-time" % "joda-time" % "1.8" withSources()
withJavadoc(),
"junit" % "junit" % "4.10" withSources()
withJavadoc())In the above snippet of the build.sbt file, "junit" % "junit" % "4.10" withSources() withJavadoc()) is added to the Seq of repository vectors. After adding the dependencies required for JUnit, reload and update the project using sbt.
The following sample creates a mutable artist member variable that is used to hold the subject under test, in this case an artist. The subject under test undergoes two distinct tests using a distinct artist that is initialized with the startup() method.
package com.oreilly.testingscala
import org.scalatest.junit.JUnitSuite
import org.junit.{Test, Before}
import org.junit.Assert._
class ArtistJUnitSuite extends JUnitSuite {
var artist:Artist = _
@Before
def startUp() {
artist = new Artist("Kenny", "Rogers")
}
@Test
def addOneAlbumAndGetCopy() {
val copyArtist = artist.addAlbum(new Album("Love will turn you around",
1982, artist))
assertEquals(copyArtist.albums.size, 1)
}
@Test
def addTwoAlbumsAndGetCopy() {
val copyArtist = artist
.addAlbum(new Album("Love will turn you around", 1982, artist))
.addAlbum(new Album("We've got tonight", 1983, artist))
assertEquals(copyArtist.albums.size, 2)
}
}For those unfamiliar with JUnit, it was the first open-source testing framework developed for Java. The original JUnit ran by creating methods that started with the word test. If the JUnit Test Runner encountered a method named testEquality and the method started with the word test, the method would be executed as a test. With the advent of Java 1.5, annotations became popular for Java developers and JUnit started using @Test annotations to let the JUnit Test Runner know that the method is a test. For developers who are feeling particularly nostalgic, there is a JUnit3Suite that can be used to relive the testing methodology from JUnit 3.x.
The @Before annotation is used to tell the runner that the startUp() method in our example is to run before each method of the test is run. Therefore for each test, a new instance of Kenny Rogers will be created before the test. This ensures that each test has a Kenny Rogers of its own. Truly, a benefit.
Not shown here, the @After annotation is used on method that preforms and cleanup after the test. The reason for a cleanup in the above example is there is no need to set the artist to Kenny Rogers again. If that were required, the following code would be used for the cleanup method.
@After
def shutDown() {this.artist = null}The method shutDown() is a kind of hack since the JUnitSuite is not immutable. If you are thinking that making the call def shutDown() {this.artist = _} would’ve worked, it wouldn’t—since within a block the _ is considered a parameter of the function that makes up the method, therefore setting it to null is required. An unfortunate circumstance, but such manipulation is sometimes required in order to make a Scala method with a Java method.
The @Before and @After constructs are known as fixtures. Fixtures provide the setup and tearDown/shutDown methods that are used. Up to this point, fixtures in ScalaTest have not been covered, but we will discuss these strategies later in this chapter.
TestNG is another popular Java-based testing framework, with many more features than JUnit. It brought many new ideas to the Java testing worlds, including DataProviders—which can provide a list of data to a testing method—and groups—which are analogous to tagging in ScalaTest. These are only a few of the features included in TestNG. ScalaTest takes great care to ensure that all the TestNG features work under ScalaTest. First, it’s necessary to include testng in the libraryDependencies setting of build.sbt. Below is a snippet to include the a testng dependency in the Seq of dependency vectors.
libraryDependencies ++= Seq("org.scalatest" % "scalatest_2.9.2" % "1.8" % "test"
withSources() withJavadoc(),
"joda-time" % "joda-time" % "1.8" withSources()
withJavadoc(),
"junit" % "junit" % "4.10" withSources()
withJavadoc(),
"org.testng" % "testng" % "6.1.1" % "test"
withSources() withJavadoc())After completing the reload and update required to bring the testng dependency into the project, the following is a sample of a TestNG test used in ScalaTest using a DataProvider and TestNG groups.
package com.oreilly.testingscala
import org.scalatest.testng.TestNGSuite
import collection.mutable.ArrayBuilder
import org.testng.annotations.{Test, DataProvider}
import org.testng.Assert._
class ArtistTestNGSuite extends TestNGSuite {
@DataProvider(name = "provider")
def provideData = {
val g = new ArrayBuilder.ofRef[Array[Object]]()
g += (Array[Object]("Heart", 5.asInstanceOf[java.lang.Integer]))
g += (Array[Object]("Jimmy Buffet", 12.asInstanceOf[java.lang.Integer]))
g.result()
}
@Test(dataProvider = "provider")
def testTheStringLength(n1:String, n2:java.lang.Integer) {
assertEquals(n1.length, n2)
}
}provideData is a method that returns an Array[Array[Object]] with test data used for the test. provideData is also annotated with @DataProvider and given an arbitrary name: provider. The data provided by the provider will call a test method requesting the data using the dateProvider parameter in a Test annotation. In the same, testTheStringLength method is annotated with Test and requests the data provider named provider. The testTheStringLength will now become two tests—one for each row of data provided by the provider. This strategy cuts boilerplate code down, and gives the test developer and a single point of focus for testing.
Below are the results generated from ArtistTestNGSuite.
> ~test-only com.oreilly.testingscala.ArtistTestNGSuite [TestNG] Running: Command line suite =============================================== Command line suite Total tests run: 2, Failures: 0, Skips: 0 =============================================== [info] ArtistTestNGSuite: [info] ArtistTestNGSuite: [info] - testTheStringLength(Heart,5) [info] - testTheStringLength(Jimmy Buffet,12) [info] Passed: : Total 2, Failed 0, Errors 0, Passed 2, Skipped 0 [success] Total time: 1 s, completed Dec 27, 2011 3:08:48 PM
Of course, what good is a test if you can’t tag for filtering purposes? In TestNG, tagging is called groups, and groups can be leveraged along with ScalaTest to include the groups in a test and run as if they were tags. In the following example, the method testTheStringLength test annotation will also include the group tag word_count_analysis.
@Test(dataProvider = "provider", groups=Array("word_count_analysis"))
def testTheStringLength(n1:String, n2:java.lang.Integer) {
assertEquals(n1.length, n2)
}Running the test again in sbt, this time with the -n switch for test only to include tests with word_count_analysis will run the same results.
> ~test-only com.oreilly.testingscala.ArtistTestNGSuite -- -n word_count_analysis [TestNG] Running: Command line suite =============================================== Command line suite Total tests run: 2, Failures: 0, Skips: 0 =============================================== [info] ArtistTestNGSuite: [info] ArtistTestNGSuite: [info] - testTheStringLength(Heart,5) [info] - testTheStringLength(Jimmy Buffet,12) [info] Passed: : Total 2, Failed 0, Errors 0, Passed 2, Skipped 0 [success] Total time: 1 s, completed Dec 27, 2011 3:07:50 PM
Each test can potentially have the same subject or subjects under test. Each test also has the potential of using the same object dependencies or data used in each test. It makes no sense to constantly set up each of those subjects and their dependencies. A fixture is the ability to create these subjects and their dependenies once and have them be reused in each test. Fixtures can allow either the same instance or different dependencies based on the needs of the test, and can also allow sharing of testing structures to ensure that certain rules pass regardless of the object being used. The Scala language itself has methods to do some of the “fixturing” for the developer, while ScalaTest contains some solutions of its own. Each Spec also has its way of producing these fixtures. JUnit and TestNG integration, which is covered in later sections, also contains its own fixture structures.
First fixture strategy doesn’t require anything from ScalaTest, since the solution is purely a Scala solution. It uses an anonymous object, which is just a fancy term for an object that can be created without a name. The anonymous object, once created, can be reused in every test, and it will generate a brand-new dependency object upon request.
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.FunSpec
class AlbumFixtureSpec extends FunSpec with ShouldMatchers {
def fixture = new {
val letterFromHome = new Album("Letter from Home", 1989,
new Band("Pat Metheny Group"))
}
describe("The Letter From Home Album by Pat Metheny") {
it("should get the year 1989 from the album") {
val album = fixture.letterFromHome
album.year should be (1989)
}
}
}The above specification contains a fixture method that creates an anonymous object in Scala with the variable letterFromHome. Every time the method fixture is called, a new object is always created. This creates a unique fixture for each individual test. Within the test block, the call fixture.letterFromHome will provide a unique Album. If the Album in this case were mutable, any other tests within the Spec would not get the mutated object that was changed in another test.
Just to drive the point home even futher, instead of using an immutable object like Album, the following example will use a mutable collection: AlbumMutableFixtureSpec will use a ListBuffer, which is a mutable list of objects—in this case, a list of albums.
package com.oreilly.testingscala
import org.scalatest.FunSpec
import org.scalatest.matchers.ShouldMatchers
class AlbumMutableFixtureSpec extends FunSpec with ShouldMatchers {
def fixture = new {
import scala.collection.mutable._
val beachBoys = new Band("Beach Boys")
val beachBoysDiscography = new ListBuffer[Album]()
beachBoysDiscography += (new Album("Surfin' Safari", 1962, beachBoys))
}
describe("Given a single fixture each beach boy discography initially
contains a single album") {
it("then after 1 album is added, the discography size should have 2") {
val discographyDeBeachBoys = fixture.beachBoysDiscography
discographyDeBeachBoys += (new Album("Surfin' Safari", 1962,
fixture.beachBoys))
discographyDeBeachBoys.size should be(2)
}
it("then after 2 albums are added, the discography size should return 3") {
val discographyDeBeachBoys = fixture.beachBoysDiscography
discographyDeBeachBoys += (new Album("Surfin' Safari", 1962,
fixture.beachBoys))
discographyDeBeachBoys += (new Album("All Summer Long", 1964,
fixture.beachBoys))
discographyDeBeachBoys.size should be(3)
}
}
}Both tests in the above example will pass. The fixture is a factory that generates a very basic Beach Boys discography and each time fixture.beachBoysDiscography is called a new instance is passed. If instead only an instance of the discography, rather than a fixture, were used, then the discography would be shared and the instance used in one test would be the same used in another test. The results would vary and be inconsistent.
An alternate strategy with ScalaTest is to create a custom Fixture trait in order to ensure that each test gets a unique subject to test. Every trait that is mixed into an object retains it’s own methods and is not shared. For a little catching up with Scala, a trait is much like an interface in Java, except that it is concrete and its member variables will be mixed into the class that extends the the trait. The following example is similar to the above test except that it employs a trait instead of an anonymous object.
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.FunSpec
class AlbumFixtureTraitSpec extends FunSpec with ShouldMatchers {
trait AlbumFixture {
val letterFromHome = new Album("Letter from Home", 1989,
new Band("Pat Metheny Group"))
}
describe("The Letter From Home Album by Pat Metheny") {
it("should get the year 1989 from the album") {
new AlbumFixture {
letterFromHome.year should be(1989)
}
}
}
}Using a trait for the fixture encapsulates all the fixtures required per test. In order to make use of the fixture, within each test, an anonymous instantiation of the trait is required. Above, new AlbumFixture is called to anonymously create an object that extends the AlbumFixture trait. Since the trait is mixed in, anything that extends the trait will have access to its variables, methods, and functions. Therefore, no special magic is required: letterFromHome is obtainable and ready to make assertions about its state.
Outside the strategies that come with the Scala language, ScalaTest has its strategy to guarantee that each test will have its very own instance. The next example uses a OneInstancePerTest trait to provide one instance per test.
package com.oreilly.testingscala
import org.scalatest.matchers.ShouldMatchers
import collection.mutable.ListBuffer
import org.scalatest.{FreeSpec, OneInstancePerTest}
class AlbumListOneInstancePerTestFreeSpec extends FreeSpec with ShouldMatchers
with OneInstancePerTest {
val graceJonesDiscography = new ListBuffer[Album]()
graceJonesDiscography += (new Album("Portfolio", 1977, new Artist("Grace",
"Jones")))
"Given an initial Grace Jones Discography" - {
"when an additional two albums are added, then the discography size should
be 3" in {
graceJonesDiscography += (new Album("Fame", 1978, new Artist("Grace",
"Jones")))
graceJonesDiscography += (new Album("Muse", 1979, new Artist("Grace",
"Jones")))
graceJonesDiscography.size should be(3)
}
"when one additional album is added, then the discography size
should be 2" in {
graceJonesDiscography += (new Album("Warm Leatherette", 1980,
new Artist("Grace", "Jones")))
graceJonesDiscography.size should be(2)
}
}
"Given an initial Grace Jones Discography " - {
"when one additional album from 1980 is added, then the discography size should be 2" in {
graceJonesDiscography += (new Album("Nightclubbing", 1981, new Artist("Grace", "Jones")))
graceJonesDiscography.size should be(2)
}
}
}The above example uses a subject that is a mutable ListBuffer of Album. Each graceJones discography is generated per test. Since this test does indeed function and passes, the assertions prove that each test is getting its own discography. The special secret in this test is the OneInstancePerTest which will create a test suite per test. Therefore each test will have its own subjects and dependencies. The example uses a FreeSpec test. Each Spec in ScalaTest is different, and what is defined as a test in each spec is different. For the last example, a test is considered to be what is contained in the in clause of a test.
To have the best control of what gets initialized as well as what gets torn down with a test, the trait BeforeAndAfter is the elixir to provide such functionality. In the following example, the BeforeAndAfter trait is included, this time with a WordSpec to initialize the Human League’s discography with one album. During the test, another album is added, and an assertion is made on the size of the mutable discography. After the test, the mutable discography is cleared.
package com.oreilly.testingscala
import collection.mutable.ListBuffer
import org.scalatest.{BeforeAndAfter, WordSpec}
import org.scalatest.matchers.ShouldMatchers
class AlbumBeforeAndAfterFixtureSpec extends WordSpec with ShouldMatchers
with BeforeAndAfter {
val humanLeagueDiscography = new ListBuffer[Album]()
before {
info("Starting to populate the discography")
humanLeagueDiscography += (new Album("Dare", 1981,
new Band("Human League")))
}
"A mutable ListBuffer of albums" should {
"have a size of 3 when two more albums are added to the Human League
Discography" in {
humanLeagueDiscography += (new Album("Hysteria", 1984,
new Band("Human League")))
humanLeagueDiscography += (new Album("Crash", 1986,
new Band("Human League")))
humanLeagueDiscography should have size (3)
}
"have a size of 2 when one more album is added to the Human League
Discography" in {
humanLeagueDiscography += (new Album("Romantic", 1990,
new Band("Human League")))
humanLeagueDiscography should have size (2)
}
}
after {
info("Clearing the discography")
humanLeagueDiscography.clear()
}
}Some additional notes regarding the above example—first, the before and after blocks are members of the BeforeAndAfter trait. The trait also guarantees that the member variables of the Spec, in this case a WordSpec, are unique per each test. In a WordSpec the test is also defined by the in block. When running the example, notice the number of times a discography is initialized and also torn down. There are two because, like the OneInstancePerTest, a separate suite is created and used per test; therefore, there will always be more than one before and after invocation.
2. Waiting for source changes... (press enter to interrupt)
[info] Compiling 1 Scala source to /home/danno/testing_scala_book.git
/testingscala/target/scala-2.9.2/test-classes...
[info] AlbumBeforeAndAfterFixtureSpec:
[info] A mutable ListBuffer of albums
[info] + Starting to populate the discography
[info] - should have a size of 3 when two more albums are added to
the Human Discography
[info] + Clearing the discography
[info] + Starting to populate the discography
[info] - should have a size of 2 when one more album is added to
the Human Discography
[info] + Clearing the discography
[info] Passed: : Total 2, Failed 0, Errors 0, Passed 2, Skipped 0ScalaTest is an excellent testing framework. A study of the internals of the framework is also a great way to understand Scala, and to understand the language itself. ScalaTest’s matching language is intuitive, and the different specification allows the test-driven developer to choose his own testing style. ScalaTest’s integration with JUnit and TestNG is also valuable for the developer who wishes to bring tests from Java into Scala, making it a great entry point from Java to Scala. There is more to learn about ScalaTest and that is the integration with ScalaCheck as we will see in the last chapter of the book.
[3] Everything is treated as an object and an object reference in Scala, but in compiled bytecode all bets are off.