Chapter 8. Testing
- Integration testing with Specs2
- Unit testing with Specs2
- Testing with ScalaTest
By now, you’ve had a thorough introduction to the basic features of Scalatra. As good test-first developers, we feel a little guilty about waiting until now to introduce testing, but here we are.
Scalatra comes with a test DSL to simplify testing your application. The DSL follows the Scalatra philosophy of not being particularly opinionated. It works with both major Scala test frameworks, ScalaTest and Specs2. It’s suited to both integration and unit testing. And if it doesn’t fully suit your needs, it’s easy to pull out the pieces that do. It’s time to write your first test.
8.1. Integration testing with Specs2
Scalatra is built on the Java Servlet API. This design permits Scalatra to sit atop servers like Jetty and Tomcat, the most mature servers on the JVM. It also complicates testing for a few reasons:
- The central method of the Servlet API returns Unit.[1] With no return value to inspect, you have to intercept the response object.
In Java, it’s called void.
- The API has a large surface area that’s difficult to stub. In version 3.0 Http-ServletRequest alone has 66 methods.
- Mocking a container is difficult because of the peculiar rules of the Servlet specification. Calling certain methods of HttpServletResponse is illegal at certain times, sendError throws an exception when called after the response is committed, and setHeader is ignored in the same state. You can’t call getWriter and getOutputStream on the same response. Simulating these rules correctly would be as daunting as building a new container.
For these reasons, the servlet layer of an application is notoriously difficult to test and is therefore notoriously undertested. Talking directly to your Scalatra servlet isn’t a viable approach.
Scalatra’s test DSL takes a different tack: it embeds a live servlet container and speaks to it through an HTTP client. It sounds a bit bulky, but in the next section you’ll see how simple it really is.
8.1.1. Getting started with Specs2
Like Scalate in chapter 7, Specs2 is integrated as a module. This involves an extra line in project/build.scala. Users of the giter8 template will find it already configured:
One important difference between scalatra-specs2 and other dependencies is the % "test" at the end of the dependency. This declares the dependency to be in test scope. Libraries in test scope are only available during test runs. Tests aren’t meant to be deployed, so this separation enforces the divide and slims down the deployable artifact.
Also included in the giter8 template is a simple test named HelloWorldSpec. Tests are found in src/test/scala.[2]
Code in src/test/scala is compiled against the test scope described previously and is also excluded from the deployable artifact.
Let’s test a simple food servlet. It returns product information for potatoes, in JSON format. First, add the dependency to Json4s and Scalatra’s JSON integration:
"org.scalatra" %% "scalatra-json" % ScalatraVersion, "org.json4s" %% "json4s-jackson" % "3.3.0",
Next, add the servlet, shown in the following listing.
Listing 8.1. The FoodServlet
package org.scalatra.book.chapter08
import org.json4s.DefaultFormats
import org.json4s.JsonDSL._
import org.scalatra.ScalatraServlet
import org.scalatra.json._
class FoodServlet extends ScalatraServlet with JacksonJsonSupport {
implicit lazy val jsonFormats = DefaultFormats
get("/foods/potatoes") {
val productJson =
("name" -> "potatoes") ~
("fairTrade" -> true) ~
("tags" -> List("vegetable", "tuber"))
productJson
}
}
Last, mount the FoodServlet in ScalatraBootstrap:
context.mount(new FoodServlet, "/*")
Rename HelloWorldSpec to FoodServletSpec, and make it look like the next listing.
Listing 8.2. Immutable Specs2 test
There’s a lot going on here in a relatively small amount of code. The first thing to notice is that you define a class that extends ScalatraSpec. Just as ScalatraServlet enables the DSL for your application, ScalatraSpec enables the DSL for your test.
After that comes an is method with some curious syntax. A Specs2 acceptance test is a literate specification: the text on the left declares the intent in a format you can discuss with your non-technical product manager to agree on the specification, and the bindings to the code that implement the test are swept off to the right. It’s a foreign syntax to those coming from the xUnit family of test frameworks, and it seems like overkill with a single assertion, but as a specification grows larger, the syntax keeps front and center exactly what you’re trying to accomplish.
The s2 syntax uses Scala’s customizable string-interpolation features to cleanly separate the descriptions from the code. The s interpolator in the standard library is common in Scala code. A string literal prefixed with s may embed Scala expressions with ${} syntax. For example, s"Byte.MaxValue == ${Byte.MaxValue}" results in String"ByteMaxValue == 127". The interpolated code is typechecked just like any other Scala code, preventing silly runtime mistakes.
Scala goes one step further, allowing libraries to provide custom String interpolators. Specs2’s is method doesn’t return a String but a special Specs2 data structure called Fragments to describe the test. The s2 syntax converts the String literal, with embedded code, into the necessary Fragments. More on the motivation for using this syntax can be found in Eric Torreborre’s article “Specs2 2.0 - Interpolated” on his blog (http://mng.bz/0AOC).
Next, you add your servlet. It’s as simple as providing the class reference and the path to mount it to. The path specification is identical to what you find in ScalatraBootstrap.
Finally, we get to the test. The get("/foods/potatoes") block is designed to resemble the corresponding Scalatra route that matches it. But the separate purposes of the core framework and test framework give rise to separate rules:
- Path parameters aren’t supported. You can specify a query string, which you’ll see later. The intent is not to define a broad matcher like a route, but rather an exact path for a single test case.
- Similarly, Boolean guards and regular expressions aren’t supported. Neither of these fit with the idea of specifying a single path for a request.
Inside the get block is the assertion of your test—that the status must equal 200. Review the code from listing 8.2 again, and read it out loud: status must_== 200. The code reads exactly as you’d describe the test. This is another example of Specs2’s philosophy of literate specifications.
The status code is just one part of a response. In the next specification, you’ll enhance the test to cover all the basics of the response.
8.1.2. Asserting over the entire response
An HTTP response can be thought of as being composed of three main parts:
- The status
- The headers
- The body
You’ve already seen how to assert the correct status code. Let’s now enhance the test to test the other two parts.
Listing 8.3. Testing headers and the body
These examples hardly require any explanation, but they’re useful to see in action. The literate style is easy to read, but because Scala isn’t a natural language parser, it does require some knowledge to write correctly.
Tip
We discussed triple-quoted string literals in section 3.3.2. Specs2 descriptions like the one in listing 8.3 frequently contain characters like quotation marks that benefit from the same escaping rules as regular expressions. We recommend using the """ syntax in s2.
One interesting note is the use of Specs2’s matchers. You’ve already seen must_==. This example introduces startsWith and contain. You could write everything as an equality check, but you’re not particularly interested in the charset of the content type nor the rest of the response body. Overspecification makes tests brittle. The rich matcher vocabulary lets you say exactly what you mean, and no more.
The test is still a bit unsatisfying. The body is JSON, but you’re testing it as a String. Even with the contain matcher, the test could fail for whitespace issues on structurally equivalent JSON. In the next section, you’ll test the JSON directly.
8.1.3. Testing as JValues
The Json4s object model was introduced in chapter 5. It would be nice to use that model directly. The following listing parses the body to a JValue and specifies the test in terms of JSON instead of a JValue.
Listing 8.4. Testing headers and the body as JValues
This is much nicer. Instead of making assertions about the output string, you make assertions about the JSON. The character is an operator provided by Json4s to find the child element of a JSON object. You could instead use \ to recursively search json for a key named name, but in this case you expect it to be a child of the root.
The result of is another JValue. In this case, you expect a String, which is represented in Json4s as a JString. So you assert that the name equals JString("potatos").
For a service that returns JSON, the JsonMethods.parse(body) will be repeated often. Let’s DRY that up with a helper trait to mix into all the tests.
Listing 8.5. Testing headers and the body with a helper trait
The JsonBodySupport takes advantage of a Scala feature called self-types. You specify that any class that extends JsonBodySupport must also be a subtype of ScalatraTests. This gives your trait access to all the members of ScalatraTests, such as body.
ScalatraSpec is a subtype of ScalatraTests, and it could be used just as well in the subtype. But by using the more abstract ScalatraTests, your helper can also be mixed into ScalaTest suites, which we’ll introduce in section 8.3. The self type allows your JsonBodySupport to refer to members of ScalatraTests, such as body.
Your new JsonBodySupport can now be mixed into any Scalatra test for easy testing of a JSON service. Similar techniques can be used to support XML, HTML, or whatever other text-based format may be prevalent in your service.
Tip
Most responses are some form of text, but you may wish to test a binary output format, such as a protocol buffer, an image, or a compressed response. bodyBytes, built into ScalatraTests, is the equivalent of body, but returns an Array[Byte].
In the next section, you’ll learn to run your tests, and we’ll also bring relief to any astute readers who caught the typo intentionally propagated through this section.
8.1.4. Running your tests
You’ve already seen how sbt compiles your code, and you’ll learn in chapter 9 how it packages your code for deployment. It’s also a great way to run tests. Just run sbt test from the command line.
The output is shown in the next listing. Framework logging is omitted for brevity.
Listing 8.6. A successful sbt test run
You have your first test and, not unexpected1y, your first test fai1ure. Cheer up: this is a good thing! It’s good practice to write the test, see it fai1, and then imp1ement (or fix) the main code. Seeing the test switch from fai1ing to passing with a change to the main code serves as a sort of test of the tests, warding off fa1se positives.
Now you need to make the test pass. The sample output shows that you have been expecting potatos when the correct output is potatoes. You can fix the assertion:
def potatoesName = get("/foods/potatoes") {
val json = JsonMethods.parse(body)
json "name" must_== JString("potatoes")
}
Now run sbt test from the console again, and you should see the following output:
The tests pass. Celebrate with the beverage of your choice, and then we’ll talk about white-box testing.
8.2. Unit testing with Specs2
The tests in the previous section were black-box tests. The specification considers only the inputs (requests) and outputs (responses) without any regard to implementation. The route could be a simple XML literal, or it could cheat and make a system call to spin up a Sinatra server and proxy the result. As long as the response matches the specification, the tests pass. This is a useful mode of operation for integration testing, when you want to test the behavior of the entire application. It’s not so good for unit testing.
Unit tests are so named because they test a single unit of the code, rather than the entire application. Why is this advantageous?
- They’re easier to write. Working with a single unit requires less setup.
- They’re easier to deploy. If your tests don’t hit the live database, you don’t need to worry about granting your test box access to the live database.
- They run faster. It’s faster to go through one layer than all the layers.
Let’s look at a unit-testing example by building a new service that launches nukes.
8.2.1. Testing with stubbed dependencies
You may be concerned that the sample application will result in the extinction of humankind as soon as you submit a successful request to launch the nukes. Worry not, because you’re going to stub out the dependency. First you need the business logic, shown in the following listing.
Listing 8.7. Stubbed dependencies
Whether it’s a database, a message queue, or a nuclear missile silo, it can be difficult to set up the external resources an integration test needs. Verifying a successful launch via the setting of a var will be a much more pleasant developer experience than inspecting a mushroom cloud. This is an ideal time for a unit test.
On top of the inherent dangers in handling nuclear warheads, astute observers will note that we also left a public var on the trait. This is generally not good design in Scala. Mutable state is hard to reason about, and the object isn’t thread-safe for a concurrent environment.
We stand by it here. This stub implementation isn’t going to be subject to a highly concurrent environment. It needs to be trivially verifiable when accessed by a single thread in a unit test, so you can concentrate on your real business problem, such as the nuke.
Next, you’ll create a servlet to provide an HTTP interface to conveniently launch the nukes from a safe, remote location.
8.2.2. Injecting dependencies into a servlet
In order to unit test your servlet, you need to be able to swap in different implementations of the nuke launcher. The simplest way to do this is through constructor injection. Your servlet will take a single constructor parameter, as shown in the following listing.
Listing 8.8. Nuke launcher servlet
Accepting an instance of the NukeLauncher trait makes it simple to configure this servlet for either test or production. First, here’s an example production ScalatraBootstrap file with the live launcher.
Listing 8.9. Example ScalatraBootstrap with live dependency
Because this is a white-box test, you can assume that the servlet delegates to a NukeLauncher. As such, it’s fair to swap in a stub instance for testing, as shown in listing 8.10. This test will introduce the unit test syntax of Specs2.
Listing 8.10. A unit specification for the Nuke launcher
The post call is the first time you’ve submitted parameters. post takes a variable argument list of String->String tuples and passes them as parameters in the form body. get has a similar signature for query parameters. In fact, all the HTTP methods take several overloads to handle the common use cases of parameters, headers, and bodies. These are best learned by looking at the Scaladoc.
Because the same post to /launch appears three times in the test, it’s extracted to a launch helper. A broken test is a broken build, so test code needs to be maintained as surely as production code. Factoring out the duplicate code, even in tests, will tend to save time in the long run.
The other new feature in listing 8.10 is the sequential declaration. By default, Specs2 runs each test in a spec in parallel. This can speed up test runs when all code under test is immutable. But when an object under test, like the StubNukeLauncher, is mutable, this parallelism results in non-deterministic test failures. The right passcode may be submitted in a different thread while the wrong passcode is being asserted. In cases where it’s impossible or awkward to write in a pure functional style, the sequential declaration offers a simple way out. Because sbt can run separate specs in parallel, as long as no state is shared between specs, sequential may not even add significantly to test time.
You can now run your unit test. The stub successfully isolates your tidy API from its burdensome dependency to allow more thorough tests than you otherwise might have written.
The Specs2 documentation calls the two forms of testing acceptance and unit tests. The acceptance tests are immutable and require a single matcher result to be returned. The unit tests run in a mutable context and are more flexible about where matching is done.
A thorough examination of the differences is included in the official Specs2 documentation. We follow the convention here to demonstrate both. In practice, people tend to pick the syntax they like better and stick to it.
Having seen the major features of Specs2, we’ll now take a look at ScalaTest.
8.3. Testing with ScalaTest
ScalaTest is an alternative test framework to Specs2. Both are mature, both are flexible, and both are maintained by highly responsive developers. The differences are more a matter of taste than of substance.
Scalatra is tested with its own DSL and employs a mix of ScalaTest and Specs2. Scalatra has multiple contributors, and different ones prefer different frameworks. The tests run together in a single report, which is a testament to sbt’s abstract test interface.
Although using both is supported, it’s not recommended. It’s good for Scalatra’s internals, because it helps test the test integrations. But for your project, look at both, learn one well, and know that you can’t go wrong.
8.3.1. Setting up Scalatra’s ScalaTest
Setting up Scalatra’s ScalaTest integration requires minimal changes from the giter8 project. It’s as easy as changing a single line in the library dependencies from scalatra-specs2 to scalatra-scalatest.
Open project/build.scala and make sure libraryDependencies contains the following line:
That’s all the setup required. The next time you reload your sbt project, it will fetch ScalaTest and the scalatra-scalatest integration library. The next thing you need to do is port the existing Specs2 test to ScalaTest.
8.3.2. Your first ScalaTest
Open your ScalatraServletSpec file from listing 8.2, and compare it to the ScalaTest port in the following listing.
Listing 8.11. A ScalaTest WordSpec
The WordSpec of ScalaTest and the mutable specification of Specs2 are similar in how they intermingle the descriptions with the implementations. This is just one of the many syntaxes that ScalaTest supports. Another worth looking at is ScalatraFunSuite, where developers coming from xUnit will feel at home.
Some matchers, like the equality test, changed from Specs2. Others, like startsWith, did not. Like Specs2, ScalaTest comes with an expressive, literate library of matchers. But, again, Scala is a compiled language, not a natural language, so you have to learn the precise vocabulary of the test framework you choose.[3]
Matcher differences are the greatest source of confusion when one works in both Specs2 and ScalaTest on a regular basis; hence, our recommendation to pick just one for your organization.
Two very important things didn’t change in this port: the get blocks that define the request, and the references to the parts of the response inside the header. These come from the abstract Scalatra test framework and operate in exactly the same fashion in both ScalaTest and Specs2.
That was easy. Now it’s time to run your ScalaTest.
8.3.3. Running ScalaTest tests
Once again, run sbt test. Just as Scalatra abstracts over the test frameworks, so does sbt. The same command works for both.
An example run follows:
[info] FoodServletWordSpec: [info] GET /foods/potatoes on FoodServlet [info] - must return status 200 [info] - must be JSON [info] - must should have name potatoes [info] ScalaTest [info] Run completed in 1 second, 187 milliseconds. [info] Total number of tests run: 3 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [info] Passed: Total 3, Failed 0, Errors 0, Passed 3 [success] Total time: 1 s, completed Aug 18, 2014 3:03:00 AM
The formatting of the test descriptions differs a bit from what you saw in listing 8.6, but the test counts and success lines are the same as in Specs2. From a tooling perspective, it matters little.
8.4. Summary
- Scalatra’s test DSL is similar to the main DSL you use in your application code. This makes it easy for any Scalatra developer to write tests.
- Scalatra offers first-class support for both of Scala’s major test libraries, Scala-Test and Specs2. You can use a wide variety of testing styles.
- Run your application inside the embedded container to integration-test your application easily.
- Alternatively, wiring up your Scalatra servlets with stub dependencies via constructor injection allows you to isolate your HTTP logic with unit tests.