Chapter 4. Working with user input

This chapter covers

• Executing code on the server using actions
• Query string, path, and form parameters
• Dealing with headers and cookies
• Before and after filters

In the previous chapter, we focused on route matchers. You saw how Sca1atra takes an incoming UR1, decomposes it into its constituent parts, and matches it to a b1ock of code in your app1ication. But route matching is on1y part of the overa11 request-response cyc1e. We’11 give you the who1e 1ife story of a request in this chapter so you can start responding to incoming HTTP requests and do some usefu1 work. 1et’s start with an overview.

4.1. The life of a request

In Scalatra, each incoming HTTP request that hits your server is answered with an HTTP response. What happens when an HTTP request hits your application?

First, Scalatra checks the URL’s path (including path parameters) to see whether it has a matching route. Assuming a matching route is found, the following things happen:

1.  Params are extracted and placed in a params map, so that you can use them inside the route’s action.

2.  before filters are executed, so you can do preprocessing.

3.  The body of the route’s action code is executed.

4.  after filters are executed.

5.  The response is written and returned to the requesting agent.

If no matching route is found, Scalatra will run a special method called notFound, so you can customize the response. Usually this involves setting a 404 response status, but you’re free to do whatever you like.

You saw how routing works in chapter 3, so let’s move on to look at the route body, which is called an action in Scalatra.

4.2. Routes and actions

The URL http://localhost:8080/hackers/ada-lovelace might map to the following route.

Listing 4.1. Retrieving info about specific hackers

The block of code inside a route matcher is called action code or an action. In a forms-based CRUD application, you might retrieve some data and then render a view inside the action. In an API, you might accept incoming data, use it to create a new object, and then return the newly created object as JSON. In either case, the action is where your application goes to work for you.

One of the things that your actions need is a way to read incoming HTTP input, so that your program can react appropriately. There are multiple ways that an HTTP application typically receives input. Let’s take a look at how Scalatra handles HTTP parameters.

4.3. HTTP parameter handling in Scalatra

HTTP parameters are key-value pairs, with the key and value separated by an equals sign: key1=first-value&key2=second-value. Parameters can be received by your application in several different ways, which we’ll go over in a moment. Once your application receives HTTP parameters as input, they get turned into a Scala Map, and they become available for you to use in your action code using the params method.

Note

If you’re new to Scala, a Map is like Ruby’s Hash, Python’s dict, JavaScript’s object, or Java’s Map. It’s a data structure that stores and retrieves values by key.

If your application receives the parameters key1=hello&key2=world, you’ll be able to access the values hello and world by calling params("key1") and params("key2") inside your route actions. When accessed using the params method, all parameters are of type String unless you explicitly force them to be something different.

Now let’s turn our attention to the various ways you can read HTTP parameters. Excluding file uploads for the moment (we’ll cover those in chapter 6), there are three kinds of HTTP parameters that you’ll typically want to read inside your actions:

• Query string parameters
• Path parameters (sometimes known as route parameters)
• Form parameters (sometimes inaccurately called POST parameters)

Let’s take a look at how some familiar real-world applications use parameters, and then look at how to model them in a Hacker Tracker application.

4.3.1. Query string parameters

Query string parameters live on the query string of the request URL. Figure 4.1 shows query string parameters for an example YouTube search, visible in the browser’s address bar. In the Hacker Tracker application, a route and action for a similarly structured search URL could look like the following listing.

Figure 4.1. A YouTube search with highlighted query string parameters

Listing 4.2. An action that reads query string params

If the application is running locally, and you hit the URL http://localhost:8080/results?search_query=scalatra&oq=scalatra, Scalatra reads the incoming HTTP parameters from the request and makes them available in the params map. In this example, the value of val searchQuery will be the string "scalatra".

4.3.2. Path parameters

Also called route parameters, path parameters are incorporated directly into the full path of the URL. If you’ve been programming for any length of time, you’ve almost certainly seen path parameters in action in the URLs of GitHub, the popular code-hosting service. Path parameters in a GitHub URL are highlighted in figure 4.2.

Figure 4.2. GitHub uses path parameters heavily in its repository browser.

The Hacker Tracker application already has a route with a path parameter in it. Let’s make parameter handling more explicit by assigning the incoming :slug parameter to a value, as follows.

Listing 4.3. An action that reads a path parameter

The route matcher here defines a path parameter called slug, which is denoted by the colon (:) in the route declaration. For each path parameter, Scalatra extracts the key and value of the path parameter from the URL’s full path and makes it available to you in the params map.

Given the URL http://localhost:8080/hackers/ada-lovelace, val slug would equal "ada-lovelace".

4.3.3. Form parameters

Form parameters aren’t carried around in the URL path or in the query string. Instead, they’re part of the body of an HTTP request. They’re often called POST parameters, because web programmers are used to thinking of HTTP POST requests as carrying this type of input, but it’s worth noting that PUT and PATCH requests can also carry form parameters.

To try posting some form parameters to your application, you could either make an HTML form with its action set to an HTTP route in your application, or use the command line. The following listing shows a request to create a new hacker in the Hacker Tracker, which uses the curl command-line utility to send the parameters name and motto to the server.

Listing 4.4. Using the curl utility to add a hacker to the tracker

Assuming you’ve got a *nix terminal and have curl installed, you should be able to copy and paste that code into your terminal to send a request to a Scalatra application running on localhost. It will pack up the parameters name and motto and send them to your application.

Tip

Curl (pronounced “see url”) is a command-line utility for making requests and inspecting responses. If you’re using a Unix-based system, chances are you already have it. If you don’t, you should seriously consider installing it, as it makes developing and experiencing RESTful APIs so much easier by exposing a command-line interface for everything that is HTTP. (Curl is also available on Windows through cygwin.)

Here’s a route and action that can read the inbound name parameter.

Listing 4.5. An action that reads form params

As with the other parameter types, form parameters are key-value pairs, and they become available inside your Scalatra actions in exactly the same way as path or query string parameters do: you use the params method to access incoming input. Note that although curl URL-encoded the value of name (so that it was sent as Grace%20Hopper), Scalatra automatically decodes incoming parameters so that val name contains the decoded string "Grace Hopper".

4.3.4. Params versus multiParams

It’s entirely possible to send multiple parameter values that share the same key. For instance, you might get query params on a request like the following.

Listing 4.6. A request with duplicate parameter keys

GET http://localhost:8080/tagged?tag=linguistic&tag=mustache

Note that the parameter key tag is in there twice. This is an entirely legal set of HTTP parameters, and indeed this sort of thing can be very useful if you want to build up arrays of parameters in some situation, such as tags or check box values.

Accessing these parameters in an action using params("tag") will retrieve only the first value. But there’s another method, multiParams("tag"), that will retrieve all the available values of tag and return them in a Scala sequence. In Scala, a Seq is a kind of iterable that has a length and whose elements have fixed index positions, starting from 0.

Let’s write a route and action that will read the incoming tag parameters from listing 4.6 and show what happens when you access them via both params and multiParams.

Listing 4.7. The difference between params and multiParams

If you run the example, you’ll see that printing params results in only the first value: linguistic. But when you loop through and print each element in the multiParamsSeq, you get all values matching the duplicate key: linguistic and mustache.

4.3.5. Dealing with unexpected input

Whenever you accept arbitrary input, you open up the possibility that a user, or another computer system, will send parameters that you don’t expect. For example, in our previous examples of using HTTP parameters via the params method, there’s no code to check either the existence or the type of incoming params.

In other words, it’s possible that someone might hit the route and action in listing 4.8 with the properly formatted URL that you expect.

Listing 4.8. An accident waiting to happen

The preceding code is expecting a request like this:

$ curl localhost:8080/results?search_query=larry

On the other hand, you might get a request like this:

$ curl localhost:8080/results

This time, the URL is missing the search_query query string parameter and the oq parameter. When it comes time to access the nonexistent search_query parameter key, Scalatra will throw an exception. If you attempt to view the page in a browser, you’ll get a page like figure 4.3.

Figure 4.3. An exception resulting from bad input

If you take a look at the stack trace, you’ll notice that the culprit for the error is the innocent-looking call to params("search_query"). When you attempt to access a nonexistent key in the params map, Scala doesn’t just return a null value, as might happen in some other languages. Instead, it throws an exception.

Why is Scala so strict in this respect? This is, in fact, one of the primary design goals of the Scala language (not just Scalatra).

An introduction to pattern matching using the Option type

We’re all used to those ugly moments when code that works just fine under some conditions gives back an unexpected error at runtime. This is often due to the program receiving some input it didn’t expect.

In Java, this situation gives you the dreaded NullPointerException, or NPE for short. In C#, you might be told that your Object reference is not set to an instance of an object. Other languages have their own messages, but what they all have in common is that these errors happen at runtime, when it’s possible that your users are actually trying to accomplish something.

Wouldn’t it be nice if you could detect these problems at compile time instead?

Martin Odersky, the designer of Scala, thought so. He designed Scala’s type system in a particularly clever (and elegant) way, allowing you to program in a style that means you may rarely see an NPE. One of the main ways of catching NPEs at compile time rather than runtime is by using the Option type.

Note

Keep in mind that none of what follows is mandatory (as we’ve just proved by generating an NPE), but relying on Scala’s type system will allow you to clean up your code dramatically and reduce your bug count. It’s the recommended programming style in Scalatra.

You can think of Option as a simple truth value. It either is or isn’t. Instead of true and false, though, Option has the subtypes Some and None. Some is a container for some value and None is the opposite: it’s the container for nothing and is used when there’s no Some.

The Option type and its subclasses Some and None give you an alternative to nulls. Instead of confidently declaring that something has the type String, but with the unspoken understanding that sometimes you’ll get back a live grenade instead, you can declare it to have a type of Option[String] and a value of either Some("wrapped value") or None.

If you use Option as a parameter type, it acts as an explicit signal to other coders (or your future self) that a return value of None is possible. In part, this is just a matter of properly communicating your intentions using the type system.

The biggest improvement Option offers over null, however, is that you can’t forget to check for None, because it’s part of the type system and can be checked at compile time. The compiler will force you to first check if you’re dealing with Some, and if you are, to explicitly say what should happen with a None. Only after that does it allow you to access the contained value.

Getting parameters as Options

Currently, params("search_query") gives you the value you’re after directly, but it contains the possibility of a runtime exception if it hits a null. To fix your action’s handling of the search_query parameter, you’ll need a way to access the params map that returns an Option.

What you want to use is params.get. This method, defined on Scala’s Map class, has a return type of Option[String]. You then use Scala’s pattern matching to check whether it’s Some(string) or None.

Listing 4.9. Using Option to handle missing parameters

The match starts pattern matching on an Option[String] returned from a call to params.get("search_query"). In the first case you match the actual value against the pattern Some(message). What that does is just match anything that is Some. By saying Some(search_query), you also tell the pattern to extract the value to search_query. The last line checks for the None case that sparked this discussion in the first place.

Checking the contents

Although the preceding solution works just fine, it ignores one important thing: the data itself. The existence of the search_query parameter and whether there really is a search query in it are completely unrelated matters. The query might be an empty string. Or it might be whitespace.

You can reuse the same code from listing 4.9. Just add a guard on the Some case that checks the message, as in the following listing.

Listing 4.10. Adding a guard on pattern match

First, the pattern matches against the type of the Some. Then, the safeguard (if) lets you make fine-grained decisions based on the value itself. In listing 4.10, the case statements effectively mean, if there’s a message and it’s non-empty, then ...

Providing default values with getOrElse

Often it’s good enough to provide a default value. You could have fixed the error in the search service using params.getOrElse. Keep in mind that Scalatra’s params are really just a standard Scala Map. This means that you can use the standard Scala getOrElse method on the Map class to good effect.

getOrElse tries to get a parameter value, and when it fails to do that, it uses a provided function to produce a default value. The following listing shows an example of how you might use it.

Listing 4.11. Giving parameters default values

get("/results") {
  "You searched for '" + params.getOrElse("search_query", "") + "'"
}

This is a lot shorter than what you saw in listing 4.9 and still manages to keep your code from throwing exceptions when your application doesn’t receive a search_query parameter.

Halting with getOrElse

If you read carefully, you might have noticed that we said getOrElse takes a function to produce the default value. This function is called only if it’s needed.

This means you can do interesting tricks with getOrElse. You could provide a function that logs the missing parameter to a debug log and after that returns a default value:

params.getOrElse("search_query", {
  log.debug("search query missing :( using default")
  ""
})

This is an interesting possibility, but something far more interesting is that you can throw exceptions from the default functions.

This might seem a bit crazy. Wasn’t the origina1 prob1em that your code cou1d unexpected1y throw an exception? We11, yes. But it’s a comp1ete1y different thing when the exception in question is being exp1icit1y thrown and you’re forced by the type system to hand1e it proper1y.

What you can do is use getOrElse and halt together to stop your action whenever it finds a missing parameter. halt is a Scalatra method that throws a HaltException. When this happens, Scalatra renders a response according to the parameters given to the halt method call.

For example, if you wanted to replicate listing 4.9 in a more succinct way, you could rewrite it as follows.

Listing 4.12. Halting with getOrElse

get("/results") {
  val search_query =
    params.getOrElse("search_query",
    halt(200, "Please provide a search query"))

"You searched for '" + search_query + "'"
}

Now that you’ve seen how to do type-safe exception handling and halt execution, let’s delve into type safety for HTTP parameters.

4.3.6. Typed parameters

So far we’ve only talked about String parameters. This has been enough for the simple search service, but in real-world applications you often need something a bit more fine-grained to satisfy the needs of type-safe libraries. Numbers need to be Ints, date strings should be converted to Dates, and decimals should be Doubles or Floats. You might also want to use your own custom types, such as Name or PhoneNumber instead of String.

Writing conversion logic for types can be boring, so Scalatra helps by giving you conversions to some of these primitive types for free, using the params.getAs method.

Using params.getAs

You can use params.getAs like this:^[1]

¹

Yes, we’re keenly aware that float arithmetic and money don’t mix well, but it makes for a simple example.

params.getAs[Double]("price") //=> Option(9.99)

Here, you give the type you want to convert to inside the brackets, and the parameter name like you would normally. The return value will be an Option of that type.

Let’s say that you want to log a message when you’re adding a new hacker to the Hacker Tracker. If the hacker was born before Unix, they’re considered classical. The action should halt if it doesn’t get properly formatted input. The following listing shows an example implementation.

Listing 4.13. Using getAs to get a parameter value with the desired type

As before, you get the name and motto parameters as strings. The birth-year parameter gets pulled out of the incoming params map as an Int, and you prove it by using it to conditionally log the hacker’s status (classical or Unix-era).

One nice thing about getAs is that it handles errors silently for you. If your action receives a value that doesn’t look like an integer (such as "nineteen seventy"), calling getAs[Int]("birth-year") will still return None, just as if the parameter wasn’t there to begin with.

By default, getAs supports conversions to the following types:

• Integer types—Int, Long, Short, Byte
• Floating-point types—Float, Double
• Boolean—The value of true converts to Some(true), false to Some(false), and everything else to None
• java.util.Date—Requires a format string to be set:

  params.getAs[Date]("publishAt"-> "MM/dd/YYYY")

The date format string is explained in more detail in the JavaDocs of SimpleDateFormat, available from Oracle at http://mng.bz/itiM.

Custom types

Scalatra’s type conversion mechanism is fully extensible. Let’s look at an example.

Let’s build a converter to convert name strings to Name instances. A name will be given in the format of LastName, FirstName, and the type converter’s job will be to convert that string to an instance of the following case class:

case class Name(firstName: String, lastName: String)

Divide the work into three parts. The first part, parsing a Name out of a String, is just a matter of using split and doing some cleanup. You can pattern-match the returned Array[String] to extract the lastName and firstName:

def toName(str: String) = str.
  split(',').
  map(_.trim) match {
      case Array(lastName, firstName) =>
      Name(lastName, firstName)
}

str.split(',').map(_.trim) returns an Array of name parts. Using map(_.trim) applies the String#trim() method on each part to strip any surrounding whitespace from the parts.

What will ultimately reach the pattern match will be a two-element Array with the last and first names. For example, for the name Doe, John, the array would look like Array(Doe, John). The pattern Array(lastName, firstName) will match that and extract lastName and firstName out for you.

Next, you need to define the type converter. A type converter is just a function that has the type of TypeConverter[T]. For this example, the type parameter T will be Name. With the toName method already defined, the code needed for the type converter is reduced to this:

val stringToName: TypeConverter[Name] = safe { str =>
  toName(str)
}

That safe { block will catch any exceptions resulting from bad casting attempts and return an Option instead.

In theory, you could stop here and start using the custom type conversion with getAs. But the code would be verbose, because you’d need to write params.getAs[Name] ("name")(stringToName).

To fix this, you can make stringToName an implicit val. A Scala implicit is like a compile-time decorator for doing type conversions. If you try to cast a variable from one type to another, and the compiler doesn’t know what to do, it’ll check in the current scope to see if there are any implicits defined that can handle the type conversion before it gives up and throws a compile-time exception.

Just add the keyword implicit to the definition, and you’re ready:

implicit val stringToName: ...

With the converter in place, all that’s left is utilizing the new converter. The following listing shows the current state of the controller (with print and logging statements taken out).

Listing 4.14. A Scalatra action with custom typed parameters

package com.constructiveproof.hackertracker

import org.scalatra._
import scalate.ScalateSupport
import org.scalatra.util.conversion.TypeConverter

class HackersController extends HackerTrackerStack {

  case class Name(
    lastName: String, firstName: String)

  def toName(str: String) = str.split(',').
    map(_.trim) match {
      case Array(lastName, firstName) =>
        Name(lastName, firstName)
  }

  implicit val stringToName:
    TypeConverter[String, Name] =
      safe { str =>
        toName(str)
      }

  post("/hackers") {
    val name = params.getAs[Name]("name").getOrElse(
      halt(BadRequest("Please provide a name")))

    val motto = params("motto")
    val birthYear = params.getAs[Int]("birth-year").getOrElse(
      halt(BadRequest("Please provide a year of birth.")))

    if (birthYear >= 1970) {
      println("Adding a hacker who was born within the Unix epoch.")
    } else {
      println("Adding a classical hacker.")
    }

    // Create a new hacker and redirect to /hackers/:slug
  }

  get("/hackers/:slug") {
    val slug = params("slug")
    // Retrieve and display info about this hacker
  }

  get("/results") {
    val searchQuery = params("search_query")
    // Search for and display matching hackers
  }

  get("/hackers/tagged") {
    val tags = multiParams("tag")
    // Retrieve and display hackers
    // matching all the given tags
  }
}

This wraps up our discussion of parameter handling in Scalatra. Now that you’ve had an introduction to basic parameter handling, you should be able to confidently grab incoming input from the request.

Now let’s turn our attention to the next parts of the request’s lifespan: before and after filters, reading headers, reading and writing cookies, and writing out a response.

4.4. Filters

Just like its Sinatra forebear, Scalatra allows you to do selective processing before and after hitting the main body of a route action.

Let’s say you’ve got a controller class that looks like the following listing.

Listing 4.15. Filters example

The code in the before filter will run before every matched route in HackersController. Keep in mind that the before filter will not be run if no matching routes are found; a request to /echo/foo/bar, for example, would not match any routes and the before filter would never run.

This before filter does two things:

• It sets a contentType on the response.
• It opens a fake database connection. You’ll see real database connections in chapter 10. For now, you’ll just stub them out.

Next, the action code runs. In this controller, multiple routes are defined: post("/hackers"), get("/hackers/:slug"), get("/results"), and get("/hackers/tagged"). If an incoming request maps to any of these routes, the before filter runs first, and then the route’s action is run. Afterward, the after filter runs.

4.4.1. Selectively running filters

It’s possible to define multiple filters at once, and to run filters selectively. If you wanted to set the content type before every request, but only open and close the database connection when it’s in use on a specific route, you could set up your filters as follows.

Listing 4.16. Multiple and selective filters

Listing 4.16 defines two before filters. The first one will set the contentType before every request.

The other filters defined here, before("/hackers") and after("/hackers"), will run only on the post("/hackers") route. The other routes will not trigger execution of these filters.

4.4.2. Filter conditions

It’s also possible to run (or not run) filters based on fairly complex conditional code. For example, if you wanted to run a before filter for a specific route, but only on POST requests, you could do this:

before("/hackers", request.requestMethod == Post) {
  DataBase.connect;
}

The second argument to the before method is a Boolean condition that checks that the HTTP request verb is POST. It’s possible to use any Boolean expression you can think of to conditionally run filters on your routes.

Filters are a great way to use the Don’t Repeat Yourself (DRY) principle to clean up your code. Next, we’ll look at several other handy helpers.

4.5. Other kinds of user input

Besides HTTP parameters, there are several other kinds of information that you can read from a request, such as request headers and cookies.

4.5.1. Request headers

Sometimes you’ll need to read headers off an incoming request. You can do this using the request.getHeader() method.

For example, if you want to know whether text/html is an acceptable content type for a given request, you can check by doing this:

request.getHeader("Accept").split(",").contains("text/html")

4.5.2. Cookies

You can easily read incoming cookies using the cookies.get method and write them using cookies.update, as shown in the following listing.

Listing 4.17. Reading and writing cookies

Listing 4.17 shows the reading and writing of cookies. The cookies method is available in any of your actions.

The cookies method, like the params method, gives you access to a Scala Map containing available cookie keys and values. As with params, if you want to go back to using null values instead of Option and pattern matching, you can use cookies("counter") to get the value out directly.

4.6. Request helpers

Scalatra also includes some built-in helper methods to accomplish common HTTP-related tasks.

4.6.1. Halting

If you want to immediately stop execution within a filter or an action, you can call halt(). You can also supply an HTTP status, which will be sent back as part of the response: halt(403). Additionally, you can send back a status, a reason, whatever headers you want, and even a response body.

For convenience, you can used named arguments if you’re sending back something complicated, as follows.

Listing 4.18. Halting

Almost anyone hitting the code in listing 4.18 will see the result shown in figure 4.4. But if your name is Arthur, execution will halt, as in figure 4.5.

Figure 4.4. The holy grail

Figure 4.5. Halt with a response body

As a creative exercise, feel free to combine this code with the cookie-counter example from section 4.5.2 so you can detect when Arthur comes back for more.

4.6.2. Redirecting

Another common task in HTTP applications is issuing a redirect. To issue a temporary (301) redirect, you can say redirect("/someplace/else") in any filter or action.

There are no built-in helpers for permanent redirects. If you want to issue a permanent redirect, your best bet is to do something like this:

halt(status = 301, headers = Map("Location" -> "http://example.org/"))

4.6.3. ActionResult

There’s another way to issue HTTP responses (and potentially redirects). Scalatra actions can return an ActionResult. An ActionResult is a conveniently named Scalatra type bundling up a specific HTTP response status, an optional redirect where applicable, and headers.

An example is worth a thousand words. Let’s rewrite listing 4.18 with an Action-Result and include a nicer form of permanent 301 redirect.

Listing 4.19. ActionResult in action

That Forbidden object is an ActionResult that’s built into Scalatra. It will cause the framework to respond to the request with a 403 status.

There are several dozen other ActionResult objects in Scalatra, mapping to most HTTP status codes. Ok maps to a 200 response, Created maps to a 201, Bad-Request maps to a 400, and NotFound maps to a 404. See the Scalatra source code at GitHub (https://github.com/scalatra/scalatra/blob/master/core/src/main/scala/org/scalatra/ActionResult.scala) for a full list.

ActionResults can make your intentions a lot clearer to readers of your code, especially when you return any of the lesser-known status codes.

4.7. Summary

• HTTP-based applications can accept input in a variety of ways, including form parameters, path parameters, and query parameters. Scalatra reads all of these types of parameters from incoming requests using the params function, which is part of the Scalatra DSL.
• The params function returns a Scala Map containing all incoming parameters. The map keys and values are strings.
• Scala’s Option type is one of the language’s key features. You can use it to check your code at compile time, guarding against runtime errors.
• You can execute code before any action in a servlet using the before() function. Combine it with conditional statements if you need to.
• Scalatra has built-in helpers for most HTTP-related tasks, including writing cookies, redirecting to alternate URLs, or halting execution.
• The ActionResult functions can be used when rendering a response. They return the proper HTTP status code for a given situation, and give an English-language explanation of your intentions, which can provide a useful explanation of what’s going on to other programmers (or your future self).