Option as a Container

Option is a binary container. It has an item or it doesn’t. It implements the four methods we need.

Here is a simplied version of the Option abstract class in the Scala library (full source here):

sealed abstract class Option[+A] { self =>                      
  ...
  def isEmpty: Boolean = this eq None                           

  final def foreach[U](f: A => U): Unit =
    if (!isEmpty) f(this.get)

  final def map[B](f: A => B): Option[B] =
    if (isEmpty) None else Some(f(this.get))

  final def flatMap[B](f: A => Option[B]): Option[B] =
    if (isEmpty) None else f(this.get)

  final def filter(p: A => Boolean): Option[A] =
    if (isEmpty || p(this.get)) this else None

  final def withFilter(p: A => Boolean): WithFilter = new WithFilter(p)

  class WithFilter(p: A => Boolean) {                           
    def map[B](f: A => B): Option[B] = self filter p map f      
    def flatMap[B](f: A => Option[B]): Option[B] = self filter p flatMap f
    def foreach[U](f: A => U): Unit = self filter p foreach f
    def withFilter(q: A => Boolean): WithFilter =
      new WithFilter(x => p(x) && q(x))
  }
}

The self => expression defines an alias for this for the Option instance. It is needed inside WithFilter below. See SelfTypeAnnotations for more details.

Test if this is actually the None instance, not value equality.

The WithFilter, which is used by withFilter combined with the other operations to avoid creation of an intermediate collection when filtering.

Here’s where the self reference defined above is used to operate on the enclosing Option instance. Using this would refer to the instance of WithFilter itself.

The final keyword prevents subclasses from overriding the implementation. It might be surprising to see the base class refer to derived classes. Normally, in object-oriented design this would be considered bad. However, with sealed type hierarchies, this file knows all the possible subclasses. Referring to derived classes makes the implementation more concise and efficient, overall, as well as safe.

The crucial feature about these Option methods shown is that the function arguments are only applied if the Option isn’t empty. This feature allows us to address a common design problem in an elegant way.

Say for example that you want to distribute some tasks around a cluster, then gather the results together. Suppose you want an elegant way to ignore those tasks that return empty results.

Wrap each task return value in an Option, where None is used for empty results and Some wraps a nonempty result. We want an easy way to filter out the None results. Here is an example, where we have the returned Options in a Vector:

// src/script/scala/progscala3/forcomps/ForOptionsFilter.scala

scala> val options: Seq[Option[Int]] = Vector(Some(10), None, Some(20))
val options: Seq[Option[Int]] = Vector(Some(10), None, Some(20))

scala> val results = for
     |   case Some(i) <- options
     | yield (2 * i)
val results: Seq[Int] = Vector(20, 40)

case Some(i) <- options pattern matches on each element in results and extracts the integers inside the Some values. Since a None won’t match, all of them are removed. We then yield the final expression we want.

As an exercise, let’s work through the translation rules. Try it yourself before reading on! Here is the first step, where we apply the first rule for converting each pat <- expr expression to a withFilter expression:

scala> val results2 = for
     |   case Some(i) <- options withFilter {
     |     case Some(i) => true
     |     case None => false
     |   }
     | yield (2 * i)
val results2: Seq[Int] = Vector(20, 40)

Finally, we convert the outer for x <- y yield (z) expression to a map call:

scala> val results3 = options withFilter {
     |   case Some(i) => true
     |   case None => false
     | } map {
     |   case Some(i) => (2 * i)
     |   case None => -1             // hack
     | }
val results3: Seq[Int] = Vector(20, 40)

The “hack” is there because we don’t actually need the case None clause, because the withFilter has already removed all Nones. However, the compiler doesn’t understand this, so it warns us we’ll risk a MathError without the clause. Try removing this clause and observe the warning you get.

Consider another design problem. Instead of independent tasks where we ignore the empty results and combine the nonempty results, consider the case where we run a sequence of dependent steps, and we want to stop the whole process as soon as we encounter a None.

Note that we have a limitation that using None means we receive no feedback about why the step returned nothing, such as a failure. We’ll address this limitation later.

We could write tedious conditional logic that tries each case, one at a time, and checks the results, but a for comprehension is more concise:

// src/script/scala/progscala3/forcomps/ForOptionsSeq.scala

scala> def positiveOption(i: Int): Option[Int] =
     |   if i > 0 then Some(i) else None

scala> val resultSuccess = for
     |   i1 <- positiveOption(5)
     |   i2 <- positiveOption(10 * i1)
     |   i3 <- positiveOption(25 * i2)
     |   i4 <- positiveOption(2  * i3)
     | yield (i1 + i2 + i3 + i4)
val resultSuccess: Option[Int] = Some(3805)

scala> val resultFail = for
     |   i1 <- positiveOption(5)
     |   i2 <- positiveOption(-1 * i1)       
     |   i3 <- positiveOption(25 * i2)
     |   i4 <- positiveOption(-2 * i3)
     | yield (i1 + i2 + i3 + i4)
val resultFail: Option[Int] = None

None is returned. The subsequent generators don’t call positiveOption, they just pass the None through.

At each step, the integer in the Some returned by positiveOption is extracted and assigned to a variable. Subsequent generators use those values. It appears we assume the “happy path” always works, which is true for the first for comprehension. It also works fine for the second for comprehension, because once a None is returned, the subsequent generators simply propagate the None and don’t call positiveOption.

Let’s look at three more container types with similar properties, Either and Try from the Scala library, and Validated from the Typelevel Cats library. Validated is a more sophisticated tool for sequencing validation steps.

Either: A Logical Extension to Option

We noted that the use of Option has the disadvantage that None carries no information that could tell us why no value is available. Did an error occur? What kind? Using Either instead is one solution. As the name suggests, Either is a container that holds one and only one of two things. In other words, where Option handled the case of zero or one items, Either handles the case of one item or another.

Either is a parameterized type with two parameters, Either[+A, +B], where the A and B are the two possible types of the element contained in the Either. Recall that +A indicates that Either is covariant in the type parameter A and similarly for +B. This means that if you need a value of type Either[Any,Any] (for example, a method parameter), you can use an instance of type Either[String,Int], because String and Int are subtypes of Any, therefore Either[String,Int] is a subtype of Either[Any,Any].

Either is also a sealed abstract class with two subclasses defined, Left and Right. That’s how we distinguish between the two possible elements.

The concept of Either predates Scala. It has been used for a long time as an alternative to throwing exceptions. By historical convention, the Left value is used to hold the error indicator, such as a message string or thrown exception, and the normal return value is returned in a Right.

Let’s port our Option example. It’s almost identical:

// src/script/scala/progscala3/forcomps/ForEithersGood.scala

scala> def positiveEither(i: Int): Either[String,Int] =
     |   if i > 0 then Right(i) else Left(s"nonpositive number $i")

scala> val result1 = for
     |   i1 <- positiveEither(5)
     |   i2 <- positiveEither(10 * i1)
     |   i3 <- positiveEither(25 * i2)
     |   i4 <- positiveEither(2  * i3)
     | yield (i1 + i2 + i3 + i4)
val result1: Either[String, Int] = Right(3805)

scala> val result2 = for
     |   i1 <- positiveEither(5)
     |   i2 <- positiveEither(-1 * i1)   
     |   i3 <- positiveEither(25 * i2)
     |   i4 <- positiveEither(-2 * i3)
     | yield (i1 + i2 + i3 + i4)
val result2: Either[String, Int] = Left(nonpositive number -5)

A Left is returned here, stopping the process.

Note how Left and Right objects are constructed in positiveEither. Note the types for result1 and result2. In particular, result2 now tells us where the first negative number was encountered, but not the second occurrence of one.

Either isn’t limited to this error-handling idiom. It could be used for any scenario where you want to hold one object or another, possibly of different types. However, union types are, such as String | Int, are better for this purpose. Superficially, they appear to serve a similar function, but union types can’t be used in this context, because they don’t have the combinators like map, flatMap, etc.

That raises some questions, though. Why do Lefts stop the for comprehension and Rights don’t? It’s because Either isn’t really symmetric in the types. Since it is always used for this error-handling idiom, the implementations of Left and Right bias towards the right as the “happy path”.

Let’s look how the combinators and some other methods work for these two types, using result1 and result2:

scala> result1    // Reminder of these values:
     | result2
val res6: Either[String, Int] = Right(3805)
val res7: Either[String, Int] = Left(nonpositive number -5)

scala> var r1  = 0
     | result1.foreach(i => r1 = i * 2)
     | var r2  = 0
     | result2.foreach(i => r1 = i * 2)
var r1: Int = 7610
var r2: Int = 0                                            

scala> val r3  = result1.map(_ * 2)
     | val r4  = result2.map(_ * 2)
     |
val r3: Either[String, Int] = Right(7610)
val r4: Either[String, Int] = Left(nonpositive number -5)

scala> val r5a = result1.flatMap(i => Right(i * 2))
     | val r5b = result1.flatMap(i => Left("hello"))
     | val r5c = result1.flatMap(i => Left[String,Double]("hello"))
     | val r5d: Either[String,Double] = result1.flatMap(i => Left("hello"))
     | val r6  = result2.flatMap(i => Right(i * 2))
     |
val r5a: Either[String, Int] = Right(7610)
val r5b: Either[String, Nothing] = Left(hello)              
val r5c: Either[String, Double] = Left(hello)
val r5d: Either[String, Double] = Left(hello)
val r6:  Either[String, Int] = Left(nonpositive number -5)

No change is made to r2 after initialization.

Note the second type for r5b vs. r5c and r5d. Using Left("hello") alone provides no information about the desired second type, so Nothing is used.

The filter and withFilter methods aren’t supported. They are somewhat redundant in this case.

You can infer that the Left method implementations ignore the function and just return their value. Right.map extracts the value, applies the function, then constructs a new Right, while Right.flatMap simply returns the value the function returns.

Finally, here is for comprehension that uses Eithers:

// src/script/scala/progscala3/forcomps/ForEithersSeq.scala

scala> val seq: Seq[Either[RuntimeException,Int]] =
     |   Vector(Right(10), Left(new RuntimeException("boo!")), Right(20))
     |
     | val results3 = for
     |   case Right(i) <- seq
     | yield 2 * i
val results3: Seq[Int] = Vector(20, 40)

// src/script/scala/progscala3/forcomps/RefTransparency.scala

scala> def addInts(s1: String, s2: String): Int = s1.toInt + s2.toInt

scala> def addInts2(s1: String, s2: String): Either[String,Int] =
     |   try
     |     Right(s1.toInt + s2.toInt)
     |   catch
     |     case nfe: NumberFormatException => Left("NFE: "+nfe.getMessage)

scala> val add12a = addInts("1", "2")
     | val add12b = addInts2("1", "2")
val add12a: Int = 3
val add12b: Either[String, Int] = Right(3)

scala> val add1x  = addInts2("1", "x")
     | val addx2  = addInts2("x", "2")
     | val addxy  = addInts2("x", "y")
val add1x: Either[String, Int] = Left(NFE: For input string: "x")
val addx2: Either[String, Int] = Left(NFE: For input string: "x")
val addxy: Either[String, Int] = Left(NFE: For input string: "x")

Try: When There Is No Do

scala.util.Try is structurally similar to Either. It is a sealed abstract class with two subclasses, Success and Failure.

Success is analogous to the conventional use of Right. It holds the normal return value. Failure is analogous to Left, but Failure always holds a Throwable, which is why Try has one type parameter, instead of two, for the value held by Success.

Here are the signatures of these types (omitting some traits that aren’t relevant to the discussion):

sealed abstract class Try[+T] extends AnyRef {...}
final case class Success[+T](value: T) extends Try[T] {...}
final case class Failure[+T](exception: Throwable) extends Try[T] {...}

Try is clearly asymmetric, unlike Either, where the asymmetry isn’t clear from the type signature, it just reflects convention and the convention determined how the combinators were implemented asymmetrically.

Let’s see how Try is used, again porting our previous example. First, if you have a list of Try values and just want to discard the Failures, a simple for comprehension does the trick:

// src/script/scala/progscala3/forcomps/ForTries.scala

scala> import scala.util.{ Try, Success, Failure }

scala> def positiveTries(i: Int): Try[Int] = Try {
     |   assert (i > 0, s"nonpositive number $i")
     |   i
     | }

scala> val result4 = for
     |   i1 <- positiveTries(5)
     |   i2 <- positiveTries(10 * i1)
     |   i3 <- positiveTries(25 * i2)
     |   i4 <- positiveTries(2  * i3)
     | yield (i1 + i2 + i3 + i4)
val result4: scala.util.Try[Int] = Success(3805)

scala> val result5 = for
     |   i1 <- positiveTries(5)
     |   i2 <- positiveTries(-1 * i1)      // FAIL!
     |   i3 <- positiveTries(25 * i2)
     |   i4 <- positiveTries(-2 * i3)
     | yield (i1 + i2 + i3 + i4)
     |
val result5: scala.util.Try[Int] =
  Failure(java.lang.AssertionError: assertion failed: nonpositive number -5)

Note the concise definition of positiveTries. If the assertion fails, the Try block will return a Failure wrapping the thrown java.lang.AssertionError. Otherwise, the result of the Try expression is wrapped in a Success. A more explicit definition of positiveTries showing the boilerplate is the following:

def positiveTries2(i: Int): Try[Int] =
  if (i > 0) Success(i)
  else Failure(new AssertionError("assertion failed"))

The for comprehensions look exactly like those for the original Option example. With type inference, there is very little boilerplate here, too. You can focus on the “happy path” logic and let Try capture errors.

Cats Validator

While using Option, Either, or Try meets most needs, there is one common scenario where using any of them remains tedious. Consider the case of form validation, where a user submits a form with several fields, all of which need to be validated. Ideally, you would validate all at once and report all errors, rather than doing one at a time, which is not a friendly user experience. Using Option, Either, or Try in a for comprehension doesn’t support this need, because processing is short-circuited as soon as a failure occurs. This is where cats.datatypes.Validated provides several useful approaches.

We’ll consider one approach here. First, start with some domain specific classes:

// src/main/scala/progscala3/forcomps/LoginFormValidation.scala

package progscala3.forcomps

case class ValidLoginForm(userName: String, password: String)        

sealed trait LoginValidation:                                        
  def error: String

case class Empty(name: String) extends LoginValidation:
  val error: String = s"The $name field can't be empty"

case class TooShort(name: String, n: Int) extends LoginValidation:
  val error: String = s"The $name field must have at least $n characters"

case class BadCharacters(name: String) extends LoginValidation:
  val error: String = s"The $name field has invalid characters"

A case class with the form fields to validate.

A trait used by other case classes that encapsulate each error.

Now we use them in the following code, where the acronym Nec stands for “non empty chain”. In this context, that means that a failed validation will have a sequence (“chain”) of one or more error objects.

// src/main/scala/progscala3/forcomps/LoginFormValidatorNec.scala
package progscala3.forcomps

import cats.implicits._
import cats.data._
import cats.data.Validated._
import scala.language.implicitConversions

/**
 * Nec variant, where NEC stands for "non empty chain".
 * @see https://typelevel.org/cats/datatypes/validated.html
 */
object LoginFormValidatorNec:

  type V[T] = ValidatedNec[LoginValidation, T]                       

  def nonEmpty(field: String, name: String): V[String] =             
    if field.length > 0 then field.validNec
    else Empty(name).invalidNec

  def notTooShort(field: String, name: String, n: Int): V[String] =
    if field.length >= n then field.validNec
    else TooShort(name, n).invalidNec

  /** For simplicity, just disallow whitespace. */
  def goodCharacters(field: String, name: String): V[String] =
    val re = raw".*s.*".r
    if re.matches(field) == false then field.validNec
    else BadCharacters(name).invalidNec

  def apply(                                                         
      userName: String, password: String): V[ValidLoginForm] =
    (nonEmpty(userName, "user name"),
    notTooShort(userName, "user name", 5),
    goodCharacters(userName, "user name"),
    nonEmpty(password, "password"),
    notTooShort(password, "password", 5),
    goodCharacters(password, "password")).mapN {
      case (s1, _, _, s2, _, _) => ValidLoginForm(s1, s2)
    }

/**
 * This method uses the matching clauses shown rather something like this:
 *   assert(LoginFormValidatorNec("123 45", "678 90") ==
 *     Invalid(Chain(BadCharacters("user name"), BadCharacters("password"))))
 * This is necessary because we use -language:strictEquality, which causes
 * these == expressions to fail compilation!
 */
@main def TryLoginFormValidatorNec =
  import LoginFormValidatorNec._
  assert(LoginFormValidatorNec("", "") ==
    Invalid(Chain(
      Empty("user name"), TooShort("user name", 5),
      Empty("password"), TooShort("password", 5))))

  assert(LoginFormValidatorNec("1234", "6789") ==
    Invalid(Chain(
      TooShort("user name", 5),
      TooShort("password", 5))))

  assert(LoginFormValidatorNec("12345", "") ==
    Invalid(Chain(
      Empty("password"), TooShort("password", 5))))

  assert(LoginFormValidatorNec("123 45", "678 90") ==
    Invalid(Chain(
      BadCharacters("user name"), BadCharacters("password"))))

  assert(LoginFormValidatorNec("12345", "67890") ==
    Valid(ValidLoginForm("12345", "67890")))

Shorthand type alias. ValidationNec will encapsulate errors or successful results.

Several functions to test that fields meet desired criteria. When successful, an appropriate ValidationNec is constructed by calling either of the extension methods on String, validNec or invalidNec.

The apply method uses a Cats function mapN for mapping over the N elements of a tuple. It returns a final ValidationNec instance with all the accumulated errors in an Invalid(Chain(…)) or if all validation criteria were met, a Valid(ValidLoginForm(…)) holding the passed-in field values.

For comparison, see also in the example code, src/main/scala/progscala3/forcomps/LoginFormValidatorSingle.scala, which handles single failures using Either, but following a similar implementation approach.

Without a tool like Validation, we would have to manage the chain of errors ourselves.