In the next example, we see how a case class can help drastically shorten our code size. If we were to store the name and age of an employee, we might need to create a Java bean with two members, one each for name and age. Then we would need to add two getters and two setters for each as well. However, all this and more could be achieved with just a single line of Scala code as in the next example. Check for a case class Employee statement:

package chapter01

import scala.util.Random

object ExampleCaseClasses {

  case class Employee(name: String, age: Int)

  def main(args: Array[String]) {
    val NUM_EMPLOYEES = 5
    val firstNames = List("Bruce", "Great", "The", "Jackie")
    val lastNames = List("Lee", "Khali", "Rock", "Chan")
    val employees = (0 until NUM_EMPLOYEES) map { i =>
      val first = Random.shuffle(firstNames).head
      val last = Random.shuffle(lastNames).head
      val fullName = s"$last, $first"
      val age = 20 + Random.nextInt(40)
      Employee(fullName, age)
    }

    employees foreach println

    val hasLee = """(Lee).*""".r
    for (employee <- employees) {
      employee match {
        case Employee(hasLee(x), age) => println("Found a Lee!")
        case _ => // Do nothing
      }
    }
  }
}

Other things to note in this example:

Other than classes and case classes, Scala also provides another basic type of data container called tuples. Tuples are typed, fixed-size, immutable list of values. By typed we mean that each of the members has a type, and this list is of fixed size. Also once we create a tuple, we can't change its member values, that is, tuples are immutable (think of Python tuples). Each of the member values are retrieved using an underscore followed by a number (the first member is numbered as ._1 ). There cannot be a zero member tuple.

Here is an example:

package chapter01

object ExampleTuples {
  def main(args: Array[String]) {
    val tuple1 = Tuple1(1)
    val tuple2 = ('a', 1) // can also be defined: ('a' -> 1)
    val tuple3 = ('a', 1, "name")

    // Access tuple members by underscore followed by 
    // member index starting with 1
    val a = tuple1._1 // res0: Int = 1
    val b = tuple2._2 // res1: Int = 1
    val c = tuple3._1 // res2: Char = a
    val d = tuple3._3 // res3: String = name
    
  }
}

Scala also has a shell that is also called a REPL (read-eval-print loop) (think of Python's shell, or Ruby IRB, or Perl's PDB). By just typing in Scala, we can access the REPL:

$ scala
Welcome to Scala version 2.11.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_76).
Type in expressions to have them evaluated.
Type :help for more information.

scala> println("Hello World!")
Hello World!

scala> 3 + 4
res1: Int = 7

scala>

Scala REPL is very powerful and convenient to use. We just set up proper classpaths and then we can import all the Java classes we need to play with, right within the Scala REPL. Notice how we use Java's HashMap in Scala REPL:

scala> val m = new java.util.HashMap[String, String]
m: java.util.HashMap[String,String] = {}

scala> m.put("Language", "Scala")
res0: String = null

scala> m.put("Creator", "Martin Odersky")
res1: String = null

scala> m
res2: java.util.HashMap[String,String] = {Creator=Martin Odersky, Language=Scala}

scala> import scala.collection.JavaConversions._
import scala.collection.JavaConversions._

scala> m.toMap
res3: scala.collection.immutable.Map[String,String] = Map(Creator -> Martin Odersky, Language -> Scala)

That is complete Java interoperability. To give you an idea of how Scala looks, we have shown you the very basic Scala code. Additional reading is required to become comfortable with Scala. Hold on to any decent book on Scala, that covers Scala in more detail, and you are good to go. Next we will see how to create a sample SBT project.

SBT is arguably the most common build tool for building and managing a Scala project and its dependencies. An SBT project consists of a build.sbt file or a Build.scala file at the project root, and optionally a project/folder with additional configuration files. However, for most parts it would suffice to have only build.sbt.

Here is what a sample build.sbt file looks like:

$ cat build.sbt
name := "BuildingScalaRecommendationEngine"

scalaVersion :="2.10.2"

version :="1.0"

libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.3.0"

Well yes, that's pretty much it. One thing to be careful with is to keep a single line spacing between the statements. We have defined a project name, project version, Scala version, and we have a complete Scala project up and running for development. Also, you need to create source code folders inside the project folder. The default location for Scala and Java code is src/main/scala and src/main/java respectively.

SBT has its own shell. Just type sbt and you will see:

$ sbt
[info] Loading global plugins from /home/tuxdna/.sbt/0.13/plugins
[info] Set current project to BuildingScalaRecommendationEngine (in build file:/tmp/BuildingScalaRecommendationEngine/code/)
>

There you can type help to see more information on the different commands. For our purposes, we only need these commands:

Given that you have already set up Apache Spark, run Spark in local mode like so:

$ bin/spark-shell

This will give you Scala like REPL, which actually it is, but the interpreter has done Spark specific initialization for us. Let's see a simple Spark program:

scala> sc.textFile("README.md").flatMap( l => l.split(" ").map( _ -> 1)).reduceByKey(_ + _).sortBy(_._2, false).take(5)

res0: Array[(String, Int)] = Array(("",158), (the,28), (to,19), (Spark,18), (and,17))

The preceding code is a word count program that outputs the top five most occurring words in the file README.md.

Some things to note in the preceding example are:

$ export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

$ cd path/to/spark-1.3.0
$ mvn -DskipTests clean package
$ sbin/start-master.sh

Starting Spark master at spark://matrix02:7077

package chapter01
import scala.collection.JavaConversions._
import scala.io.Source
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

import org.apache.spark.mllib.clustering.KMeans
import org.apache.spark.mllib.linalg.Vectors

object IrisKMeans {
  def main(args: Array[String]) {

    val appName = "IrisKMeans"
    val master = "local"
    val conf = new SparkConf().setAppName(appName).setMaster(master)
    val sc = new SparkContext(conf)

    println("Loading iris data from URL...")
    val url = "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"
    val src = Source.fromURL(url).getLines.filter(_.size > 0).toList
    val textData = sc.parallelize(src)
    val parsedData = textData
      .map(_.split(",").dropRight(1).map(_.toDouble))
      .map(Vectors.dense(_)).cache()

    val numClusters = 3
    val numIterations = 20
    val clusters = KMeans.train(parsedData, numClusters, numIterations)

    // Evaluate clustering by computing Within Set Sum of Squared Errors
    val WSSSE = clusters.computeCost(parsedData)
    println("Within Set Sum of Squared Errors = " + WSSSE)
  }
}

$ sbt 'run-main chapter01.IrisKMeans'  2>/dev/null

Loading iris data from URL ...
Within Set Sum of Squared Errors = 78.94084142614648