The developer can decide to persist and/or cache an RDD for future usage. An RDD may persist in memory only or on disk only—in memory if available, or on disk otherwise as de-serialized or serialized Java objects.

For instance, an RDD, rdd, can be cached through serialization through a simple statement, as shown in the following code:

rdd.persist(StorageLevel.MEMORY_ONLY_SER).cache

Scala supports lazy values natively. The left side of the assignment, which can either be a value, object reference, or method, is performed once, that is, the first time it is invoked, as shown in the following code:

class Pipeline {
  lazy val x = { println("x"); 1.5}
  lazy val m = { println("m"); 3}

  val n = { println("n"); 6}   
  def f = (m <<1)
  def g(j: Int) = Math.pow(x, j)
}
val pipeline = new Pipeline //1
pipeline.g(pipeline.f)  //2

The order of the variables printed is n, m, and then x. The instantiation of the Pipeline class initializes n, but not m or x (line 1). At a later stage, the g method is called, which in turn invokes the f method. The f method initializes the value m it needs, and then g initializes x to compute its power to m<<1 (line 2).

Spark applies the same principle to RDDs by executing the transformation only when an action is performed. In other words, Spark postpones memory allocation, parallelization, and computation until the driver code gets the result through the execution of an action. The cascading effect of invoking all these transformations backwards is performed by the direct acyclic graph scheduler.

Spark is implemented in Scala, so you should not be too surprised that the most relevant Scala higher methods on collections are supported in Spark. The first table describes the transformation methods using Spark, as well as their counterparts in the Scala standard library. We use the (K, V) notation for (key, value) pairs:

Action methods trigger the collection or the reduction of the datasets from all partitions back to the driver, as listed here:

Scala methods such as fold, find, drop, flatten, min, max, and sum are not currently implemented in Spark. Other Scala methods such as zip have to be used carefully, as there is no guarantee that the order of the two collections in zip is maintained between partitions.

In a perfect world, variables are immutable and local to each partition to avoid race conditions. However, there are circumstances where variables have to be shared without breaking the immutability provided by Spark. To this extent, Spark duplicates shared variables and copies them to each partition of the dataset. Spark supports the following types of shared variables:

The four design principles can be summarized in the following diagram:

Interaction between Spark driver and RDDs

The preceding diagram illustrates the most common interaction between the Spark driver and its workers, as listed in the following steps:

The easiest way to learn Spark is to deploy a localhost in standalone mode. You can either deploy a precompiled version of Spark from the website, or build the JAR files using the simple build tool (sbt) or maven [12:14] as follows:

Apache supports multiple deployment modes:

The communication between a master node (or driver), a cluster manager, and set of slave (or worker) nodes is illustrated in the following diagram:

Master-worker communication through a cluster manager

Use any of the following methods to use the Spark shell:

The following screenshot captures the output for launching a Spark shell. The standard output dumps: