Suppose you have written an application in Scala that can tell you how many words are there in a document or text file as follows:

package com.chapter16.SparkTesting
import org.apache.spark._
import org.apache.spark.sql.SparkSession
class wordCounterTestDemo {
  val spark = SparkSession
    .builder
    .master("local[*]")
    .config("spark.sql.warehouse.dir", "E:/Exp/")
    .appName(s"OneVsRestExample")
    .getOrCreate()
  def myWordCounter(fileName: String): Long = {
    val input = spark.sparkContext.textFile(fileName)
    val counts = input.flatMap(_.split(" ")).distinct()
    val counter = counts.count()
    counter
  }
}

The preceding code simply parses a text file and performs a flatMap operation by simply splitting the words. Then, it performs another operation to take only the distinct words into consideration. Finally, the myWordCounter method counts how many words are there and returns the value of the counter.

Now, before proceeding into formal testing, let's check if the preceding method works well. Just add the main method and create an object as follows:

package com.chapter16.SparkTesting
import org.apache.spark._
import org.apache.spark.sql.SparkSession
object wordCounter {
  val spark = SparkSession
    .builder
    .master("local[*]")
    .config("spark.sql.warehouse.dir", "E:/Exp/")
    .appName("Testing")
    .getOrCreate()    
  val fileName = "data/words.txt";
  def myWordCounter(fileName: String): Long = {
    val input = spark.sparkContext.textFile(fileName)
    val counts = input.flatMap(_.split(" ")).distinct()
    val counter = counts.count()
    counter
  }
  def main(args: Array[String]): Unit = {
    val counter = myWordCounter(fileName)
    println("Number of words: " + counter)
  }
}

If you execute the preceding code, you should observe the following output: Number of words: 214. Fantastic! It really works as a local application. Now, test the preceding test case using Scala JUnit test case.

package com.chapter16.SparkTesting
import org.scalatest.Assertions._
import org.junit.Test
import org.apache.spark.sql.SparkSession
class wordCountTest {
  val spark = SparkSession
    .builder
    .master("local[*]")
    .config("spark.sql.warehouse.dir", "E:/Exp/")
    .appName(s"OneVsRestExample")
    .getOrCreate()   
    @Test def test() {
      val fileName = "data/words.txt"
      val obj = new wordCounterTestDemo()
      assert(obj.myWordCounter(fileName) == 214)
           }
    spark.stop()
}

If you see the earlier code carefully, I have used the Test annotation before the test() method. Inside the test() method, I invoked the assert() method, where the actual testing occurs. Here we tried to check if the return value of the myWordCounter() method is equal to 214. Now run the earlier code as a Scala Unit test as follows (Figure 7):

Now if the test case passes, you should observe the following output on your Eclipse IDE (Figure 8):

Now, for example, try to assert in the following way:

assert(obj.myWordCounter(fileName) == 210)

If the preceding test case fails, you should observe the following output (Figure 9):

Now let's have a look at method 2 and how it helps us for the betterment.