In this chapter, we will test Apache Spark jobs and learn how to separate logic from the Spark engine.

We will first cover unit testing of our code, which will then be used by the integration test in SparkSession. Later, we will be mocking data sources using partial functions, and then learn how to leverage ScalaCheck for property-based testing for a test as well as types in Scala. By the end of this chapter, we will have performed tests in different versions of Spark.

In this chapter, we will be covering the following topics:

• Separating logic from Spark engine-unit testing
• Integration testing using SparkSession
• Mocking data sources using partial functions
• Using ScalaCheck for property-based testing
• Testing in different versions of Spark