Extending an RDD

This is a simple test that has a lot of hidden complexity. Let's start by creating a list of the record, as shown in the following code block:

class InheritanceRdd extends FunSuite {
val spark: SparkContext = SparkSession
.builder().master("local[2]").getOrCreate().sparkContext

test("use extended RDD") {
//given
val rdd = spark.makeRDD(List(Record(1, "d1")))

The Record is just a case class that has an amount and description, so the amount is 1 and d1 is the description.

We then created MultipledRDD and passed rdd to it, and then set the multiplier equal to 10, as shown in the following code:

val extendedRdd = new MultipliedRDD(rdd, 10)

We are passing the parent RDD because it has data that was loaded in another RDD. In this way, we build the inheritance chain of two RDD's.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.191.62.122