Creating StreamingContext

A new StreamingContext can be created in two ways:

  1. Create a StreamingContext using an existing SparkContext as follows:
      StreamingContext(sparkContext: SparkContext, batchDuration: Duration)
scala> val ssc = new StreamingContext(sc, Seconds(10))
  1. Create a StreamingContext by providing the configuration necessary for a new SparkContext as follows:
      StreamingContext(conf: SparkConf, batchDuration: Duration)
scala> val conf = new SparkConf().setMaster("local[1]")
.setAppName("TextStreams")
scala> val ssc = new StreamingContext(conf, Seconds(10))
  1. A third method is to use getOrCreate(), which is used to either recreate a StreamingContext from checkpoint data or to create a new StreamingContext. If checkpoint data exists in the provided checkpointPath, then StreamingContext will be recreated from the checkpoint data. If the data does not exist, then the StreamingContext will be created by calling the provided creatingFunc:
        def getOrCreate(
checkpointPath: String,
creatingFunc: () => StreamingContext,
hadoopConf: Configuration = SparkHadoopUtil.get.conf,
createOnError: Boolean = false
): StreamingContext
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.15.3.136