Shared Java and Scala APIs

Once you have a SparkContext created, it will serve as your main entry point. In the next chapter, you will learn how to use our SparkContext instance to load and save data. You can also use the SparkContext instance to launch more Spark jobs and add or remove dependencies. Some of the non-data-driven methods you can use on the SparkContext instance are as follows:

Method

Use

addJar(path)

Adds the JAR file for all future jobs run through the SparkContext instance

addFile(path)

Downloads the file to all nodes on the cluster

stop()

Shuts down the SparkContext connection

clearFiles()

Removes the files so that new nodes will not download them

clearJars()

Removes the JAR files from being required for future jobs

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.142.166.31