Driver program

SparkContext is initialized in the Driver JVM. Spark driver can be considered as the master of Spark applications. The following are the responsibilities of Spark Driver program:

  • It creates the physical plan of execution of tasks based on the DAG of operations.
  • It schedules the tasks on the executors. It passes the task bundle to executors based. Data locality principle is used while passing the tasks to executors.
  • Spark driver tracks RDD partitions to executor mapping for executing future tasks.

Spark Driver is the single point of failure for a Spark application. If a driver dies, the application is finished. So, the application needs to be restarted again if the Driver terminates.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.191.181.36