SparkContext is initialized in the Driver JVM. Spark driver can be considered as the master of Spark applications. The following are the responsibilities of Spark Driver program:
- It creates the physical plan of execution of tasks based on the DAG of operations.
- It schedules the tasks on the executors. It passes the task bundle to executors based. Data locality principle is used while passing the tasks to executors.
- Spark driver tracks RDD partitions to executor mapping for executing future tasks.
Spark Driver is the single point of failure for a Spark application. If a driver dies, the application is finished. So, the application needs to be restarted again if the Driver terminates.