YARN cluster mode

In the YARN cluster mode, the Driver runs on a node inside the cluster (typically where the application master is). Client first contacts the resource manager requesting resources to run the Spark job. The resource manager allocates a container (container zero) and responds to the client. The client then submits the code to the cluster and then launches the Driver and Spark application master in the container zero. The Driver runs along with the application master and the Spark application master, and then creates the executors on the containers allocated by the resource manager. The YARN containers can be on any node in the cluster controlled by the node manager. So, all allocations are managed by the resource manager.

Even the Spark application master needs to talk to the resource manager to get subsequent containers to launch executors.

The following is the Yarn-cluster mode deployment of Spark:

There is no shell mode in YARN cluster mode, since the Driver itself is running inside YARN.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.144.97.47