Spark on Mesos

Mesos deployment is similar to Spark standalone mode and the Driver communicates with the Mesos Master, which then allocates the resources needed to run the executors. As seen in standalone mode, the Driver then communicates with the executors to run the job. Thus, the Driver in Mesos deployment first talks to the master and then secures the container's request on all the Mesos slave nodes.

When the containers are allocated to the Spark job, the Driver then gets the executors started up and then runs the code in the executors. When the Spark job is completed and Driver exits, the Mesos master is notified, and all the resources in the form of containers on the Mesos slave nodes are reclaimed.

Multiple clients interacting with the cluster create their own executors on the slave nodes. Also, each client will have its own Driver component. Both client and cluster mode are possible just like YARN mode

The following is the mesos-based deployment of Spark depicting the Driver connecting to Mesos Master Node, which also has the cluster manager of all the resources on all the Mesos slaves:

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.138.67.203