The Mesos master usually replaces the Spark master as the cluster manager (aka the resource manager) when using Mesos. Now, when a driver program creates a Spark job and starts assigning the related tasks for scheduling, Mesos determines which computing nodes handle which tasks. We assume that you have already configured and installed Mesos on your machine.
Depending upon hardware configuration, it takes a while. On my machine (Ubuntu 14.04 64-bit, with Core i7 and 32 GB of RAM), it took 1 hour to complete the build.
To submit and compute your Spark jobs by utilizing the Mesos cluster mode, make sure to check that the Spark binary packages are available in a place accessible by Mesos. Additionally, make sure that your Spark driver program can be configured in such a way that it is automatically connected to Mesos. The second option is installing Spark in the same location as the Mesos slave nodes. Then, you will have to configure the spark.mesos.executor.home parameter to point the location of Spark distribution. It is to be noted that the default location that could point is the SPARK_HOME.
When Mesos executes a Spark job on a Mesos worker node (aka computing node) for the first time, the Spark binary packages have to be available on that worker node. This will ensure that the Spark Mesos executor is running in the backend.
1. Having the URIs/URLs (including HTTP) via http://,
2. Using the Amazon S3 via s3n://,
3. Using the HDFS via hdfs://.
If you set the HADOOP_CONF_DIR environment variable, the parameter is usually set as hdfs://...; otherwise file://.
You can specify the Master URLs for Mesos as follows:
- mesos://host:5050 for a single-master Mesos cluster, and mesos://zk://host1:2181,host2:2181,host3:2181/mesos for a multimaster Mesos cluster controlled by the ZooKeeper.