In a YARN cluster, a client connects to RM, submits the request, and exits. RM launches the AM of the Spark application on one of the NM and the AM negotiates resources with the AM and then launches the driver thread. NMs launch containers that connect to AM. After containers initialize, AM launches executors in containers, which connect to the Driver in the AM to run the application.
Logical Representation of Spark Application in YARN Cluster Mode
In the following command, we will deploy a Spark PI application on a YARN cluster:
cd $SPARK_HOME
./bin/spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-2.1.1.jar
To deploy Spark applications on YARN from a remote system, copy Hadoop configurations files on a remote system and set the environment variable HADOOP_CONF_DIR, which should point to the directory that contains Hadoop configure files.