In YARN client mode, Spark Driver runs in client nodes. The client submits requests to RM, RM launches applications for the application on the cluster, which negotiates resources with RM. Then the AM connects to the NM as per the info given by RM, which launches the container for the application. Executor processes run inside containers and run the applications. Executor processes connect to the driver application on the client node.
Logical Representation of Spark Applications in YARN Client Mode
In YARN, the AM and containers are scheduled by RM, however, tasks that run the application are scheduled by Spark Driver itself.
To run spark shell in YARN client mode, execute the following commands:
cd $SPARK_HOME
./bin/spark-shell -master yarn -deploy-mode client
