Along with the job configurations mentioned in Spark standalone, here are a few useful configurations available in YARN mode:
- --num-executors: Spark on YARN cluster allows users to define the number of executors for the application. This option is not available yet in Spark standalone
- --queue: This option allows users to provide a queue name of YARN for submitting the application.
Apart from Spark standalone and YARN, Spark can also run on Mesos cluster manager as well.