Useful job configuration

Along with the job configurations mentioned in Spark standalone, here are a few useful configurations available in YARN mode:

  • --num-executors: Spark on YARN cluster allows users to define the number of executors for the application. This option is not available yet in Spark standalone
  • --queue: This option allows users to provide a queue name of YARN for submitting the application.

Apart from Spark standalone and YARN, Spark can also run on Mesos cluster manager as well.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.143.239.44