Start slave

After starting Spark master, Spark slave can be started using the Spark master URL, as follows:

cd $SPARK_HOME 
sbin/start-slave.sh spark://spark-master:7077
Initializing Spark Slave

The following is the Spark master web UI showing two active workers:

Spark Master UI with Two Slave Nodes

All slave/worker nodes can be started together. Hostnames of the nodes should be added in the $SPARK_HOME/conf/slaves file on the master node, as follows:

Spark slaves file

Then, execute the following command to start all slave nodes together:

cd $SPARK_HOME 
sbin/start-slaves.sh  
  • start-slaves.sh script helps to login to all slave nodes using SSH protocol. Hence, it is suggested to set up password less SSH between master and slave nodes. Otherwise, passwords need to be entered manually for every login.
  • If a username on Spark master is different than Spark slave node then entry is Hostname should be added as follows: username@slave-host.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.117.188.138