Jobtracker high availability

In MRv1, if the jobtracker fails, all running jobs and tasks are lost. Also, the jobtracker service along with its jobs need to be manually restarted. To avoid these issues, the jobtracker needs to be configured for high availability. CDH5 comes inbuilt with the jobtracker HA package.

Configuring jobtracker high availability

Use the following steps from user hduser to configure and HA jobtracker for your cluster:

  1. Stop all the tasktrackers by executing the following command on all the nodes that host tasktrackers:
    $ sudo service hadoop-0.20-mapreduce-tasktracker stop
  2. Stop the jobtracker by executing the following command on the node that hosts the jobtracker:
    $ sudo service hadoop-0.20-mapreduce-jobtracker stop
  3. Remove the installed jobtracker using the following command from node1.hcluster:
    $ sudo yum remove hadoop-0.20-mapreduce-jobtracker
  4. Install the following HA jobtracker package on two independent nodes, which in our case would be node1.hcluster and node2.hcluster:
    $ sudo yum install hadoop-0.20-mapreduce-jobtrackerha
  5. If you intend to use automatic failover for the jobtracker, install the ZooKeeper failover controller using the following command on node1.hcluster and node2.hcluster:
    $ sudo yum install hadoop-0.20-mapreduce-zkfc
  6. Update the mapred-site.xml file to include the following properties and copy it to all the nodes in the cluster:
    <property>
  <name>mapred.job.tracker</name>
  <value>myjobtracker</value>
</property>
<property>
  <name>mapred.jobtrackers.myjobtracker</name>
  <value>jt1,jt2</value>
</property>
<property>
  <name>mapred.jobtracker.rpc-address.myjobtracker.jt1</name>
  <value>node1.hcluster:8021</value>
</property>
<property>
  <name>mapred.jobtracker.rpc-address.myjobtracker.jt2</name>
  <value>node2.hcluster:8022</value>
</property>
<property>
  <name>mapred.job.tracker.http.address.myjobtracker.jt1</name>
  <value>0.0.0.0:50030</value>
</property>
<property>
  <name>mapred.job.tracker.http.address.myjobtracker.jt2</name>
  <value>0.0.0.0:50031</value>
</property>
<property>
  <name>mapred.ha.jobtracker.rpc-address.myjobtracker.jt1</name>
  <value>node1.hcluster:8023</value>
</property>
<property>
  <name>mapred.ha.jobtracker.rpc-address.myjobtracker.jt2</name>
  <value>node2.hcluster:8024</value>
</property>
<property>
  <name>mapred.ha.jobtracker.http-redirect-address.myjobtracker.jt1</name>
  <value>node1.hcluster:50030</value>
</property>
<property>
  <name>mapred.ha.jobtracker.http-redirect-address.myjobtracker.jt2</name>
  <value>node2.hcluster:50031</value>
</property>
<property>
  <name>mapred.ha.fencing.methods</name>
  <value>shell(/bin/true)</value>
</property>
  7. Start the HA jobtracker daemons on all on the nodes for the jobtrackers. For our configuration, execute the following command on node1.hcluster and node2.hcluster:
    $ sudo service hadoop-0.20-mapreduce-jobtrackerha start
  8. If automatic failover is not set up, both jobtrackers will start in the standby state. If you try visiting the URL http://node1.hcluster:50030/, the page will be redirected with a message as shown in the following screenshot:
    Configuring jobtracker high availability
  9. To transition jobtracker jt1 to active state, use the following command:
    $ sudo -u mapred hadoop mrhaadmin -transitionToActive jt1

Your high available jobtracker is now configured. However, the failover has to be done manually in case the active jobtracker fails.

Configuring automatic failover for jobtracker high availability

To configure automatic failover, use the following steps:

  1. Add the following properties to mapred-site.xml and copy it to all the nodes in the cluster:
    <property>
  <name>mapred.ha.automatic-failover.enabled</name>
  <value>true</value>
</property>
<property>
  <name>mapred.ha.zkfc.port</name>
  <value>8018</value>
</property>
  2. Add the following property to core-site.xml file and copy it to all the nodes in the cluster:
    <property>
  <name>ha.zookeeper.quorum</name>
  <value>node3.hcluster:2181, node4.hcluster:2181, node5.hcluster:2181 </value>
</property>
  3. Initialize the HA state of the jobtracker in ZooKeeper using one of the following commands:
    $ sudo service hadoop-0.20-mapreduce-zkfc init
$ sudo -u mapred hadoop mrzkfc –formatZK
  4. Enable automatic failover by starting the ZooKeeper failover controller and the HA jobtracker on the jobtracker nodes using the following commands:
    $ sudo service hadoop-0.20-mapreduce-zkfc start
$ sudo service hadoop-0.20-mapreduce-jobtrackerha start

Your cluster is now configured for automatic failover of HA jobtracker. You can test this configuration by manually killing the active jobtracker using kill -9 to see whether the failover occurs.

With this knowledge, as an administrator, you can now set up a robust highly available Hadoop cluster running CDH5.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.191.202.240