The practice of removing nodes from a Hadoop cluster is very common. Hardware might have failed, or machines might need to be upgraded. In this recipe, we will show you the steps to safely remove a worker node from a Hadoop cluster.
It is assumed that your cluster is up and running and you have configured the following properties in mapred-site.xml:
<property>
<name>dfs.hosts.exclude</name>
<value>/path/to/hadoop/dfs_excludes</value>
<final>true</final>
</property>
<property>
<name>mapred.hosts.exclude</name>
<value>/path/to/hadoop/mapred_excludes </value>
<final>true</final>
</property>In addition, there should be two files located in the Hadoop home folder on the head node: dfs_excludes and mapred_excludes.
Perform the following steps to decommission a node in the Hadoop cluster:
- Add the hostname of the node you wish to decommission to the
dfs_excludesandmapred_excludesfiles on the head node:$ vi /path/to/hadoop/dfs_excludes worker1 $ vi /path/to/hadoop/mapred_excludes worker1
- Notify the NameNode to re-read the exclude list and disconnect the worker node which will be decommissioned:
$ hadoop dfsadmin –refreshNodes
- Notify the JobTracker to re-read the exclude list and disconnect the worker node which will be decommissioned:
$ hadoop mradmin –refreshNodes
- Check the status of the decommissioning process:
$ hadoop dfsadmin -report
First, we added the hostname of the node we wanted to decommission to the dfs_excludes and
mapred_excludes files we created in a previous recipe. Next, we issued the hadoop dfsadmin –refreshNodes command to notify the NameNode to disconnect from all of the hosts listed in the dfs_excludes file. Similarly, we issued the
hadoop mradmin –refreshNodes command to notify the JobTracker to stop using the TaskTrackers on the nodes listed in the mapred_excludes file.