In a production environment, we should enable rack awareness for a more robust cluster as it might happen that the whole rack will go down due to a problem with the central switch for this specific rack, or there might be a fire that will affect all the machines on the rack. If we enable and set up a cluster with rack awareness, we can avoid the whole cluster going down and always access the data that lies on nodes on other racks; meanwhile, the affected rack can be brought up. Now, let's see how to enable and configure this feature.

To make the cluster rack aware, we can write a script that enables the master node to assign data blocks according to the topology for high availability in case of a rack failure. For this, we need to include a parameter in the hadoop-site.xml file, which is topology.script.file.name (net.topology.script.file.name in Hadoop 2); for this, we give the path of the executable script that returns a list of IPs or hostnames and the rack where they reside:

<property>
  <name>topology.script.file.name</name>
  <value>/home/shashwat/hadoop/conf/hadoopRackAwareness.sh</value>
</property>

The following is the hadoopRackAwareness.sh script:

#!/bin/sh
#SourceFrom : http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
#Credit : Apache Wiki

HADOOP_CONF=/home/shashwat/hadoop
while [ $# -gt 0 ] ; do
nodeArg=$1
exec< ${HADOOP_CONF}/IPRackFile.list
  result="" 
while read line ; do
ar=( $line ) 
if [ "${ar[0]}" = "$nodeArg" ] ; then
result="${ar[1]}"
fi
done
shift
if [ -z "$result" ] ; then
echo -n "/default/rack "
else
echo -n "$result "
fi
done

The following is the content of IPRackFile.list:

192.168.1.10    /rack1
192.168.1.11    /rack1
192.168.1.12    /rack1
10.3.0.3        /rack2
10.3.0.4        /rack2
10.3.0.5        /rack2
192.3.0.4       /rack3
192.3.0.5       /rack3
192.3.0.6       /rack3

After making these changes, restart the cluster and issue this command that will show a new line with the rack information:

bin/hadoop dfsadmin –report

For the number of Map and Reduce limits, we need to consider the following parameters:

To set these two values, we need to consider available CPU cores, disk, and memory. Suppose we have an eight core processor, and if the job is CPU intensive, we can set four Maps that will finish. Then, the reducer will start; for less CPU-intensive jobs, we can keep map task at 40 and reduce task at 20. After fixing these, we need to see if there is a long wait, and then we can reduce the number so as to make the process faster. We must understand that if we set too big a number, there will be a lot of context switching and swapping of data between memory and disk, which might reduce the overall slow processing, so we need to make it balanced according to the system resources we have.

mapred.tasktracker.map.tasks.maximum = 2 + cpu_numer * 2/3
mapred.tasktracker.reduce.tasks.maximum = 2 + cpu_numer * 1/3