In the Quorum-based storage HA technique, the two namenodes use a Quorum Journal Manager (QJM) to share edit logs. As we already know, edits log is the activity log of all file operations on the cluster. Both the active namenode and the standby namenode need to have their edit logs file in sync. To achieve this, the active namenode communicates with the standby namenode using the JournalNode (JN) daemons. The active namenode reports every change of its edits logs file to the JournalNodes daemons. The standby namenode reads the edit logs from the JournalNodes daemons and applies all changes to its own namespace. By doing this, the two namenodes are always in sync.

In the following diagram, you see a typical architecture of HDFS HA using QJM. There are two namenodes, active and standby, that communicate with each other via JournalNodes.

The JournalNodes are daemons that run on JournalNode machines. It is advisable to use at least three JournalNode daemons to make sure the edit logs are written to three different locations allowing JournalNodes to tolerate the failure of a single machine. The system can tolerate the failure of (N-1)/2 JournalNodes, where N is the number of JournalNodes configured.

At any time, only the active namenode can write to the JournalNodes. This is handled by the JournalNodes to avoid the updates of the shared edit logs by two namenodes, which could cause data loss as well as incorrect results. The standby namenode will only have read access to the JournalNodes to update its own namespace.

When configuring HA, the secondary namenode is not used. As you will recall, the secondary namenode is a checkpoint service that performs periodic checkpoints of the edit logs in the primary namenode and updates the fsimage. The standby namenode in an HA environment performs the checkpoints and does not need a secondary namenode.

The datanodes of the cluster update both the active as well as the standby namenode with the location and heartbeat information, thus making it possible for a quick failover.

When the active namenode fails, the standby namenode assumes the responsibility of writing to the JournalNodes and takes over the active role.

<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/admin/.ssh/id_rsa</value>
</property>

<property>
  <name>dfs.ha.fencing.methods</name>
  <value>sshfence([[username][:port]])</value>
</property>
<property>
  <name>dfs.ha.fencing.ssh.connect-timeout</name>
  <value>30000</value>
</property>

<property>
  <name>dfs.ha.fencing.methods</name>
  <value>shell(/path/to/my/script.sh –namenode=$target_host–nameservice=$target_nameserviceid)</value>
</property>

<property>
  <name>dfs.ha.fencing.methods</name>
  <value> shell(/bin/true)</value>
</property>

In this approach, the active and standby namenodes need to have access to a common shared storage such as Network File System (NFS). The active namenode logs a copy of its own namespace modifications on the shared network storage. The standby namenode reads these modifications and applies them to its own namespace and keeps its namespace in sync with the active namenode. In the event of a failover, the standby namenode would have to wait till all the operations from the shared edits log have been applied to its namespace before transitioning into the active state. All datanodes are configured to send to block information and heartbeats to both namenodes. A fencing method needs to be deployed for this approach to make sure that the previously active namenode is shut down before the standby namenode becomes active.

The hardware for the namenode machines in this architecture should be equivalent and just like the hardware setup of a non-HA namenode. The shared storage should be accessible by both the namenodes and should be configured for redundancy to handle failures. Redundancy should be configured for the disk, network, and power. The access and hardware of the shared storage are very critical to this architecture and should be of a high quality with multiple network access paths. Redundancy prevents the NFS (shared storage) from becoming the single point of failure.

Configuring HDFS high availability by shared storage using NFS

Almost all the configuration parameters for the hdfs-site.xml configuration file are similar to the one we did for Quorum-based storage. However, we need to update the property as shown in the following code to set up HDFS HA using NFS:

<property>
  <name>dfs.namenode.shared.edits.dir</name>        
  <value>file:///mnt/shared_storage</value>
</property>

Here, the property dfs.namenode.shared.edits.dir points to a shared directory, which has been locally mounted.

Once you are done deciding and configuring the desired method for HDFS HA (Quorum-based or Shared Storage), you need to perform the following steps:

1. Stop all Hadoop daemons from hduser using the following command on every node:

   $ for x in 'cd /etc/init.d; ls hadoop*'; do sudo service $x stop ; done
   

2. Install the namenode package from hduser on the node you want to configure as standby. As per our configuration, it is node2.hcluster.

   $ sudo yum install hadoop-hdfs-namenode
   

3. If you are using the Quorum-based storage approach, install the JournalNode package from hduser on the nodes you want to use as JournalNodes. As per our configuration, we would need to install the JournalNode package on node3.hcluster, node4.hcluster, and node5.hcluster:

   $ sudo yum install hadoop-hdfs-journalnode
   

   This step can be skipped if you are using the Shared Storage approach.

4. Start the JournalNode daemon using the following command on all the nodes where they will run:

   sudo service hadoop-hdfs-journalnode start
   

   In our configuration, the nodes are node3.hcluster, node4.hcluster, and node5.hcluster.

5. Next, go to the primary namenode and execute the following command from hduser to initialize the shared edits directory from the local namenode edits directory:

   $ sudo -u hdfs hdfs namenode -initializeSharedEdits
   

6. Next, start the primary namenode from hduser using the following command:

   $ sudo service hadoop-hdfs-namenode start
   

7. Start the standby namenode from hduser using the following command:

   $ sudo -u hdfs hdfs namenode -bootstrapStandby
   $ sudo service hadoop-hdfs-namenode start
   

8. Restart all Hadoop daemons from hduser on all the nodes using the following command:

   $ for x in 'cd /etc/init.d; ls hadoop*'; do sudo service $x start ; done
   

The preceding steps should start the namenodes on node1.hcluster as well as node2.hcluster along with the other Hadoop daemons. When a namenode starts, it is initially in the standby mode.

Use the hdfs haadmin command to perform the various administrative operations for HDFS HA. To see all the options available with this command, use the hdfs haadmin –help command as shown in the following code:

$ hdfs haadmin -help
Usage: DFSHAAdmin [-ns <nameserviceId>]
    [-transitionToActive <serviceId>]
    [-transitionToStandby <serviceId>]
    [-failover [--forcefence] [--forceactive] <serviceId> <serviceId>]
    [-getServiceState <serviceId>]
    [-checkHealth <serviceId>]
    [-help <command>]

The following is the description of each option available for the hdfs haadmin command:

• transitionToActive: This option is used when you want to change the state of a standby namenode to active.
• transitionToStandby: This option is used when you want to change the state of an active namenode to standby.

  Usage of the preceding two options is not usually done on production systems as they do not support fencing.

• failover: This option is used to perform a failover of the namenodes. Using this flag, the administrator can set the standby namenode to the active state. For our configuration, we can use the following command to set the namenode on node1.hcluster to active:

  $ sudo -u hdfs hdfs haadmin -failover --forceactive nn2 nn1
  

  After the namenode enters the standby state, it starts checkpointing the fsimage of the active namenode. The fencing configuration takes effect in case of any failures during failover.

• getServiceState: This option is used to print the current status of the namenode. To print the status of the namenodes we have configured, you can use the following commands:

  $ sudo -u hdfs hdfs haadmin -getServiceState nn1
  $ sudo -u hdfs hdfs haadmin -getServiceState nn2
  

• checkHealth: This option is used to check the health of the specified namenode. The return value of 0 indicates that the namenode is healthy. A nonzero value is returned if the namenode is unhealthy. As per the current implementation feature, the option will indicate an unhealthy status only if the namenode is down.

The following screenshot shows the summary section of the active namenode:

The following screenshot shows the summary section of the standby namenode:

NameNode Journal Status for Quorum-based storage approach

In the Quorum-based storage approach, only the active namenode will be allowed to perform write operations to the JournalNodes. As shown in the following screenshot, this information is displayed under the NameNode Journal Status section:

The following screenshot shows the NameNode Journal Status of the standby namenode. In the Quorum-based storage approach, the standby namenode is only allowed to perform read operations on the JournalNodes. Also, this information is displayed under the NameNode Journal Status section:

Using the preceding steps, an administrator can perform a manual transition from one namenode to the other.

NameNode Journal Status for the Shared Storage-based approach

The following screenshot shows the NameNode Journal Status for the standby namenode in the Shared storage-based approach. As you can see in the following screenshot, the standby namenode in the Shared Storage-based approach is only allowed to read from the shared storage:

As you can see in the following screenshot, the active namenode in a Shared Storage configuration is allowed to write to the common shared location:

To perform an automatic failover where no manual intervention is required, we need to use Apache Zookeeper. As you saw in Chapter 3, Cloudera's Distribution Including Apache Hadoop, Apache Zookeeper is a distributed coordination service.

To configure automatic failover, the following two additional components are installed on an HDFS deployment:

The ZooKeeper service is responsible for the following two operations:

The ZKFC component is a ZooKeeper client that helps in monitoring and managing the state of the namenode. A ZKFC service runs on each machine that runs a namenode.

The ZKFC service is responsible for the following operations:

The ZooKeeper daemons typically run on three or five nodes (number of nodes should be an odd number) and can be collocated with the active and standby namenodes.

Perform the following steps as user hduser to configure ZooKeeper for automatic failover:

Your cluster is now configured for automatic failover. You can test this configuration by manually killing the active the namenode using kill -9 to see if the failover occurs.