Let's start with an example. Assume you have 1 TB of data to read from a single machine with a single hard disk. Assuming the disk read rate is 100 MBps, it will take about 2 hours and 45 minutes to read the file. If you could split this data over 10 hard disks and read them all in parallel, it would have decreased the read time by 10—more or less. From a layman's perspective, this is what HDFS does; it breaks the data into fixed sized blocks (default is 64 MB) and distributes them over a number of slave machines.

HDFS is a filesystem that runs on top of a regular filesystem. Production installations generally have ext3 filesystems running beneath HDFS. By distributing data across several nodes, the storage layer can scale to a very large virtual storage that scales linearly. To provide reliability to store data, the data is stored with redundancy. Each block is replicated three times by default. HDFS is architected in such a way that each data block gets replicated to different servers and if possible on different racks. This saves data from disk, server, or complete rack failure. In the event of a disk or a server failure, data is replicated to a new location to meet the replication factor. If this reminds you of Cassandra, or any other distributed system, you are on the right track. But as we will see very soon, unlike Cassandra, HDFS has a single point of failure due to its master-slave design.

Despite all these good features, HDFS has a few shortcomings too.

HDFS uses the master-slave mechanism to distribute data across multiple servers. The master node is usually backed by a powerful machine so that it does not fail. The slave machines are data nodes; these are commodity hardware. The reason of master nodes being beefy machines is that it is a single point failure. If the master node (that is, the NameNode) goes down, the storage is down—unlike the Cassandra model. To load the data to HDFS, the client connects to the master node and sends an upload request. The master node tells the client to send parts of data to various data nodes. Note that data does not stream through the master node. It just directs the client to appropriate data nodes and maintains the metadata about the location of various parts of a file.

Figure 8.1: Client makes a request to NameNode to write a block. NameNode returns the nodes where the block is to be written. Client picks one DataNode from the nodes list in the previous step and forwards to other nodes

There are two processes one needs to know about to understand how the data is distributed and managed by HDFS.

MapReduce or MR is a very simple concept once you know it. It is algorithm 101: divide and conquer. The job is broken into small independent tasks and distributed across multiple machines; the result gets sorted and merged together to generate the final result. The ability to distribute a large computational burden over multiple servers into a small computational load over multiple servers enables a Hadoop programmer to have effectively limitless CPU capability for data analysis. MR is the processing part of Hadoop; it virtualizes the CPU. Figure 8.2 depicts this process.

As an end user, you need to write a Mapper and a Reducer for the tasks you need to get done. The Hadoop framework performs the heavy lifting of getting data from a source and splitting it into maps of keys and values based on what the data source is. It may be a line from a text file, a row from a relational database, or a key-value from the Cassandra column family. These maps of key-value pairs (indicated as Input key-val pairs in the next figure) are forwarded to the Mapper that you have provided to Hadoop. Mapper performs unit tasks of the key-value pair; for example, for a word count task, you may want to remove punctuations, split the words by whitespace, iterate in this split array of words, forward key as an individual word, and set value as one. These make the intermediate key-value pair, as indicated in the next figure.

These results are sorted by the key and forwarded to the Reducer interface that you provided. Reducers can use this property, that coming tuples have the same key. Understanding the last sentence is important to a beginner. What it means is that you can just iterate in the incoming iterator and do things such as group or count—basically reduce or fold the map by key.

Figure 8.2: Hadoop MapReduce framework in action (simplified)

The reduced values are then stored in a storage of your choice that can be HDFS, RDBMS, Cassandra, or one of the many other storage options.

There are two main processes that you should know about in the context of Hadoop MapReduce.

Hadoop is a very robust and reliable architecture. It is meant to be run on commodity hardware and hence takes care of failure automatically. It detects failure of a task and retries the failed tasks. It is fault tolerant. A down DataNode is replicated (redundant) and a system heals by itself, in case of unavailability of a DataNode.

Hadoop allows servers to join the cluster or leave it without any repercussion. Rack-aware storage of data saves the cluster against disk failures, rack/machine power failure, and even a complete rack going down.

Figure 8.3 shows the famous schema of reliable Hadoop infrastructure using commodity hardware for slaves and heavy-duty servers (top of the rack) for the masters. Please note that these are physical servers as they are in the data centers. Later, when discussing using Cassandra as a data store, we will use a ring representation. Even in that case, the physical configuration may be the same as the one represented in Figure 8.3, but the logical configuration, as we have been seeing throughout this book, will be a ring-like structure to emphasize the token distribution.