The following points should give a good idea of the data storage process:

All data in HDFS is written in blocks, usually of size 128 MB. Thus, a single file of say size 512 MB would be split into four blocks (4 * 128 MB). These blocks are then written to DataNodes. To maintain redundancy and high availability, each block is replicated to create duplicate copies. In general, Hadoop installations have a replication factor of three, indicating that each block of data is replicated three times.

This guarantees redundancy such that in the event one of the servers fails or stops responding, there would always be a second and even a third copy available. To ensure that this process works seamlessly, the DataNode places the replicas in independent servers and can also ensure that the blocks are placed on servers in different racks in a data center. This is due to the fact that even if all the replicas were on independent servers, but all the servers were on the same rack, a rack power failure would mean that no replica would be available.

The general process of writing data into HDFS is as follows:

1. The NameNode receives a request to write a new file to HDFS.
2. Since the data has to be written in blocks or chunks, the HDFS client (the entity that made the request) begins caching data into a local buffer and once the buffer reaches the allocated chunk size (for example, 128 MB), it informs the NameNode that it is ready to write the first block (chunk) of data.
3. The NameNode, based on information available to it about the state of the HDFS cluster, responds with information on the destination DataNode where the block needs to be stored.
4. The HDFS client writes data to the target DataNode and informs the NameNode once the write process for the block has completed.
5. The target DataNode, subsequently, begins copying its copy of the block of data to a second DataNode, which will serve as a replica for the current block.

6. Once the second DataNode completes the write process, it sends the block of data to the third DataNode.
7. This process repeats until all the blocks corresponding to the data (or equivalently, the file) are copied across different nodes.

Note that the number of chunks will depend on the file size. The following image illustrated the distribution of the data across 5 datanodes.

The HDFS architecture in the first release of Hadoop, also known as Hadoop 1, had the following characteristics:

• Single NameNode: Only one NameNode was available, and as a result it also acted as a single point of failure since it stored all the cluster metadata.
• Multiple DataNodes that stored blocks of data, processed client requests, and performed I/O operations (create, read, delete, and so on) on the blocks.
• The HDFS architecture in the second release of Hadoop, also known as Hadoop 2, provided all the benefits of the original HDFS design and also added some new features, most notably, the ability to have multiple NameNodes that can act as primary and secondary NameNodes. Other features included the facility to have multiple namespaces as well as HDFS Federation.
• HDFS Federation deserves special mention. The following excerpt from http://hadoop.apache.org explains the subject in a very precise manner:

The secondary NameNode is not a backup node in the sense that it cannot perform the same tasks as the NameNode in the event that the NameNode is not available. However, it makes the NameNode restart process much more efficient by performing housekeeping operations.