StatefulSet with persistent volume; it requires a StorageClass that provisions a volume dynamically. Because each Pod is created by StatefulSets, it will create a persistent volume claim (PVC) with a different identifier. If your StatefulSets specify a static name of PVC, there will be trouble if multiple Pods try to attach the same PVC.

If you have StorageClass on your cluster, update datanode.yaml to add spec.volumeClaimTemplates as follows:

$ curl https://raw.githubusercontent.com/kubernetes-cookbook/second-edition/master/chapter3/3-4/datanode-pv.yaml
...
  volumeClaimTemplates:
  - metadata:
      name: hdfs-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

This tells Kubernetes to create a PVC and PV when a new Pod is created by StatefulSet. So, that Pod template (spec.template.spec.containers.volumeMounts) should specify hdfs-data, as follows:

$ curl https://raw.githubusercontent.com/kubernetes-cookbook/second-edition/master/chapter3/3-4/datanode-pv.yaml
...
          volumeMounts:
          - mountPath: /hadoop/dfs/data
            name: hdfs-data

Let's recreate HDFS cluster again:

//delete DataNodes
$ kubectl delete -f https://raw.githubusercontent.com/kubernetes-cookbook/second-edition/master/chapter3/3-4/datanode.yaml
service "hdfs-datanode-svc" deleted
statefulset "hdfs-datanode" deleted

//delete NameNode
$ kubectl delete -f https://raw.githubusercontent.com/kubernetes-cookbook/second-edition/master/chapter3/3-4/namenode.yaml 
service "hdfs-namenode-svc" deleted
statefulset "hdfs-namenode" deleted


//create NameNode again
$ kubectl create -f https://raw.githubusercontent.com/kubernetes-cookbook/second-edition/master/chapter3/3-4/namenode.yaml
service "hdfs-namenode-svc" created
statefulset "hdfs-namenode" created

//create DataNode which uses Persistent Volume (datanode-pv.yaml)
$ kubectl create -f https://raw.githubusercontent.com/kubernetes-cookbook/second-edition/master/chapter3/3-4/datanode-pv.yaml
service "hdfs-datanode-svc" created
statefulset "hdfs-datanode" created

//3 PVC has been created automatically
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
hdfs-data-hdfs-datanode-0 Bound pvc-bc79975d-f5bd-11e7-ac7a-42010a8a00ef 10Gi RWO standard 53s
hdfs-data-hdfs-datanode-1 Bound pvc-c753a336-f5bd-11e7-ac7a-42010a8a00ef 10Gi RWO standard 35s
hdfs-data-hdfs-datanode-2 Bound pvc-d1e10587-f5bd-11e7-ac7a-42010a8a00ef 10Gi RWO standard 17s

To demonstrate, use kubectl exec to access the NameNode, then copy some dummy files to HDFS:

$ kubectl exec -it hdfs-namenode-0 -- /bin/bash
root@hdfs-namenode-0:/# hadoop fs -put /lib/x86_64-linux-gnu/* /
root@hdfs-namenode-0:/# exit
command terminated with exit code 255

//delete DataNode-1
$ kubectl delete pod hdfs-datanode-1
pod "hdfs-datanode-1" deleted

At this moment, DataNode-1 is restarting, as shown in the following image. However, the data directory of DataNode-1 is kept by PVC as hdfs-data-hdfs-datanode-1. The new Pod hdfs-datanode-1 will take over this PVC again:

Therefore, when you access HDFS after hdfs-datanode-1 has recovered, you don't see any data loss or re-sync processes:

$ kubectl exec -it hdfs-namenode-0 -- /bin/bash
root@hdfs-namenode-0:/# hdfs fsck /
Connecting to namenode via http://hdfs-namenode-0.hdfs-namenode-svc.default.svc.cluster.local:50070/fsck?ugi=root&path=%2F
FSCK started by root (auth:SIMPLE) from /10.52.1.13 for path / at Wed Jan 10 04:32:30 UTC 2018
....................................................................................................
.................................................................Status: HEALTHY
 Total size: 22045160 B
 Total dirs: 2
 Total files: 165
 Total symlinks: 0
 Total blocks (validated): 165 (avg. block size 133607 B)
 Minimally replicated blocks: 165 (100.0 %)
 Over-replicated blocks: 0 (0.0 %)
 Under-replicated blocks: 0 (0.0 %)
 Mis-replicated blocks: 0 (0.0 %)
 Default replication factor: 3
 Average block replication: 3.0
 Corrupt blocks: 0
 Missing replicas: 0 (0.0 %)
 Number of data-nodes: 3
 Number of racks: 1
FSCK ended at Wed Jan 10 04:32:30 UTC 2018 in 85 milliseconds


The filesystem under path '/' is HEALTHY

As you see, the Pod and PV pair is fully managed by StatefulSets. It is convenient if you want to scale more HDFS DataNode using just the kubectl scale command to make it double or hundreds—whatever you need:

//make double size of HDFS DataNodes
$ kubectl scale statefulset hdfs-datanode --replicas=6
statefulset "hdfs-datanode" scaled

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hdfs-datanode-0 1/1 Running 0 20m
hdfs-datanode-1 1/1 Running 0 13m
hdfs-datanode-2 1/1 Running 0 20m
hdfs-datanode-3 1/1 Running 0 56s
hdfs-datanode-4 1/1 Running 0 38s
hdfs-datanode-5 1/1 Running 0 21s
hdfs-namenode-0 1/1 Running 0 21m

$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
hdfs-data-hdfs-datanode-0 Bound pvc-bc79975d-f5bd-11e7-ac7a-42010a8a00ef 10Gi RWO standard 21m
hdfs-data-hdfs-datanode-1 Bound pvc-c753a336-f5bd-11e7-ac7a-42010a8a00ef 10Gi RWO standard 21m
hdfs-data-hdfs-datanode-2 Bound pvc-d1e10587-f5bd-11e7-ac7a-42010a8a00ef 10Gi RWO standard 21m
hdfs-data-hdfs-datanode-3 Bound pvc-888b6e0d-f5c0-11e7-ac7a-42010a8a00ef 10Gi RWO standard 1m
hdfs-data-hdfs-datanode-4 Bound pvc-932e6148-f5c0-11e7-ac7a-42010a8a00ef 10Gi RWO standard 1m
hdfs-data-hdfs-datanode-5 Bound pvc-9dd71bf5-f5c0-11e7-ac7a-42010a8a00ef 10Gi RWO standard 1m