Chapter 13: DaemonSet – Maintaining Pod Singletons on Nodes

The previous chapters have explained and demonstrated how to use the most common Kubernetes controllers for managing Pods, such as ReplicaSet, Deployment, and StatefulSet. Generally, when running cloud application components that contain the actual business logic you will need either Deployments or StatefulSets for controlling your Pods. In some cases, when you need to run batch workloads as part of your application, you will use Jobs and CronJobs.

However, in some cases, you will need to run components that have a supporting function and, for example, execute maintenance tasks or aggregate logs and metrics. More specifically, if you have any tasks that need to be executed for each Node in the cluster, they can be performed using a DaemonSet. This is the last type of Pod management controller that we are going to introduce in this part of the book. The purpose of a DaemonSet is to ensure that each Node (unless specified otherwise) runs a single replica of a Pod. If you add a new Node to the cluster, it will automatically get a Pod replica scheduled. Similarly, if you remove a Node from the cluster, the Pod replica will be terminated – the DaemonSet will execute all required actions.

In this chapter, we will cover the following topics:

  • Introducing the DaemonSet object
  • Creating and managing DaemonSets
  • Common use cases for DaemonSets
  • Alternatives to DaemonSets

Technical requirements

For this chapter, you will need the following:

  • A Kubernetes cluster deployed. You can use either a local or cloud-based cluster, but in order to fully understand the concepts we recommend using a multi-node, cloud-based Kubernetes cluster.
  • The Kubernetes CLI (kubectl) installed on your local machine and configured to manage your Kubernetes cluster.

Kubernetes cluster deployment (local and cloud-based) and kubectl installation were covered in Chapter 3, Installing Your First Kubernetes Cluster.

You can download the latest code samples for this chapter from the official GitHub repository: https://github.com/PacktPublishing/The-Kubernetes-Bible/tree/master/Chapter13.

Introducing the DaemonSet object

The term daemon in operating systems has a long history and, in short, is used to describe a program that runs as a background process, without interactive control from the user. In many cases, daemons are responsible for handling maintenance tasks, serving network requests, or monitoring hardware activities. These are often processes that you want to run reliably, all the time, in the background, from the time you boot the operating system to when you shut it down.

Tip

Daemons are associated in most cases with Unix-like operating systems. In Windows, you will more commonly encounter the term Windows service.

In Kubernetes, you may need a similar functionality where your Pods behave like classic operating system daemons on each of the Nodes in the cluster. For this, Kubernetes offers a dedicated Pod management controller named DaemonSet. The role of a DaemonSet is straightforward: run a single Pod replica on each of the Nodes in the cluster and manage them automatically. There are variety of use cases that require such Node-local facilities and usually they serve important and fundamental roles for the whole cluster – we will discuss some common use cases in the sections coming up, but generally you need them for the following:

  • Node monitoring in the cluster.
  • Logs and telemetry gathering from individual Nodes and sometimes Pods running on a Node.
  • Managing cluster storage – this is especially important for handling requests from provisioners for PersistentVolumeClaims and PersistentVolumes.

All you have learned in the previous chapters about ReplicaSets, Deployments, and StatefulSets applies more or less to the DaemonSet. Its specification requires you to provide a Pod template, Pod label selectors, and optionally Node selectors if you want to schedule the Pods only on a subset of Nodes.

Depending on the case, you may not need to communicate with the DaemonSet from other Pods or from an external network. For example, if the job of your DaemonSet is just to perform a periodic cleanup of the filesystem on the Node, it is unlikely you would like to communicate with such Pods. If your use case requires any ingress or egress communication with the DaemonSet Pods, then you have the following common patterns:

  • Map container ports to host ports: Since the DaemonSet Pods are guaranteed to be singletons on cluster Nodes, it is possible to use mapped host ports. The clients must know the Node IP addresses.
  • Pushing data to a different service: In some cases, it may be enough that the DaemonSet is responsible for sending updates to other services without needing to allow ingress traffic.
  • Headless service matching DaemonSet Pod label selectors: This is a similar pattern to the case of StatefulSets, where you can use the cluster DNS to retrieve multiple A records for Pods using the DNS name of the headless service.
  • Normal service matching DaemonSet Pod label selectors: Less commonly, you may need to reach any Pod in the DaemonSet. Using a normal Service object, for example of the ClusterIP type, will allow you to communicate with a random Pod in the DaemonSet.

We will now show how you can create and manage an example DaemonSet in your cluster.

Creating and managing DaemonSets

In order to demonstrate how DaemonSets work, we will use nginx running in a Pod container that returns simple information about the Node IP address where it is currently scheduled. The IP address will be provided to the container using an environment variable and based on that, a modified version of index.html in /usr/share/nginx/html will be created. To access the DaemonSet endpoints, we will use a headless service, similar to what we did for StatefulSet in Chapter 12, StatefulSet – Deploy Stateful Applications. Most of the real use cases of DaemonSets are rather complex and involve mounting various system resources to the Pods. We will keep our DaemonSet example as simple as possible to show the principles.

Important note

If you would like to work on a real example of a DaemonSet, we have provided a working version of Prometheus node-exporter deployed as a DaemonSet behind a headless Service: https://github.com/PacktPublishing/Kubernetes-for-Beginners/blob/master/Chapter13/02_daemonset-prometheus-nodeexporter/node-exporter.yaml. When following the guide in this section, the only difference is that you need to use node-exporter as the Service name, use port 9100 and append the /metrics path for requests sent using wget. This DaemonSet exposes Node metrics in Prometheus data model format on port 9100 under the /metrics path.

We will now go through all the YAML manifests required to create our DaemonSet and apply them to the cluster.

Creating a DaemonSet

First, let's take a look at the StatefulSet YAML manifest file named nginx-daemonset.yaml (full version available in the official GitHub repository for the book: https://github.com/PacktPublishing/Kubernetes-for-Beginners/blob/master/Chapter13/01_daemonset-example/nginx-daemonset.yaml):

apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: nginx-daemonset-example

spec:

  selector:

    matchLabels:

      app: nginx-daemon

      environment: test

# (to be continued in the next paragraph)

The first part of the preceding file contains the metadata and Pod label selector for the DaemonSet, quite similar to what you have seen in Deployments and StatefulSets. In the second part of the file, we present the Pod template that will be used by the DaemonSet:

# (continued)

  template:

    metadata:

      labels:

        app: nginx-daemon

        environment: test

    spec:

      containers:

      - name: nginx

        image: nginx:1.17

        ports:

        - containerPort: 80

        env:

        - name: NODE_IP

          valueFrom:

            fieldRef:

              fieldPath: status.hostIP

        command:

        - /bin/sh

        - -c

        - |

          echo "You have been served by Pod running on Node with IP address: $(NODE_IP)" > /usr/share/nginx/html/index.html

          nginx -g "daemon off;"

As you can see, the structure of DaemonSet spec is similar to what you know from Deployments and StatefulSets. The general idea is the same, you need to configure the Pod template and use a proper label selector to match the Pod labels. Note that you do not see the replicas field here, as the number of Pods running in the cluster will be dependent on the number of Nodes in the cluster. The specification has two main components:

  • selector: A label selector, which defines how to identify Pods that the DaemonSet owns. This can include set-based and equality-based selectors.
  • template: This defines the template for Pod creation. Labels used in metadata must match the selector.

It is also common to specify .spec.template.spec.nodeSelector or .spec.template.spec.tolerations in order to control the Nodes where the DaemonSet Pods are deployed. We cover Pod scheduling in detail in Chapter 19, Advanced Techniques for Scheduling Pods. Additionally, you can specify .spec.updateStrategy, .spec.revisionHistoryLimit, and .spec.minReadySeconds, which are similar to what you have learned about Deployment objects.

Tip

If you run hybrid Linux-Windows Kubernetes clusters, one of the common use cases for Node selectors or Node affinity for DaemonSets is ensuring that the Pods are scheduled only on Linux Nodes or only on Windows Nodes. This makes sense as the container runtime and operating system are very different between such Nodes.

Apart from that, in our Pod template we have used a similar override of command for the nginx container as we did in the cases of Deployments and StatefulSets in the previous chapters. The command creates index.html in /usr/share/nginx/html/ with information about the IP address of the Node that runs the Pod serving the request. After that, it starts the nginx web server with the standard entrypoint command for the image. To provide the information about the Node IP address, we use an additional NODE_IP environment variable populated by status.hostIP of the Pod object (at runtime).

Next, let's take a quick look at the headless Service named nginx-daemon-headless. Create an nginx-daemon-headless-service.yaml file with the following YAML manifest:

apiVersion: v1

kind: Service

metadata:

  name: nginx-daemon-headless

spec:

  selector:

    app: nginx-daemon

    environment: test

  clusterIP: None

  ports:

  - port: 80

    protocol: TCP

    targetPort: 80

As we explained in the case of the StatefulSet example, the specification is very similar to a normal Service, the only difference is that it has the None value for the clusterIP field. This will result in the creation of an nginx-daemon-headless headless Service. A headless Service allows us to return all Pods' IP addresses behind the Service as DNS A records instead of a single DNS A record with a Service clusterIP.

We have all the required YAML manifest files for our demonstration and we can proceed with applying the manifests to the cluster. Please follow these steps:

  1. Create the nginx-daemon-headless headless Service using the following command:

    $ kubectl apply -f ./nginx-daemon-headless-service.yaml

  2. Create an nginx-daemonset-example DaemonSet using the following command:

    $ kubectl apply -f ./nginx-daemonset.yaml

  3. Now, you can use the kubectl describe command to observe the creation of the DaemonSet:

    $ kubectl describe daemonset nginx-daemonset-example

  4. Alternatively, you can use ds as an abbreviation for daemonset when using the kubectl commands.
  5. If you use the kubectl get pods command, you can see that there will be one Pod scheduled for each of the Nodes in the cluster:

    $ $ kubectl get pods -o wide

    NAME                          ... IP             NODE                              ...

    nginx-daemonset-example-5w8jx ... 10.244.1.144   aks-nodepool1-77120516-vmss000000 ...

    nginx-daemonset-example-tzqmc ... 10.244.0.90    aks-nodepool1-77120516-vmss000001 ...

In our case, we have two Nodes in the cluster, so exactly two Pods have been created.

We have successfully deployed the DaemonSet and we can now verify that it works as expected. To do that, follow the given steps:

  1. First, we need to know the IP addresses of the individual Nodes in order to cross-check the output of further commands:

    $ kubectl get node -o wide

    NAME                                ...  INTERNAL-IP  ...

    aks-nodepool1-77120516-vmss000000   ...  10.240.0.4   ...

    aks-nodepool1-77120516-vmss000001   ...  10.240.0.5   ...

  2. Create an interactive busybox Pod and start the Bourne shell process. The following command will create the Pod and immediately attach your terminal so that you can interact from within the Pod:

    $ kubectl run -i --tty busybox --image=busybox:1.28 --rm --restart=Never -- sh

  3. We need to check how our nginx-daemon-headless headless Service is resolved by the cluster DNS:

    $ nslookup nginx-daemon-headless

    Server:    10.0.0.10

    Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

    Name:      nginx-daemon-headless

    Address 1: 10.244.1.144 10-244-1-144.nginx-daemon-headless.default.svc.cluster.local

    Address 2: 10.244.0.90 10-244-0-90.nginx-daemon-headless.default.svc.cluster.local

    We have been provided with two DNS names for the individual Pods in the DaemonSet: 10-244-1-144.nginx-daemon-headless.default.svc.cluster.local, which belongs to the aks-nodepool1-77120516-vmss000000 Node, and 10-244-0-90.nginx-daemon-headless.default.svc.cluster.local, which belongs to the aks-nodepool1-77120516-vmss000001 Node.

  4. We can now use the DNS names (also in short-name form) to communicate with the Pods of the DaemonSet. First, we will try to connect to the 10-244-1-144.nginx-daemon-headless Pod:

    $ wget http://10-244-1-144.nginx-daemon-headless && cat index.html

    Connecting to 10-244-1-144.nginx-daemon-headless (10.244.1.144:80)

    ...

    You have been served by Pod running on Node with IP address: 10.240.0.4

    As expected, the Pod is scheduled on the aks-nodepool1-77120516-vmss000000 Node, which has an IP address of 10.240.0.4 (you can cross-check this with earlier commands' output).

  5. Let's do a similar check for the other Pod in the cluster, 10-244-0-90.nginx-daemon-headless:

    $ rm index.html && wget http://10-244-0-90.nginx-daemon-headless && cat index.html

    Connecting to 10-244-0-90.nginx-daemon-headless (10.244.0.90:80)

    ...

    You have been served by Pod running on Node with IP address: 10.240.0.5

    And again, as expected, we have been served by a Pod running on the aks-nodepool1-77120516-vmss000001 Node.

This demonstrates the most important principles underlying how DaemonSet Pods are scheduled and how you can interact with them using headless Services. We will now show how you can modify the DaemonSet to roll out a new version of a container image for the Pods.

Modifying a DaemonSet

Updating a DaemonSet can be done in a similar way as for Deployments. If you modify the Pod template of the DaemonSet, this will trigger a rollout of a new revision of DaemonSet according to its updateStrategy. There are two strategies available:

  • RollingUpdate: The default strategy, which allows you to roll out a new version of your daemon in a controlled way. It is similar to rolling updates in Deployments in that you can define .spec.updateStrategy.rollingUpdate.maxUnavailable to control how many Pods in the clusters are unavailable at most during the rollout (defaults to 1) and .spec.minReadySeconds (defaults to 0). It is guaranteed that at most one Pod of DaemonSet will be in running state on each node in the cluster during the rollout process.
  • OnDelete: This strategy implements the legacy behavior of StatefulSet updates prior to Kubernetes 1.6. In this type of strategy, the DaemonSet will not automatically update the Pod by recreating them. You need to manually delete a Pod on a Node in order to get the new Pod template applied. This is useful in scenarios when you need to do additional manual actions or verifications before proceeding to the next Node.

The rollout of a new DaemonSet revision can be controlled in similar ways as for a Deployment object. You can use the kubectl rollout status command and perform imperative rollbacks using the kubectl rollout undo command. Let's demonstrate how you can declaratively update the container image in a DaemonSet Pod to a newer version:

  1. Modify the nginx-daemonset.yaml YAML manifest file so that it uses nginx:1.18 container image in the template:

    apiVersion: apps/v1

    kind: DaemonSet

    metadata:

      name: nginx-daemonset-example

    spec:

    ...

      template:

    ...

        spec:

          containers:

          - name: nginx

            image: nginx:1.18

  2. Apply the manifest file to the cluster:

    $ kubectl apply -f ./nginx-daemonset.yaml --record

  3. Immediately after that, use the kubectl rollout status command to see the progress in real time:

    $ kubectl rollout status ds nginx-daemonset-example

    Waiting for daemon set "nginx-daemonset-example" rollout to finish: 0 out of 2 new pods have been updated...

    Waiting for daemon set "nginx-daemonset-example" rollout to finish: 1 out of 2 new pods have been updated...

    daemon set "nginx-daemonset-example" successfully rolled out

  4. Similarly, using the kubectl describe command, you can see events for the DaemonSet that exactly show what the order was of the Pod recreation:

    $ kubectl describe ds nginx-daemonset-example

    ...

    Events:

      Type    Reason            Age   From                  Message

      ----    ------            ----  ----                  -------

      Normal  SuccessfulDelete  113s  daemonset-controller  Deleted pod: nginx-daemonset-example-5w8jx

      Normal  SuccessfulCreate  74s   daemonset-controller  Created pod: nginx-daemonset-example-jsh7x

      Normal  SuccessfulDelete  73s   daemonset-controller  Deleted pod: nginx-daemonset-example-tzqmc

      Normal  SuccessfulCreate  41s   daemonset-controller  Created pod: nginx-daemonset-example-kgqbj

You can see that the Pods were replaced one by one. This is because we had the default value of .spec.updateStrategy.rollingUpdate.maxUnavailable, which is 1.

Tip

You can change the DaemonSet container image imperatively using the kubectl set image ds nginx-daemonset-example nginx=nginx:1.18 --record command. This approach is recommended only for non-production scenarios.

Additionally, DaemonSet will automatically create Pods if a new Node joins the cluster (providing that it matches the selector and affinity parameters). If a Node is removed from the cluster, the Pod will be terminated also. The same will happen if you modify the labels or taints on a Node so that it matches the DaemonSet – a new Pod will be created for that Node. If you modify the labels or taints for a Node in a way that it no longer matches the DaemonSet, the existing Pod will be terminated.

Next, we will show how you can delete a DaemonSet.

Deleting a DaemonSet

In order to delete a DaemonSet object, there are two possibilities:

  • Delete the DaemonSet together with Pods that it owns.
  • Delete the DaemonSet and leave the Pods unaffected.

To delete the DaemonSet together with Pods, you can use the regular the kubectl delete command:

$ kubectl delete ds nginx-daemonset-example

You will see that the Pods will first get terminated and then the DaemonSet will be deleted.

Now, if you would like to delete just the DaemonSet, you need to use the --cascade=orphan option with kubectl delete:

$ kubectl delete ds nginx-daemonset-example --cascade=orphan

After this command, if you inspect what Pods are in the cluster, you will still see all the Pods that were owned by the nginx-daemonset-example DaemonSet.

Important note

If you are draining a node using the kubectl drain command and this node is running Pods owned by a DaemonSet, you need to pass the --ignore-daemonsets flag to drain the node completely.

Let's now take a look at the most common use cases for DaemonSets in Kubernetes.

Common use cases for DaemonSets

At this point, you may wonder what is the actual use of the DaemonSet and what are the real-life use cases for this Kubernetes object? In general, DaemonSets are used either for very fundamental functions of the cluster, without which it is not useable, or for helper workloads performing maintenance or data collection. We have summarized the common and interesting use cases for DaemonSets in the following points:

And the list goes on – as you can see, DaemonSet is another building block provided for engineers designing the workloads running on Kubernetes clusters. In many cases, DaemonSets are the hidden backbone of a cluster that makes it fully operational.

Next, let's discuss what possible alternatives there are to using DaemonSets.

Alternatives to DaemonSets

The reason for using DaemonSets is quite simple – you would like to have exactly one Pod with a particular function on each Node in the cluster. However, sometimes you should consider different approaches that may fit your needs better:

  • In log-gathering scenarios, you need to evaluate if you want to design your log pipeline architecture based on DaemonSets or the sidecar container pattern. Both have their advantages and disadvantages, but in general, running sidecar containers may be easier to implement and be more robust, even though it may require more system resources.
  • If you just want to run periodic tasks, and you do not need to do it on each Node in the cluster, a better solution can be using Kubernetes CronJobs. Again, it is important to know what the actual use case is and whether running a separate Pod on each Node is a must-have requirement.
  • Operating system daemons (for example, provided by systemd in Ubuntu) can be used to do similar tasks as DaemonSets. The drawback of this approach is that you cannot manage these native daemons using the same tools as you manage Kubernetes clusters with, for example kubectl. But at the same time, you do not have the dependency on any Kubernetes service, which may be a good thing in some cases.
  • Static Pods (https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/) can be used to achieve a similar result. This type of Pod is created based on a specific directory watched by kubelet for static manifest files. Static Pods cannot be managed using kubectl and they are most useful for cluster bootstraping functions.

Finally, we can now summarize our knowledge about DaemonSets.

Summary

In this chapter, you have learned how to work with DaemonSets in Kubernetes, and how they are used to manage special types of workloads or processes that must run as a singleton on each Node in the cluster. You first created an example DaemonSet and learned what the most important parts of its specification are. Next, you practiced how to roll out a new revision of a DaemonSet to the cluster and saw how you can monitor the deployment. Additionally, we discussed what the most common use cases are for this special type of Kubernetes object and what alternatives there are that you could consider.

This was the last type of Pod management controller that we discuss in this part of the book. In the next part, you will learn all the details required to effectively deploy Kubernetes clusters in different cloud environments. We will first take a look at working with clusters deployed on Google Kubernetes Engine.

Further reading

For more information regarding DaemonSets and their use cases in Kubernetes, please refer to the following Packt books:

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.144.35.148