The previous chapters have explained and demonstrated how to use the most common Kubernetes controllers for managing Pods, such as ReplicaSet, Deployment, and StatefulSet. Generally, when running cloud application components that contain the actual business logic you will need either Deployments or StatefulSets for controlling your Pods. In some cases, when you need to run batch workloads as part of your application, you will use Jobs and CronJobs.
However, in some cases, you will need to run components that have a supporting function and, for example, execute maintenance tasks or aggregate logs and metrics. More specifically, if you have any tasks that need to be executed for each Node in the cluster, they can be performed using a DaemonSet. This is the last type of Pod management controller that we are going to introduce in this part of the book. The purpose of a DaemonSet is to ensure that each Node (unless specified otherwise) runs a single replica of a Pod. If you add a new Node to the cluster, it will automatically get a Pod replica scheduled. Similarly, if you remove a Node from the cluster, the Pod replica will be terminated – the DaemonSet will execute all required actions.
In this chapter, we will cover the following topics:
For this chapter, you will need the following:
Kubernetes cluster deployment (local and cloud-based) and kubectl installation were covered in Chapter 3, Installing Your First Kubernetes Cluster.
You can download the latest code samples for this chapter from the official GitHub repository: https://github.com/PacktPublishing/The-Kubernetes-Bible/tree/master/Chapter13.
The term daemon in operating systems has a long history and, in short, is used to describe a program that runs as a background process, without interactive control from the user. In many cases, daemons are responsible for handling maintenance tasks, serving network requests, or monitoring hardware activities. These are often processes that you want to run reliably, all the time, in the background, from the time you boot the operating system to when you shut it down.
Tip
Daemons are associated in most cases with Unix-like operating systems. In Windows, you will more commonly encounter the term Windows service.
In Kubernetes, you may need a similar functionality where your Pods behave like classic operating system daemons on each of the Nodes in the cluster. For this, Kubernetes offers a dedicated Pod management controller named DaemonSet. The role of a DaemonSet is straightforward: run a single Pod replica on each of the Nodes in the cluster and manage them automatically. There are variety of use cases that require such Node-local facilities and usually they serve important and fundamental roles for the whole cluster – we will discuss some common use cases in the sections coming up, but generally you need them for the following:
All you have learned in the previous chapters about ReplicaSets, Deployments, and StatefulSets applies more or less to the DaemonSet. Its specification requires you to provide a Pod template, Pod label selectors, and optionally Node selectors if you want to schedule the Pods only on a subset of Nodes.
Depending on the case, you may not need to communicate with the DaemonSet from other Pods or from an external network. For example, if the job of your DaemonSet is just to perform a periodic cleanup of the filesystem on the Node, it is unlikely you would like to communicate with such Pods. If your use case requires any ingress or egress communication with the DaemonSet Pods, then you have the following common patterns:
We will now show how you can create and manage an example DaemonSet in your cluster.
In order to demonstrate how DaemonSets work, we will use nginx running in a Pod container that returns simple information about the Node IP address where it is currently scheduled. The IP address will be provided to the container using an environment variable and based on that, a modified version of index.html in /usr/share/nginx/html will be created. To access the DaemonSet endpoints, we will use a headless service, similar to what we did for StatefulSet in Chapter 12, StatefulSet – Deploy Stateful Applications. Most of the real use cases of DaemonSets are rather complex and involve mounting various system resources to the Pods. We will keep our DaemonSet example as simple as possible to show the principles.
Important note
If you would like to work on a real example of a DaemonSet, we have provided a working version of Prometheus node-exporter deployed as a DaemonSet behind a headless Service: https://github.com/PacktPublishing/Kubernetes-for-Beginners/blob/master/Chapter13/02_daemonset-prometheus-nodeexporter/node-exporter.yaml. When following the guide in this section, the only difference is that you need to use node-exporter as the Service name, use port 9100 and append the /metrics path for requests sent using wget. This DaemonSet exposes Node metrics in Prometheus data model format on port 9100 under the /metrics path.
We will now go through all the YAML manifests required to create our DaemonSet and apply them to the cluster.
First, let's take a look at the StatefulSet YAML manifest file named nginx-daemonset.yaml (full version available in the official GitHub repository for the book: https://github.com/PacktPublishing/Kubernetes-for-Beginners/blob/master/Chapter13/01_daemonset-example/nginx-daemonset.yaml):
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-daemonset-example
spec:
selector:
matchLabels:
app: nginx-daemon
environment: test
# (to be continued in the next paragraph)
The first part of the preceding file contains the metadata and Pod label selector for the DaemonSet, quite similar to what you have seen in Deployments and StatefulSets. In the second part of the file, we present the Pod template that will be used by the DaemonSet:
# (continued)
template:
metadata:
labels:
app: nginx-daemon
environment: test
spec:
containers:
- name: nginx
image: nginx:1.17
ports:
- containerPort: 80
env:
- name: NODE_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
command:
- /bin/sh
- -c
- |
echo "You have been served by Pod running on Node with IP address: $(NODE_IP)" > /usr/share/nginx/html/index.html
nginx -g "daemon off;"
As you can see, the structure of DaemonSet spec is similar to what you know from Deployments and StatefulSets. The general idea is the same, you need to configure the Pod template and use a proper label selector to match the Pod labels. Note that you do not see the replicas field here, as the number of Pods running in the cluster will be dependent on the number of Nodes in the cluster. The specification has two main components:
It is also common to specify .spec.template.spec.nodeSelector or .spec.template.spec.tolerations in order to control the Nodes where the DaemonSet Pods are deployed. We cover Pod scheduling in detail in Chapter 19, Advanced Techniques for Scheduling Pods. Additionally, you can specify .spec.updateStrategy, .spec.revisionHistoryLimit, and .spec.minReadySeconds, which are similar to what you have learned about Deployment objects.
Tip
If you run hybrid Linux-Windows Kubernetes clusters, one of the common use cases for Node selectors or Node affinity for DaemonSets is ensuring that the Pods are scheduled only on Linux Nodes or only on Windows Nodes. This makes sense as the container runtime and operating system are very different between such Nodes.
Apart from that, in our Pod template we have used a similar override of command for the nginx container as we did in the cases of Deployments and StatefulSets in the previous chapters. The command creates index.html in /usr/share/nginx/html/ with information about the IP address of the Node that runs the Pod serving the request. After that, it starts the nginx web server with the standard entrypoint command for the image. To provide the information about the Node IP address, we use an additional NODE_IP environment variable populated by status.hostIP of the Pod object (at runtime).
Next, let's take a quick look at the headless Service named nginx-daemon-headless. Create an nginx-daemon-headless-service.yaml file with the following YAML manifest:
apiVersion: v1
kind: Service
metadata:
name: nginx-daemon-headless
spec:
selector:
app: nginx-daemon
environment: test
clusterIP: None
ports:
- port: 80
protocol: TCP
targetPort: 80
As we explained in the case of the StatefulSet example, the specification is very similar to a normal Service, the only difference is that it has the None value for the clusterIP field. This will result in the creation of an nginx-daemon-headless headless Service. A headless Service allows us to return all Pods' IP addresses behind the Service as DNS A records instead of a single DNS A record with a Service clusterIP.
We have all the required YAML manifest files for our demonstration and we can proceed with applying the manifests to the cluster. Please follow these steps:
$ kubectl apply -f ./nginx-daemon-headless-service.yaml
$ kubectl apply -f ./nginx-daemonset.yaml
$ kubectl describe daemonset nginx-daemonset-example
$ $ kubectl get pods -o wide
NAME ... IP NODE ...
nginx-daemonset-example-5w8jx ... 10.244.1.144 aks-nodepool1-77120516-vmss000000 ...
nginx-daemonset-example-tzqmc ... 10.244.0.90 aks-nodepool1-77120516-vmss000001 ...
In our case, we have two Nodes in the cluster, so exactly two Pods have been created.
We have successfully deployed the DaemonSet and we can now verify that it works as expected. To do that, follow the given steps:
$ kubectl get node -o wide
NAME ... INTERNAL-IP ...
aks-nodepool1-77120516-vmss000000 ... 10.240.0.4 ...
aks-nodepool1-77120516-vmss000001 ... 10.240.0.5 ...
$ kubectl run -i --tty busybox --image=busybox:1.28 --rm --restart=Never -- sh
$ nslookup nginx-daemon-headless
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: nginx-daemon-headless
Address 1: 10.244.1.144 10-244-1-144.nginx-daemon-headless.default.svc.cluster.local
Address 2: 10.244.0.90 10-244-0-90.nginx-daemon-headless.default.svc.cluster.local
We have been provided with two DNS names for the individual Pods in the DaemonSet: 10-244-1-144.nginx-daemon-headless.default.svc.cluster.local, which belongs to the aks-nodepool1-77120516-vmss000000 Node, and 10-244-0-90.nginx-daemon-headless.default.svc.cluster.local, which belongs to the aks-nodepool1-77120516-vmss000001 Node.
$ wget http://10-244-1-144.nginx-daemon-headless && cat index.html
Connecting to 10-244-1-144.nginx-daemon-headless (10.244.1.144:80)
...
You have been served by Pod running on Node with IP address: 10.240.0.4
As expected, the Pod is scheduled on the aks-nodepool1-77120516-vmss000000 Node, which has an IP address of 10.240.0.4 (you can cross-check this with earlier commands' output).
$ rm index.html && wget http://10-244-0-90.nginx-daemon-headless && cat index.html
Connecting to 10-244-0-90.nginx-daemon-headless (10.244.0.90:80)
...
You have been served by Pod running on Node with IP address: 10.240.0.5
And again, as expected, we have been served by a Pod running on the aks-nodepool1-77120516-vmss000001 Node.
This demonstrates the most important principles underlying how DaemonSet Pods are scheduled and how you can interact with them using headless Services. We will now show how you can modify the DaemonSet to roll out a new version of a container image for the Pods.
Updating a DaemonSet can be done in a similar way as for Deployments. If you modify the Pod template of the DaemonSet, this will trigger a rollout of a new revision of DaemonSet according to its updateStrategy. There are two strategies available:
The rollout of a new DaemonSet revision can be controlled in similar ways as for a Deployment object. You can use the kubectl rollout status command and perform imperative rollbacks using the kubectl rollout undo command. Let's demonstrate how you can declaratively update the container image in a DaemonSet Pod to a newer version:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-daemonset-example
spec:
...
template:
...
spec:
containers:
- name: nginx
image: nginx:1.18
$ kubectl apply -f ./nginx-daemonset.yaml --record
$ kubectl rollout status ds nginx-daemonset-example
Waiting for daemon set "nginx-daemonset-example" rollout to finish: 0 out of 2 new pods have been updated...
Waiting for daemon set "nginx-daemonset-example" rollout to finish: 1 out of 2 new pods have been updated...
daemon set "nginx-daemonset-example" successfully rolled out
$ kubectl describe ds nginx-daemonset-example
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulDelete 113s daemonset-controller Deleted pod: nginx-daemonset-example-5w8jx
Normal SuccessfulCreate 74s daemonset-controller Created pod: nginx-daemonset-example-jsh7x
Normal SuccessfulDelete 73s daemonset-controller Deleted pod: nginx-daemonset-example-tzqmc
Normal SuccessfulCreate 41s daemonset-controller Created pod: nginx-daemonset-example-kgqbj
You can see that the Pods were replaced one by one. This is because we had the default value of .spec.updateStrategy.rollingUpdate.maxUnavailable, which is 1.
Tip
You can change the DaemonSet container image imperatively using the kubectl set image ds nginx-daemonset-example nginx=nginx:1.18 --record command. This approach is recommended only for non-production scenarios.
Additionally, DaemonSet will automatically create Pods if a new Node joins the cluster (providing that it matches the selector and affinity parameters). If a Node is removed from the cluster, the Pod will be terminated also. The same will happen if you modify the labels or taints on a Node so that it matches the DaemonSet – a new Pod will be created for that Node. If you modify the labels or taints for a Node in a way that it no longer matches the DaemonSet, the existing Pod will be terminated.
Next, we will show how you can delete a DaemonSet.
In order to delete a DaemonSet object, there are two possibilities:
To delete the DaemonSet together with Pods, you can use the regular the kubectl delete command:
$ kubectl delete ds nginx-daemonset-example
You will see that the Pods will first get terminated and then the DaemonSet will be deleted.
Now, if you would like to delete just the DaemonSet, you need to use the --cascade=orphan option with kubectl delete:
$ kubectl delete ds nginx-daemonset-example --cascade=orphan
After this command, if you inspect what Pods are in the cluster, you will still see all the Pods that were owned by the nginx-daemonset-example DaemonSet.
Important note
If you are draining a node using the kubectl drain command and this node is running Pods owned by a DaemonSet, you need to pass the --ignore-daemonsets flag to drain the node completely.
Let's now take a look at the most common use cases for DaemonSets in Kubernetes.
At this point, you may wonder what is the actual use of the DaemonSet and what are the real-life use cases for this Kubernetes object? In general, DaemonSets are used either for very fundamental functions of the cluster, without which it is not useable, or for helper workloads performing maintenance or data collection. We have summarized the common and interesting use cases for DaemonSets in the following points:
And the list goes on – as you can see, DaemonSet is another building block provided for engineers designing the workloads running on Kubernetes clusters. In many cases, DaemonSets are the hidden backbone of a cluster that makes it fully operational.
Next, let's discuss what possible alternatives there are to using DaemonSets.
The reason for using DaemonSets is quite simple – you would like to have exactly one Pod with a particular function on each Node in the cluster. However, sometimes you should consider different approaches that may fit your needs better:
Finally, we can now summarize our knowledge about DaemonSets.
In this chapter, you have learned how to work with DaemonSets in Kubernetes, and how they are used to manage special types of workloads or processes that must run as a singleton on each Node in the cluster. You first created an example DaemonSet and learned what the most important parts of its specification are. Next, you practiced how to roll out a new revision of a DaemonSet to the cluster and saw how you can monitor the deployment. Additionally, we discussed what the most common use cases are for this special type of Kubernetes object and what alternatives there are that you could consider.
This was the last type of Pod management controller that we discuss in this part of the book. In the next part, you will learn all the details required to effectively deploy Kubernetes clusters in different cloud environments. We will first take a look at working with clusters deployed on Google Kubernetes Engine.
For more information regarding DaemonSets and their use cases in Kubernetes, please refer to the following Packt books:
3.144.35.148