Chapter 6: Securing Cluster Components

In previous chapters, we looked at the architecture of a Kubernetes cluster. A Kubernetes cluster consists of master components—including kube-apiserver, etcd, kube-scheduler, CoreDNS, kube-controller-manager, and cloud-controller-manager—and node components, including kubelet, kube-proxy, and container-runtime. Master components are responsible for cluster management. They form the control plane of the cluster. Node components, on the other hand, are responsible for the functioning of pods and containers on the node.

In Chapter 3, Threat Modeling, we briefly discussed that components in a Kubernetes cluster need to be configured to ensure the security of the cluster. A compromise of any cluster component can cause a data breach. Misconfiguration of environments is one of the primary reasons for data breaches in traditional or microservices environments. It is important to understand the configurations for each component and how each setting can open up a new attack surface. So, it's important for cluster administrators to understand different configurations.

In this chapter, we look in detail at how to secure each component in a cluster. In many cases, it will not be possible to follow all security best practices, but it is important to highlight the risks and have a mitigation strategy in place if an attacker tries to exploit a vulnerable configuration.

For each master and node component, we briefly discuss the function of components with a security-relevant configuration in a Kubernetes cluster and look in detail at each configuration. We look at the possible settings for these configurations and highlight the recommended practices. Finally, we introduce kube-bench and walk through how this can be used to evaluate the security posture of your cluster.

In this chapter, we will cover the following topics:

• Securing kube-apiserver
• Securing kubelet
• Securing etcd
• Securing kube-scheduler
• Securing kube-controller-manager
• Securing CoreDNS
• Benchmarking a cluster's security configuration

Securing kube-apiserver

kube-apiserver is the gateway to your cluster. It implements a representational state transfer (REST) application programming interface (API) to authorize and validate requests for objects. It is the central gateway that communicates and manages other components within the Kubernetes cluster. It performs three main functions:

• API management: kube-apiserver exposes APIs for cluster management. These APIs are used by developers and cluster administrators to modify the state of the cluster.
• Request handling: Requests for object management and cluster management are validated and processed.
• Internal messaging: The API server interacts with other components in the cluster to ensure the cluster functions properly.

A request to the API server goes through the following steps before being processed:

1. Authentication: kube-apiserver first validates the origin of the request. kube-apiserver supports multiple modes of authentication including client certificates, bearer tokens, and HyperText Transfer Protocol (HTTP) authentication.
2. Authorization: Once the identity of origin is validated, the API server validates that the origin is allowed to execute the request. kube-apiserver, by default, supports Attribute-Based Access Control (ABAC), Role-Based Access Control (RBAC), node authorization, and Webhooks for authorization. RBAC is the recommended mode of authorization.
3. Admission controller: Once kube-apiserver authenticates and authorizes the request, admission controllers parse the request to check if it's allowed within the cluster. If the request is rejected by any admission controller, the request is dropped.

kube-apiserver is the brain of the cluster. Compromise of the API server causes cluster compromise, so it's essential that the API server is secure. Kubernetes provides a myriad of settings to configure the API server. Let's look at some of the security-relevant configurations next.

To secure the API server, you should do the following:

• Disable anonymous authentication: Use the anonymous-auth=false flag to set anonymous authentication to false. This ensures that requests rejected by all authentication modules are not treated as anonymous and are discarded.
• Disable basic authentication: Basic authentication is supported for convenience in kube-apiserver and should not be used. Basic authentication passwords persist indefinitely. kube-apiserver uses the --basic-auth-file argument to enable basic authentication. Ensure that this argument is not used.
• Disable token authentication: --token-auth-file enables token-based authentication for your cluster. Token-based authentication is not recommended. Static tokens persist forever and need a restart of the API server to update. Client certificates should be used for authentication.
• Ensure connections with kubelet use HTTPS: By default, --kubelet-https is set to true. Ensure that this argument is not set to false for kube-apiserver.
• Disable profiling: Enabling profiling using --profiling exposes unnecessary system and program details. Unless you are experiencing performance issues, disable profiling by setting --profiling=false.
• Disable AlwaysAdmit: --enable-admission-plugins can be used to enable admission control plugins that are not enabled by default. AlwaysAdmit accepts the request. Ensure that the plugin is not in the --enabled-admission-plugins list.
• Use AlwaysPullImages: The AlwaysPullImages admission control ensures that images on the nodes cannot be used without correct credentials. This prevents malicious pods from spinning up containers for images that already exist on the node.
• Use SecurityContextDeny: This admission controller should be used if PodSecurityPolicy is not enabled. SecurityContextDeny ensures that pods cannot modify SecurityContext to escalate privileges.
• Enable auditing: Auditing is enabled by default in kube-apiserver. Ensure that --audit-log-path is set to a file in a secure location. Additionally, ensure that the maxage, maxsize, and maxbackup parameters for auditing are set to meet compliance expectations.
• Disable AlwaysAllow authorization: Authorization mode ensures that requests from users with correct privileges are parsed by the API server. Do not use AlwaysAllow with --authorization-mode.
• Enable RBAC authorization: RBAC is the recommended authorization mode for the API server. ABAC is difficult to use and manage. The ease of use, and easy updates to, RBAC roles and role bindings makes RBAC suitable for environments that scale often.
• Ensure requests to kubelet use valid certificates: By default, kube-apiserver uses HTTPS for requests to kubelet. Enabling --kubelet-certificate-authority, --kubelet-client-key, and --kubelet-client-key ensures that the communication uses valid HTTPS certificates.
• Enable service-account-lookup: In addition to ensuring that the service account token is valid, kube-apiserver should also verify that the token is present in etcd. Ensure that --service-account-lookup is not set to false.
• Enable PodSecurityPolicy: --enable-admission-plugins can be used to enable PodSecurityPolicy. As we have seen in Chapter 5, Configuring Kubernetes Security Boundaries, PodSecurityPolicy is used to define the security-sensitive criteria for a pod. We will dive deep into creating pod security policies in Chapter 8, Securing Kubernetes Pods.
• Use a service account key file: Use of --service-account-key-file enables rotation of keys for service accounts. If this is not specified, kube-apiserver uses the private key from the Transport Layer Security (TLS) certificates to sign the service account tokens.
• Enable authorized requests to etcd: --etcd-certfile and --etcd-keyfile can be used to identify requests to etcd. This ensures that any unidentified requests can be rejected by etcd.
• Do not disable the ServiceAccount admission controller: This admission control automates service accounts. Enabling ServiceAccount ensures that custom ServiceAccount with restricted permissions can be used with different Kubernetes objects.
• Do not use self-signed certificates for requests: If HTTPS is enabled for kube-apiserver, a --tls-cert-file and a --tls-private-key-file should be provided to ensure that self-signed certificates are not used.
• Secure connections to etcd: Setting --etcd-cafile allows kube-apiserver to verify itself to etcd over Secure Sockets Layer (SSL) using a certificate file.
• Use secure TLS connections: Set --tls-cipher-suites to strong ciphers only. --tls-min-version is used to set the minimum-supported TLS version. TLS 1.2 is the recommended minimum version.
• Enable advanced auditing: Advanced auditing can be disabled by setting the --feature-gates to AdvancedAuditing=false. Ensure that this field is present and is set to true. Advanced auditing helps in an investigation if a breach happens.

On Minikube, the kube-apiserver configuration looks like this:

$ps aux | grep kube-api

root      4016  6.1 17.2 495148 342896 ?       Ssl  01:03   0:16 kube-apiserver --advertise-address=192.168.99.100 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/var/lib/minikube/certs/ca.crt --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=8443 --service-account-key-file=/var/lib/minikube/certs/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/var/lib/minikube/certs/apiserver.crt --tls-private-key-file=/var/lib/minikube/certs/apiserver.key

As you can see, by default on Minikube, kube-apiserver does not follow all security best practices. For example, PodSecurityPolicy is not enabled by default, and strong cipher suites and the tls minimum version are not set by default. It's the responsibility of the cluster administrator to ensure that the API server is securely configured.

Securing kubelet

kubelet is the node agent for Kubernetes. It manages the life cycle of objects within the Kubernetes cluster and ensures that the objects are in a healthy state on the node.

To secure kubelet, you should do the following:

• Disable anonymous authentication: If anonymous authentication is enabled, requests that are rejected by other authentication methods are treated as anonymous. Ensure that --anonymous-auth=false is set for each instance of kubelet.
• Set the authorization mode: The authorization mode for kubelet is set using config files. A config file is specified using the --config parameter. Ensure that the authorization mode does not have AlwaysAllow in the list.
• Rotate kubelet certificates: kubelet certificates can be rotated using a RotateCertificates configuration in the kubelet configuration file. This should be used in conjunction with RotateKubeletServerCertificate to auto-request rotation of server certificates.
• Provide a Certificate Authority (CA) bundle: A CA bundle is used by kubelet to verify client certificates. This can be set using the ClientCAFile parameter in the config file.
• Disable the read-only port: The read-only port is enabled for kubelet by default, and should be disabled. The read-only port is served with no authentication or authorization.
• Enable the NodeRestriction admission controller: The NodeRestriction admission controller only allows kubelet to modify the node and pod objects on the node it is bound to.
• Restrict access to the Kubelet API: Only the kube-apiserver component interacts with the kubelet API. If you try to communicate with the kubelet API on the node, it is forbidden. This is ensured by using RBAC for kubelet.

On Minikube, the kubelet configuration looks like this:

root      4286  2.6  4.6 1345544 92420 ?       Ssl  01:03   0:18 /var/lib/minikube/binaries/v1.17.3/kubelet --authorization-mode=Webhook --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cgroup-driver=cgroupfs --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-domain=cluster.local --config=/var/lib/kubelet/config.yaml --container-runtime=docker --fail-swap-on=false --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.99.100 --pod-manifest-path=/etc/kubernetes/manifests

Similar to the API server, not all secure configurations are used by default on a kubelet—for example, disabling the read-only port. Next, we talk about how cluster administrators can secure etcd.

Securing etcd

etcd is a key-value store that is used by Kubernetes for data storage. It stores the state, configuration, and secrets of the Kubernetes cluster. Only kube-apiserver should have access to etcd. Compromise of etcd can lead to a cluster compromise.

To secure etcd, you should do the following:

• Restrict node access: Use Linux firewalls to ensure that only nodes that need access to etcd are allowed access.
• Ensure the API server uses TLS: --cert-file and --key-file ensure that requests to etcd are secure.
• Use valid certificates: --client-cert-auth ensures that communication from clients is made using valid certificates, and setting --auto-tls to false ensures that self-signed certificates are not used.
• Encrypt data at rest: --encryption-provider-config is passed to the API server to ensure that data is encrypted at rest in etcd.

On Minikube, the etcd configuration looks like this:

$ ps aux | grep etcd

root      3992  1.9  2.4 10612080 48680 ?      Ssl  01:03   0:18 etcd --advertise-client-urls=https://192.168.99.100:2379 --cert-file=/var/lib/minikube/certs/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/minikube/etcd --initial-advertise-peer-urls=https://192.168.99.100:2380 --initial-cluster=minikube=https://192.168.99.100:2380 --key-file=/var/lib/minikube/certs/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://192.168.99.100:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://192.168.99.100:2380 --name=minikube --peer-cert-file=/var/lib/minikube/certs/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/var/lib/minikube/certs/etcd/peer.key --peer-trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt

etcd stores sensitive data of a Kubernetes cluster, such as private keys and secrets. Compromise of etcd is compromise of the api-server component. Cluster administrators should pay special attention while setting up etcd.

Securing kube-scheduler

Next, we look at kube-scheduler. As we have already discussed in Chapter 1, Kubernetes Architecture, kube-scheduler is responsible for assigning a node to a pod. Once the pod is assigned to a node, the kubelet executes the pod. kube-scheduler first filters the set of nodes on which the pod can run, then, based on the scoring of each node, it assigns the pod to the filtered node with the highest score. Compromise of the kube-scheduler component impacts the performance and availability of the pods in the cluster.

To secure kube-scheduler, you should do the following:

• Disable profiling: Profiling of kube-scheduler exposes system details. Setting --profiling to false reduces the attack surface.
• Disable external connections to kube-scheduler: External connections should be disabled for kube-scheduler. AllowExtTrafficLocalEndpoints is set to true, enabling external connections to kube-scheduler. Ensure that this feature is disabled using --feature-gates.
• Enable AppArmor: By default, AppArmor is enabled for kube-scheduler. Ensure that AppArmor is not disabled for kube-scheduler.

On Minikube, the kube-scheduler configuration looks like this:

$ps aux | grep kube-scheduler

root      3939  0.5  2.0 144308 41640 ?        Ssl  01:03   0:02 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/scheduler.conf --authorization-kubeconfig=/etc/kubernetes/scheduler.conf --bind-address=0.0.0.0 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true

Similar to kube-apiserver, the scheduler also does not follow all security best practices such as disabling profiling.

Securing kube-controller-manager

kube-controller-manager manages the control loop for the cluster. It monitors the cluster for changes through the API server and aims to move the cluster from the current state to the desired state. Multiple controller managers are shipped by default with kube-controller-manager, such as a replication controller and a namespace controller. Compromise of kube-controller-manager can result in updates to the cluster being rejected.

To secure kube-controller-manager, you should use --use-service-account-credentials which, when used with RBAC ensures that control loops run with minimum privileges.

On Minikube, the kube-controller-manager configuration looks like this:

$ps aux | grep kube-controller-manager

root      3927  1.8  4.5 209520 90072 ?        Ssl  01:03   0:11 kube-controller-manager --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=0.0.0.0 --client-ca-file=/var/lib/minikube/certs/ca.crt --cluster-signing-cert-file=/var/lib/minikube/certs/ca.crt --cluster-signing-key-file=/var/lib/minikube/certs/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt --root-ca-file=/var/lib/minikube/certs/ca.crt --service-account-private-key-file=/var/lib/minikube/certs/sa.key --use-service-account-credentials=true

Next, let's talk about securing CoreDNS.

Securing CoreDNS

kube-dns was the default Domain Name System (DNS) server for a Kubernetes cluster. The DNS server helps internal objects such as services, pods, and containers locate each other. kube-dns is comprised of three containers, detailed as follows:

• kube-dns: This container uses SkyDNS to perform DNS resolution services.
• dnsmasq: A lightweight DNS resolver. It caches responses from SkyDNS.
• sidecar: This monitors health and handles metrics reporting for DNS.

kube-dns has been superseded by CoreDNS since version 1.11 because of security vulnerabilities in dnsmasq and performance issues in SkyDNS. CoreDNS is a single container that provides all the functions of kube-dns.

To edit the configuration file for CoreDNS, you can use kubectl, like this:

$ kubectl -n kube-system edit configmap coredns

By default, the CoreDNS config file on Minikube looks like this:

# Please edit the object below. Lines beginning with a '#'

# will be ignored, and an empty file will abort the edit.

# If an error occurs while saving this file will be

# reopened with the relevant failures.

apiVersion: v1

data:

  Corefile: |

    .:53 {

        errors

        health {

           lameduck 5s

        }

        ready

        kubernetes cluster.local in-addr.arpa ip6.arpa {

           pods insecure

           fallthrough in-addr.arpa ip6.arpa

           ttl 30

        }

        prometheus :9153

        forward . /etc/resolv.conf

        cache 30

        loop

        reload

        loadbalance

    }

To secure CoreDNS, do the following:

• Ensure that the health plugin is not disabled: The health plugin monitors the status of CoreDNS. It is used to confirm if CoreDNS is up and running. It is enabled by adding health to the list of plugins to be enabled in Corefile.
• Enable istio for CoreDNS: istio is a service mesh that is used by Kubernetes to provide service discovery, load balancing, and authentication. It is not available by default in Kubernetes and needs to be added as an external dependency. You can add istio to your cluster by starting the istio service and adding a proxy for the istio service to the config file, like this:

  global:53 {

           errors

           proxy . {cluster IP of this istio-core-dns service}

      }

Now that we have looked at different configurations of cluster components, it is important to realize that as the components become more sophisticated, more configuration parameters will be added. It's not possible for a cluster administrator to remember these configurations. So, next, we talk about a tool that helps cluster administrators monitor the security posture of cluster components.

Benchmarking a cluster's security configuration

The Center for Internet Security (CIS) released a benchmark of Kubernetes that can be used by cluster administrators to ensure that the cluster follows the recommended security configuration. The published Kubernetes benchmark is more than 200 pages.

kube-bench is an automated tool written in Go and published by Aqua Security that runs tests documented in the CIS benchmark. The tests are written in YAML Ain't Markup Language (YAML), making it easy to evolve.

kube-bench can be run on a node directly using the kube-bench binary, as follows:

$kube-bench node --benchmark cis-1.4

For clusters hosted on gke, eks, and aks, kube-bench is run as a pod. Once the pod finishes running, you can look at the logs to see the results, as illustrated in the following code block:

$ kubectl apply -f job-gke.yaml

$ kubectl get pods

NAME               READY   STATUS      RESTARTS   AGE

kube-bench-2plpm   0/1     Completed   0          5m20s

$ kubectl logs kube-bench-2plpm

[INFO] 4 Worker Node Security Configuration

[INFO] 4.1 Worker Node Configuration Files

[WARN] 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Not Scored)

[WARN] 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Not Scored)

[PASS] 4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored)

[PASS] 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored)

[WARN] 4.1.5 Ensure that the kubelet.conf file permissions are set to 644 or more restrictive (Not Scored)

[WARN] 4.1.6 Ensure that the kubelet.conf file ownership is set to root:root (Not Scored)

[WARN] 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Not Scored)

......

== Summary ==

0 checks PASS

0 checks FAIL

37 checks WARN

0 checks INFO

It is important to investigate the checks that have a FAIL status. You should aim to have zero checks that fail. If this is not possible for any reason, you should have a risk mitigation plan in place for the failed check.

kube-bench is a helpful tool for monitoring cluster components that are following security best practices. It is recommended to add/modify kube-bench rules to suit your environment. Most developers run kube-bench while starting a new cluster, but it's important to run it regularly to monitor that the cluster components are secure.

Summary

In this chapter, we looked at different security-sensitive configurations for each master and node component: kube-apiserver, kube-scheduler, kube-controller-manager, kubelet, CoreDNS, and etcd. We learned how each component can be secured. By default, components might not follow all the security best practices, so it is the responsibility of the cluster administrators to ensure that the components are secure. Finally, we looked at kube-bench, which can be used to understand the security baseline for your running cluster.

It is important to understand these configurations and ensure that the components follow these checklists to reduce the chance of a compromise.

In the next chapter, we'll look at authentication and authorization mechanisms in Kubernetes. We briefly talked about some admission controllers in this chapter. We'll dive deep into different admission controllers and, finally, talk about how they can be leveraged to provide a finer-grained access control.

Questions

1. What is token-based authentication?
2. What is a NodeRestriction admission controller?
3. How do you ensure data is encrypted at rest in etcd?
4. Why did CoreDNS supersede kube-dns?
5. How do you use kube-bench on an Elastic Kubernetes Service (EKS) cluster?

Further reading

You can refer to the following links for more information on the topics covered in this chapter:

• CIS Benchmarks: https://www.cisecurity.org/benchmark/kubernetes/
• GitHub (kube-bench): https://github.com/aquasecurity/kube-bench