Monitoring, Alerting, and Auditing

The community seems to be standardizing on Prometheus for monitoring Kubernetes clusters, so a good start is to familiarize yourself with it. Since there are so many moving parts (from nodes to pods to services), alerting on each event is not practical. What you can do, however, is think about who needs to be informed about what kind of event. For example, a policy could be that node-related or namespace-related events are handled by admins, and developers are paged for pod-level events. The same applies more or less for logs, but here you also should be aware of where and when your sensitive data lands on disk; see Chapter 7 for details.

Another useful feature Kubernetes offers via the API server is auditing, effectively recording the sequence of activities affecting the cluster. Different strategies are available in the auditing policy (from no logging to logging event metadata, request and response bodies), and you can choose between a simple log backend as well as using a webhook for integrating with third-party systems.

Host Security

Much discussion of Kubernetes security focuses on the setup of Kubernetes itself, or on securing the containerized applications that run within it. There is another layer to be secured: the host machines on which Kubernetes runs.

Whether they are bare-metal or VMs, there are many steps you can take to restrict access to your hosts that are essentially the same as you would take in a traditional cluster. For example, you should, of course, restrict access to the machines, and you might well configure them to use a dedicated VPN. These general machine security measures are outside the scope of this book, but there are some specific things you would do well to consider when you are running Kubernetes (or any container orchestrator). Resources such as OpenSCAP and OVAL can help with a broader security assessment.

Sandboxing and Runtime Protection

Sandboxing is the ability to isolate containers from each other and from the underlying host, so that code running within one container can’t effect change outside that container.

Runtime protection is the concept of limiting the set of code that can be executed within the container itself. If attackers can access a container, but can’t execute their own code within it, the potential damage is limited.

While the end goal of these two concepts is different, some overlap exists in the mechanisms used to achieve them. For example, seccomp or AppArmor profiles can limit a container to have access to a limited set of system calls. This restricts what the container can do (runtime protection) and increases its isolation by giving it less access to the kernel (sandboxing).

Containers share the host’s kernel, so vulnerabilities in the kernel could conceivably allow an exploit to escape one container and move to another or to the host.

At time of writing, several projects and vendors are aimed at improving sandboxing and/or runtime protection, all with their own strengths and characteristics:

• seccomp is a kernel mechanism for limiting the system calls that application code can make. A securityContext or PodSecurityPolicy can specify the seccomp profile.

• AppArmor and SELinux are kernel security modules, also configurable through profiles attached to securityContext or PodSecurityPolicy in Kubernetes.

• Kata Containers run each application inside a “lightweight” VM (so each has its own kernel).

• Google’s gVisor project runs container code in a sandbox through a kernel API implemented in user space (if that’s not a contradiction in terms).

• Nabla containers use unikernel technology to provide isolation and limit access to the shared kernel.

• Enterprise container security solutions such as those from Aqua Security and Twistlock use regular containers, with proprietary runtime protection technology that includes whitelisting/blacklisting the set of executables that can run in a given container.

Multitenancy

In some deployments, multiple “tenants” using the same cluster don’t fully trust each other or might not be trusted by the cluster operator. Tenants could be users, groups of users, or applications. The level of trust between users depends on the environment; for example, in a platform-as-a-service (PaaS) environment where users can upload and run their own applications, they should be treated as entirely untrusted, whereas in an enterprise, the tenants might map to different organizational teams, who do cooperate and trust each other to some extent, but who want to limit the risk of someone outside one team maliciously or inadvertently affecting another team’s application.

Multitenant isolation has two parts:

• The control plane, so that users can’t, for example, run kubectl commands that impact other tenants’ resources

• The runtime and networking environment, where container workloads should not be able to interfere with each other or steal resources

The Kubernetes namespace gives us the first building block for multitenancy in the control plane. Typically, there will be one namespace per tenant. Users are given RBAC permissions to create, update, and delete resources within the namespace that maps to their tenancy (see Chapter 3 and Chapter 4). Resource quotas allow each namespace (and thereby, each tenant) to be restricted to a limit of compute and storage resources, and of Kubernetes objects (for example, upper bounds on the number of services, secrets, and persistent volume claims allowed within the namespace).

In an enterprise environment, this may be sufficient. Namespace-based RBAC controls mean that one team can’t update application code and associated Kubernetes resources that they are not responsible for. Quotas mean that one team’s application can’t use all available resources so that another is starved.

However, this is unlikely to be sufficient protection in a fully untrusted environment. Here, tenant workloads should be isolated from each other at a container level so that if there were to be an escape from one container (perhaps because a user deploys code with a serious vulnerability, perhaps even deliberately), they can’t affect or inspect other tenants’ applications or data. Container sandboxing, as described in “Sandboxing and Runtime Protection”, is largely designed to solve this problem (for example, the gVisor approach is based on the way Google isolates user workloads from each other in Google App Engine).

Another approach to workload isolation is to assign one or more nodes to each tenant, and then schedule pods so that workloads are only ever colocated with other workloads from the same tenant. This is straightforward (using taints and tolerations), but potentially wasteful unless each tenant needs a node’s worth of compute resources.

In an untrusted multitenant environment, you would want strong network policies to isolate traffic so that it can’t flow between namespaces. See “Network Policies” for more information.

Dynamic Admission Control

From Kubernetes 1.9, dynamic admission controllers allow for flexible, extensible mechanisms for making checks before allowing a resource to be deployed to a cluster. As an example, check out Kelsey Hightower’s Grafeas tutorial, which includes a validating webhook that ensures that only signed images are admitted.

Network Protection

In a typical traditional deployment, a significant proportion of the security measures is network based: firewalling and the use of VPNs come immediately to mind. In the cloud-native world, similar approaches are dedicated to restricting traffic so that only approved flows can take place.

Security solutions for containers have for some years talked about network micro- or nano-segmentation, and these approaches have been made fairly common practice in Kubernetes deployments through the use of network policies (as discussed in “Network Policies”).

At the time of writing, service meshes are a popular topic—although, in our opinion, currently at the stage of “early adopter” rather than “early majority” market penetration.

Static Analysis of YAML

In many organizations, the YAML associated with an application could be written by a developer who may not have intimate knowledge of the security policies that the organization requires (or who may simply make a mistake). Static analysis tools such as kubetest and kubesec can be useful to look for issues in YAML configuration files (for example, checking for the use of the privileged flag for a pod), or to enforce a particular labeling policy. Just like image scanning (see “Scanning Container Images”), this is an example of “shift-left,” where security practices are dealt with and enforced earlier in the development lifecycle.

Fork Bombs and Resource-Based Attacks

A fork bomb is a process that continually launches copies of itself, with the intention of using all the available resources, effectively creating a denial-of-service attack. Kubernetes addresses this by allowing you to configure a limit on the number of processes within a pod. At the time of writing, this is an Alpha feature that may be subject to significant changes in the future.

Other resource-based attacks might involve trying to consume excessive memory and CPU, denying those resources to legitimate workloads. You can set resource limits to limit exposure to this kind of attack.

Cryptocurrency Mining

A famous attack on Tesla exploited control-plane insecurities to allow hackers to use the company’s resources to mine cryptocurrency. Following the advice in Chapter 2 will go a long way to preventing this from happening in your cluster.

Additional research shows other approaches that would-be miners are attempting. Ensuring that only trusted images can run in your cluster would prevent, for example, a bad actor with approved access to the cluster from running an unexpected mining image. See Chapter 5 for advice on preventing untrusted or compromised images from running.

Runtime protection can add another layer of defense to ensure that even if an approved image has a vulnerability that allows code to be injected into a running container, that code can’t be executed.

Monitoring for unusual activity, such as unexpected CPU usage and unexpected resources being scaled out, can help spot when your resources are being used by an attacker. See “Monitoring, Alerting, and Auditing”.

Kubernetes Security Updates

From time to time, security issues in Kubernetes itself are unearthed, and the project has a security process for dealing with these. The project documentation includes instructions for reporting a vulnerability in Kubernetes to the security team.

If you want to be alerted as soon as vulnerabilities in Kubernetes are announced, subscribe to the kubernetes-announce mailing list.

To learn more about the topics discussed in this chapter, check out the resources on the accompanying website, in the “Advanced Topics” section.

We’ve reached the conclusion of the book and want to thank you for sticking with us until the very end. Enjoy Kubernetes in production—and stay safe!