Applying the Principle of Least Privilege

The principle of least privilege has two consequences on secrets management in Kubernetes:

• We want to ensure that containerized code can read only the secrets that it needs.

• It’s a good idea to have a different set of secrets for different environments (like production, development, and testing). The development and test credentials can be shared with a wider set of team members without necessarily giving them full access to the production credentials.

Secret Encryption

Since secret values protect sensitive data, we want them to be hard to access. Ideally, they should by protected at rest and in transit:

At rest

Secrets should always be stored on disk in encrypted form, so that an attacker with access to the filesystem cannot simply read them from the file. In this chapter, you will see how secrets are stored in Kubernetes, and our options for encrypting at rest.

In transit

Secrets should be encrypted whenever they are sent as network traffic, so that an attacker snooping on the network cannot read them as they pass.

Encryption in transit is achieved by encrypting the traffic between the Kubernetes control-plane components using TLS (see Chapter 2). In order to consider how we can store our secrets safely in encrypted form, let’s look at how Kubernetes manages secrets, and the options for where they can be stored.

Kubernetes Secret Storage

The Kubernetes secret resource type is a mechanism for passing secrets to your code without them appearing in plain text in the pod’s YAML. Instead, the pod specification refers to a secret by name, and the actual value of the secret is configured separately.

The default storage for secret values is etcd, or you can use third-party secret storage solutions.

Passing Secrets into Containerized Code

There are three ways to get secrets (or any other kind of information) into a container so that they are accessible to the application:

• Building them into the image itself

• Passing them into the container as environment variables

• Mounting a volume into a container so that code can read the information out of a file on that volume

You might be thinking that the container application could query secrets through some kind of network activity, but then the question arises: how do you stop that information from being available to bad actor entities without requiring credentials of some kind? Then those credentials are a secret that needs to be passed into the container first, and we are back to the set of three options.

Kubernetes secrets support the last two of these approaches, although for reasons we will cover shortly, the third option of mounting a volume is generally considered the safest, so if you are short of time, skip ahead to “Passing Secrets in Files”. Before we come to that, let’s consider why building secrets into container images is really not a great idea.

$ kubectl describe pod nginx-env-6c5b7b8ddd-dwpvk
Name:           nginx-env-6c5b7b8ddd-dwpvk
...
Containers:
  nginx-env:
  ...
    Environment:
      NOT_SO_SECRET:  some_value
  ...

$ sudo docker inspect b5ad78e251a3
[
  {
   "Id": "b5ad78e251a3f94c10b9336ccfe88e576548b4f387f5c7040...",
   ...
   "Config": {
      "Hostname": "nginx-env-6c5b7b8ddd-dwpvk",
     ...
     "Env": [
      "NOT_SO_SECRET=some_value",
      ...
    ]
   }
  }
]

Secret Rotation and Revocation

For humans, it is no longer considered advisable to require frequent password changes, but this advice doesn’t apply to the secrets used by machines. The longer a given secret remains valid, the more likely that it has been compromised. By regularly changing, or “rotating,” secret values, we can ensure that a secret stops being of any use to an attacker.

For both the human- and machine-readable credentials related to your applications, you should have a mechanism in place that allows you to revoke a value if you discover it has been compromised. If you have a regular secret rotation process in place, you can have confidence that you can revoke or change a secret in the event of an emergency without harmful effects on the system.

Depending on how your application code is written, you may need to restart a pod in order for a new secret value to take effect. For example, an application might read a database password (from a file or from an environment variable) just once as part of its initialization; it would need to be restarted in order to read the new value when it changes.

A different approach in the application might be to reread the secret value, perhaps on a regular basis, or in response to a failure using the currently held value. If the application code can cope with a secret being updated, this leads us to a benefit of the file-based approach to passing secrets: Kubernetes can update the secret value written to file without having to restart the pod. If you’re using the environment variable mechanism in Kubernetes to pass secrets, they can’t be updated live without a pod restart (although commercial solutions provide this capability).

Secret Access from Within the Container

If attackers gain execution access to a container, there is a high likelihood they will have access to the secrets within that container. This includes access via kubectl exec and docker exec.

In this situation, runtime protection can help (see “Sandboxing and Runtime Protection”). The fewer tools that the attacker can run inside the container, the better. For example, if the secrets are held in a file, preventing the attacker from being able to run commands like cat, more, or less make it harder to reach the secrets.

You can also limit the attacker’s ability to access secrets by not building those commands into the container image in the first place. See Chapter 5 for more on reducing the attack surface in an image.

Secret Access from a Kubelet

Prior to Kubernetes 1.7, any kubelet could access any secret, but nowadays node authorization ensures that a kubelet can access only the secrets related to those pods that are scheduled to its node. If a node is compromised, this limits the effect of secrets access. You can ensure that this is in use by confirming that --enable-admission-plugins includes NodeRestriction.

To learn more about secrets and how to use them, check out the resources on the accompanying website, in the “Secrets” section.

So far, we have discussed how to set up your Kubernetes cluster with security in mind, and approaches for building and running application code safely. We now move on to discuss advanced security features that might apply, depending on your particular needs, and some new projects and capabilities that are under development at the time of writing.