In previous chapters, we looked at the need to get the permissions correctly configured to protect other containers running on a host and indeed the host itself. In Chapter 6, “Container Image CVEs,” we will also look at protecting against common vulnerabilities and exploits (CVEs) to plug security holes in container images. The third major aspect of container security is at least as important from an operational perspective. That is the need to capture and potentially automatically remediate any issues when anomalous behavior is discovered from your running containers.

Only a handful of trustworthy and battle-worn container runtime security applications exist. Of those there is one Open Source tool that stands out from the crowd. Created by a company called Sysdig (sysdig.com) in 2016 and a member of the Cloud Native Computing Forum (CNCF), Falco (falco.org) excels at both container and host security rules enforcement and alerting. Of the more popular commercial tools there are Prisma Cloud Compute Edition (formerly Twistlock prior to acquisition) and Aqua from AquaSec.

Falco (sysdig.com/opensource/falco) offers exceptional Open Source functionality that can be used to create rulesets to force containers to behave in precisely the way you want. It also integrates with Kubernetes API Audit Events, which means that all sorts of orchestrator actions can be secured in addition. You can find more information here:

falco.org/docs/event-sources/kubernetes-audit.

In this chapter, we will look at installing Falco and then explore its features and how it can help secure our container runtime and underlying hosts, in the same way that some commercial products do, but without any associated fees. We will also explore using some of its rulesets and how to make changes to them yourself.

Running Falco

Following true Cloud Native methodology, we will use a container image to spawn Falco. That said, there are Linux rpm, deb, and binary files that you can install or execute directly, too, which appears to be the preferred route for their installation.

You can run Falco either on a host or by a userland container that additionally needs to access a pre-installed driver on the underlying host. Falco works by tapping into the kernel with elevated permissions to pick up the kernel's system calls (syscalls), and the driver is needed to offer that required functionality. We also need to provide Falco with the requisite permissions to enable such functionality. As described in Chapter 1, “What Is A Container?,” for a container runtime we define these permissions using kernel capabilities. To get an idea of what is available, you could do worse than looking over some of the names of the kernel capabilities in the manual (using the command man capabilities). Various versions of the manual are online too, such as this:

man7.org/linux/man-pages/man7/capabilities.7.html

To protect the underlying host, we will run Falco with as few privileges as possible. Be warned, however, that you will need a kernel version of v5.8 or higher to make use of the extended Berkeley Packet Filter (eBPF) driver without running a one-off --privileged container to install that driver to the underlying host(s) that Falco will run on. The Berkeley Packet Filter has been extended to allow increased access to the networking stack to applications via the kernel.

If you are lucky enough to have a kernel of v5.8 or later, the way around the one-off driver installation is to add the CAP_SYS_BPF option to your running container at startup time, which the more modern kernels will support. Add it using this command-line switch:

--cap--add SYS_BPF

For this demonstration, we will not assume that you have that kernel version, so we will install the driver on a host where we will use the one-off container method. The commands are as follows:

$ docker pull falcosecurity/falco-driver-loader:latest
$ docker run --rm -it --privileged -v /root/.falco:/root/.falco 
-v /proc:/host/proc:ro -v /boot:/host/boot:ro  

-v /lib/modules:/host/lib/modules:ro 
-v /usr:/host/usr:ro -v /etc:/host/etc:ro 
falcosecurity/falco-driver-loader:latest

As you can see, we are using the insecure --privileged switch to gain the elevated permissions required to install the Falco driver. Listing 3.1 shows part of the output from the command, in which Dynamic Kernel Module Support (DKMS) is called into action on Debian derivatives and a kernel module is used.

Although the kernel version (4.15.0.20-generic) seems like a long way off from version 5.8, around version v4.19 the versions jumped to v5.4. To check that the process has automatically loaded up the kernel module as hoped, we can run this lsmod command:

$ lsmod | grep falco
falco                 634880  0

As we can see, falco is present in the list of loaded modules, so we can continue to proceed. Obviously, if you installed packages directly onto the host as the root user, this step would not be needed, but it is important to illustrate that container protection security tools also have trade-offs, and suffice to say functionality like rootless mode will not accommodate such functionality without some heartache. Relinquishing an undefined security control, such as having a common attack vector across all hosts, to onboard a security tool to protect running containers is a necessary evil; in this case, the kernel module is essential to Falco's functionality. Be aware that you are allowing the tool to tap into the very lowest level of a host's innards (and its running containers), so you need to be completely sure that the security product to which you are offering privileged access is fully trustworthy. On a large, containerized estate, with orchestrators and potentially tens of thousands of running containers on differing varieties of hosts, the fact that you are adding another attack vector to each and every host in the estate needs to be carefully considered. You are effectively opening up a predictable security hole (that is, it is predictable if an attacker knows that a privileged container runs on each host) that can be exploited throughout the estate if a vulnerability is found.

Next, to run our Falco container, we will run the following long command all on one line ideally to enable the kernel capability CAP_SYS_PTRACE. According to the SYS_PTRACE man page (man7.org/linux/man-pages/man2/ptrace.2.html), we can control and manipulate other processes with this privilege as well as move data into the memory space of processes.

$ docker run --rm -it --security-opt apparmor:unconfined  
--cap-add SYS_PTRACE  
--pid=host $(ls /dev/falco* | xargs -I {} echo --device {}) -v
/var/run/docker.sock:/var/run/docker.sock  
falcosecurity/falco-no-driver:latest

Note that we're demonstrating Falco on a Linux Mint machine (which is based on Ubuntu 18.04), and this command uses AppArmor effectively to stop rogue processes accessing several locked-away parts of a system. To use it, we also need to add the following switch to provide the required permissions to our container:

--security-opt apparmor:unconfined

As demonstrated in Chapter 1, you might also recognize that the container is offered the ability to access the host's process table namespace with the --pid switch on the Docker command.

Think about this for a moment. From a security vendor's perspective, AppArmor has clearly made an effort to reduce the attack surface its product brings to each host. However, from an organization's point of view, there's definitely a significant trade-off. We are effectively switching off all the protection afforded by AppArmor for this container and offering the tool the ability to poison or break other processes. That applies not just to our container runtime but our host(s) as a whole. Do not be mistaken; Falco is certainly not alone when it comes to this elevated permissions requirement for runtime protection.

After we have run the previous command, its brief output includes information as follows:

2020-08-09T12:27:54+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2020-08-09T12:27:54+0000: Loading rules from file
/etc/falco/falco_rules.yaml:
2020-08-09T12:27:54+0000: Loading rules from file
/etc/falco/falco_rules.local.yaml:
2020-08-09T12:27:54+0000: Loading rules from file
/etc/falco/k8s_audit_rules.yaml:

Thanks to the fact that we entered the command as shown earlier, without adding -d to daemonize the container and detach the terminal from it, the STDOUT output (direct to the terminal) immediately starts listing some useful insights into what's happening on the host machine. Let's see what we can expect from Falco by looking at some of the output now. The first example is related to filesystem access:

2020-08-09T13:35:47.930163243+0000: Warning Sensitive file opened for 
reading by non-trusted program (user=<NA> program=pkexec 
command=pkexec 
/usr/lib/x86_64-linux-gnu/cinnamon-settings-daemon/csd-backlight-helper
--set-brightness 828 -b firmware -b platform 
-b raw file=/etc/pam.d/common-account parent=csd-power 
gparent=cinnamon-sessio ggparent=lightdm gggparent=lightdm 
container_id=host image=<NA>)

We can see that “Sensitive file opened for reading by non-trusted program” has flagged an issue. Let's try to spawn a container from an image:

2020-08-09T13:45:46.935191270+0000: Notice A shell was spawned in a 
container with an attached terminal (user=root <NA> (id=8f31495aeedf)
shell=bash parent=<NA> cmdline=bash terminal=34816
container_id=8f31495aeedf image=<NA>)

As we can see, Bash was used to access a running container. The flagged issue is listed as “A shell was spawned in a container with an attached terminal.”

Another flagged issue, this time more specific to the host, is as shown here:

2020-08-09T13:48:37.040867784+0000: Error File below / or /root opened 
for writing (user=root command=bash parent=sudo 
file=/root/.bash_history-18236.tmp program=bash container_id=host
image=<NA>)
2020-08-09T13:48:37.041053025+0000: Warning Shell history had been 
deleted or renamed (user=root type=rename command=bash 
fd.name=<NA> name=<NA> path=<NA>
oldpath=/root/.bash_history-18236.tmp host (id=host))

We can see that in the /root directory a process has written to a temporary file while the .bash_history file, used to record typed Bash commands, was probably opened/closed and appended to.

Another example alert might be this container warning:

2020-08-09T15:41:28.324617000+0000: Notice Container with sensitive 
mount started (user=root command=container:3369c68859c6 
dangly_goldwasser (id=3369c68859c6)
image=falcosecurity/falco-no-driver:latest mounts=/var/run/docker.sock:/var/run/docker.sock::true:rprivate)

We can see that a volume has been mounted by none other than Falco itself so that it can mount the Docker socket to tap into Docker Engine.

Configuring Rules

Next, we will look at how Falco's rulesets are constructed. Here is a more desktop-oriented rule, which should prevent applications (other than Skype or WebEx) from accessing the local camera:

- rule: access_camera
  desc: a process other than skype/webex tries to access the camera
  condition: evt.type = open and fd.name = /dev/video0 and not proc.name
  in (skype, webex)
  output: Unexpected process opening camera video device
 (command=%proc.cmdline)
  priority: WARNING

As we can see, the rule consists of a name and description followed by three criteria. They are the condition Falco should look out for, the output it should report, and the priority level of the output.

Here is a container-specific rule to examine a bit closer:

- rule: change_thread_namespace
  desc: an attempt to change a program/thread's namespace
  (commonly done as a part of creating a container) by calling setns.
  condition: syscall.type = setns and not proc.name in
  (docker, sysdig, dragent)
  output: "Namespace change (setns) by unexpected program
  (user=%user.name command=%proc.cmdline container=%container.id)"
  priority: WARNING

This rule pays close attention to a container moving between namespaces. The setns syscall that is marked as important is used to change namespace. The rule, however, ignores the event if docker, sysdig, or dragent initiate it.

Another example is a case study that Sysdig wrote about to help explain how a CVE could be mitigated using Falco, at the end of 2019. It was CVE-2019-14287 (cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14287) that allowed a simple command to be run to make the sudo command run commands as the root user. To exploit the CVE, it was apparently as simple as using the sudo command as follows:

$ sudo -u#-1

In Listing 3.2 we can see the rule that the Sysdig team concocted to detect and then block the exploit within the CVE.

Summary

There is no doubt that Falco offers extensive security functionality for both container runtime and hosts. It also plays nicely with Kubernetes, and as of version v0.13.0 API Audit Events can be captured as an event source so that Falco can be rolled out across a cluster to offer genuine insight into what containers and hosts are getting up to in the quiet hours. Supported Kubernetes actions from Falco include the creation and deletion of resources (pods, deployments, daemon sets, and so on), changes to ConfigMaps and secrets, volume mounts, host networking, granting cluster-admin access, and using ConfigMaps for overly sensitive information.

Even the Open Source version of Falco is an impressive, battle-hardened piece of software. And, its commercial products have a notable enterprise client list using the paid-for Enterprise Falco and Sysdig Secure products. If you are to trust your cloud estate with security tools running with elevated permissions, then clearly it makes sense to use the most reputable tool that you can find on the market to avoid bouts of insomnia.