native, lxc, windows, etc.

Originally the Docker project leveraged work done by the Linux Containers Project (LXC) to control container life cycles. LXC was the one and only execution driver until the release of Docker 0.9 and the newly added support for a standardized execution driver API. This was done with the intention of bringing many other virtualization layers into the fold, including non-container engines. It was envisioned that things like FreeBSD jails and Solaris Zones might appear as execution drivers. We haven’t seen that really play out so far. The LXC driver is no more, and now Docker automatically utilizes either the native or windows execution drivers that ship with Docker.

Because LXC was not a part of the Docker project, it was hard for Docker to ensure that the LXC project didn’t introduce changes that caused issues with Docker. It also made it challenging for Docker to ensure that important changes required in the next LXC release were prioritized.

As a result, in version 0.9, the LXC execution driver was replaced with libcontainer, a Docker-native Go library for working with containers, namespaces, and cgroups. All modern Docker builds use libcontainer as the default backend.

AUFS, Device Mapper, BTRFS, vfs, etc.

Various backends have different limitations that may or may not make them your best choice. In some cases, your choices of which backend to use are limited by what your distribution of Linux actually supports. Using drivers that are built in to the kernel that your distribution ships with will make life ever so much easier. It’s generally best to stay near the tested path here as well. We’ve seen all manner of oddities from various backends since Docker’s release. And, as usual, the common case is always the best supported one. Different backends also report different statistics up through the Docker Remote API (/info endpoint). This is potentially useful for monitoring your Docker systems.

AUFS

The original backend, and at the time of this writing the officially recommended one, is AUFS: a union filesystem driver with reasonable support on various popular Linux distributions. It was never accepted into the mainline kernel, however, and this has limited its availability on various distributions. It is not supported on recent versions of RedHat, Fedora, or CentOS, for example. It is not shipped in the standard Ubuntu distribution, but is in the Ubuntu linux-image-extra package.

Its status as a second-class citizen in the kernel has lead to the development of many of the other backends now available. Older, but still recent, versions of AUFS had a limitation of 42 layers, which might constrain how you build base images if you are running on such a version. If you are shipping images for public consumption, you should definitely keep this limitation in mind because even if you don’t have it, someone else probably does. The current limit in Docker for AUFS is 127 layers, which is probably well more than you should ever use for performance reasons. AUFS has been a pretty good performer on recent kernels and is quite well-tested with Docker.

devicemapper

RedHat’s various distributions have not supported AUFS recently, so RedHat contributed a backend to the Docker project based on devicemapper, which is a heavily tested layer of the Linux kernel that underpins things like LVM, disk encryption, and other software RAID implementations.

The Docker backend was written quickly to just get some support for Docker into RedHat-based distributions, and at first had some major flaws. Most of these have been addressed now and it’s reasonably stable. But even in later versions of Docker, it has been shown to be only somewhat reliable in production. Unlike AUFS, which can usually be unloaded from the kernel and then reloaded, devicemapper often has other kernel modules that depend on it. That means that the worst failure modes currently require a reboot of the whole server on which Docker is running. Performance is reasonable, but no one would call it speedy when using the loopback mode (the default). It does support using disk partitions raw, which should be faster. It does not have much tolerance for having anything at all written into the container during runtime. It’s the default choice on RedHat/CentOS distributions before RedHat/CentOS 7.

BTRFS

btrfs is fundamentally a copy-on-write filesystem, which means it’s a pretty good fit for the Docker image model. On systems that don’t support AUFS and where the btrfs driver is present, it’s the default backend. This includes, for example, RHEL and CentOS 7. It also works on various Ubuntu versions. Like AUFS and unlike devicemapper, Docker is using the backend in the way it was intended. That means it’s both pretty stable in production and also a good performer. It scales reasonably to thousands of containers on the same system. A major drawback for Red Hat–based systems is that btrfs does not support SELinux. If you have btrfs available, we currently recommend it as the most stable backend for production. The space is changing rapidly, however, and new backends keep becoming available.

vfs

The vfs driver is the simplest, and slowest, to start up of the supported drivers. It doesn’t really support copy-on-write. Instead, it makes a new directory and copies over all of the existing data. It was originally intended for use in tests and for mounting host volumes. It is very slow to create new containers, but runtime performance is native, which is a real benefit. It is very simple in mechanism, which means there is less to go wrong. Docker, Inc., does not recommend it for production use so you should proceed with caution if you think it’s the right solution for your production environment.

overlayfs

overlayfs is now the union filesystem driver that is supported in the mainline Linux kernel as of version 3.18. That’s good news for its long-term support. It also means that it’s likely to get a lot of attention to performance and will be available on most Linux distributions once they catch up with version 3.18 and higher kernels. It is a bit like AUFS but fundamentally simpler underneath, which leads to very strong performance. The Docker backend is still under active development, but we expect it to be a good option going forward.

You can use docker info to see which storage backend your system is running. It will also tell you what the underlying filesystem is in cases where there is one. In some cases, like with devicemapper on raw partitions or with btrfs, there won’t be a different underlying filesystem.

Like with execution drivers, storage backends can be swapped via command-line arguments to docker on startup. If we wanted to switch our Ubuntu system from AUFS to devicemapper, we would do that like this:

$ docker daemon --storage-driver=devicemapper

That will work on pretty much any Linux system that can support Docker because devicemapper is almost always present. You will need to have the actual underlying dependencies in place for the other drivers. For example, without AUFS in the kernel—usually via a kernel module—Docker will not start up with AUFS set as the storage driver.

Getting the right storage driver for your systems and deployment needs is one of the more important technical items to get right when taking Docker to production. Be conservative; make sure the path you choose is well-supported in your kernel and distribution.

The /sys filesystem

The primary way to control cgroups in a fine-grained manner, even if you configured them with Docker, is to manage them yourself. This is the most powerful method because changes don’t just happen at creation time—they can be done on the fly.

On systems with systemd, there are command-line tools like systemctl that you can use to do this. But since cgroups are built into the kernel, the method that works everywhere is to talk to the kernel directly via the /sys filesystem. If you’re not familiar with /sys, it’s a filesystem that directly exposes a number of kernel settings and outputs. You can use it with simple command-line tools to tell the kernel how to behave in a number of ways.

It’s important to note that this method of configuring cgroups controls for containers only works directly on the Docker server and is not available remotely via any API. If you use this method, you’ll need to figure out how to script this for your own environment.

Let’s use an example of changing the CPU cgroups settings for a container we already have running. First we need to get the long ID of the container, and then we need to find it in the /sys filesystem. Here’s what that looks like:

$ docker ps --no-trunc
CONTAINER ID IMAGE        COMMAND       CREATED     STATUS    NAMES
dcbbaa763... 0415448f2cc2 "supervisord" 3 weeks ago Up 2 days romantic_morse

Here we’ve had docker ps give us the long ID in the output, and the ID we want is “dcbbaa763daff1dc0a91e7675d3c93895cb6a6d83371e25b7f0bd62803ed8e86”. You can see why Docker normally truncates this. In the examples we’re going to truncate it, too, to make it at least a little readable and fit into the constraints of a printed page. But you need to use the long one!

Now that we have the ID, we can find our container’s cgroup in the /sys filesystem. Cgroups are laid out so that each kind of setting is grouped into a module and that module is exposed at a different place in the /sys filesystem. So when we look at CPU settings, we won’t see blkio settings, for example. You might take a look around in the /sys to see what else is there. But for now we’re looking at the CPU controller, so let’s inspect what that gives us. You need root access on the system to do this because you’re manipulating kernel settings:

$ ls /sys/fs/cgroup/cpu/docker/dcbbaa763daf
cgroup.clone_children  cpu.cfs_period_us  cpu.rt_runtime_us  notify_on_release
cgroup.event_control   cpu.cfs_quota_us   cpu.shares         tasks
cgroup.procs           cpu.rt_period_us   cpu.stat

You can see that under cgroups, there is a docker directory that contains all of the Docker containers that are running on this host. You can’t set cgroups for things that aren’t running because they apply to running processes. This is an important point that you should consider. Docker takes care of reapplying cgroups settings for you when you start and stop containers. Without that mechanism, you are somewhat on your own.

Back to our task. Let’s inspect the CPU shares for this container. Remember that we earlier set these via the Docker command-line tool. But for a normal container where no settings were passed, this setting is the default:

$ cat /sys/fs/cgroup/cpu/docker/dcbbaa763daf/cpu.shares
1024

1024 CPU shares means we are not limited at all. Let’s tell the kernel that this container should be limited to half that:

$ echo 512 > /sys/fs/cgroup/cpu/docker/dcbbaa763daf/cpu.shares
$ cat /sys/fs/cgroup/cpu/docker/dcbbaa763daf/cpu.shares
512

There you have it. We’ve changed the container’s settings on the fly. This method is very powerful because it allows you to set any cgroups setting at all for the container. But as we mentioned earlier, it’s entirely ephemeral. When the container is stopped and restarted, the setting is reset to the default:

$ docker stop dcbbaa763daf
dcbbaa763daf
$ cat /sys/fs/cgroup/cpu/docker/dcbbaa763daf/cpu.shares
cat: /sys/fs/.../cpu.shares: No such file or directory

You can see that the directory path doesn’t even exist any more now that the container is stopped. And when we start it back up, the directory comes back but the setting is back to 1024:

$ docker start dcbbaa763daf
dcbbaa763daf
$ cat /sys/fs/cgroup/cpu/docker/dcbbaa763daf/cpu.shares
1024

If you were to change these kinds of settings in a production system via the /sys fileystem directly, you’d want to tool that directly. A daemon that watches the docker events stream and changes settings at container startup, for example, is a possibility. Currently, the community has not contributed much rich tooling to this aspect. It’s likely that Docker will eventually expand the native driver’s functionality to allow this level of configuration.

Exploring Namespaces

One of the easiest to demonstrate is the UTS namespace, so let’s use docker exec to get a shell in a container and take a look. From within the docker server, run the following:

$ hostname
docker2
$ docker exec -i -t 28970c706db0 /bin/bash -l
# hostname
28970c706db0

Although docker exec will work from a remote system, here we ssh into the Docker server itself in order to demonstrate that the hostname of the server is different from inside the container.

That docker exec command line gets us an interactive process (-i) and allocates a pseudo-tty (-t), and then executes /bin/bash while executing all the normal login process in the bash shell (-l). Once we have a terminal open inside the container’s namespace, we ask for the hostname and get back the container ID. That’s the default hostname for a Docker container unless you tell Docker to name it otherwise. This is a pretty simple example, but it should clearly show that we’re not in the same namespace as the host.

Another example that’s easy to understand and demonstrate is with PID namespaces. Let’s log in to the Docker server again, take a look at the process list of one of our containers, and then get the same list from inside the container:

$ docker exec -i -t 28970c706db0 /bin/bash -l
# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 22:20 ?        00:00:00 /bin/bash
root        22     0  0 22:53 ?        00:00:00 /bin/bash -l
# exit
logout
$ ps axlf
...
46049     1  20   0 706552 18228 futex_ Ssl  ?       2:16 /usr/bin/docker -d
46135 46049  20   0  18104  1892 n_tty_ Ss+  pts/0   0:00  \_ /bin/bash

What we can see here is that from inside our container, the original command run by Docker from the CMD in our Dockerfile is /bin/bash and it has been assigned the PID 1 inside the container. You might recall that this is the PID normally used by the init process on Unix systems. In this case, the /bin/bash we started to create the container in the first place is the first PID, so it gets ID 1. But in the Docker host’s main namespace, we do a little work to find our container’s processes and we see the PID there is not 1, it’s 46135 and it’s a child of the docker daemon, which is PID 46049.

The other namespaces work in essentially the same manner and you probably get the idea by now. It’s worth pointing out here that when we were working with nsenter back in Chapter 4, we had to pass a pretty arcane (at that point) set of arguments to the command when we ran it to enter a container from the Docker server. Let’s look at that command line now:

$ sudo nsenter --target $PID --mount --uts --ipc --net --pid
root@3c4f916619a5:/#

After explaining namespaces in detail, this probably makes a lot more sense to you. You’re telling nsenter exactly which of the namespaces you want to enter. It can also be educational to use nsenter to only enter parts of the namespace of a throwaway container to see what you get. In the example above, we enter all of the namespaces we just talked about.

When it comes down to it, namespaces are the primary thing that make a container look like a container. Combine them with cgroups and you have a reasonably robust isolation between processes on the same kernel.

UID 0

The first and most overarching security risk in a container is that the root user in the container is actually the root user on the system. There are extra constraints on root in a container, and namespaces do a good job of isolating root in the container from the most dangerous parts of the /proc and /sys filesystems, for example. But generally speaking you have root access so if you can get access to resources outside of your namespace, then the kernel will see you as root. And Docker starts all services in containers as root by default which means you are then responsible for managing privilege in your applications just like you are on any Linux system. Let’s explore some of the limits on root access and look at some obvious holes. This is not intended to be an exhaustive statement on container security, but rather to give you a healthy understanding of some of the classes of security risks.

First, we’ll fire up a container and get a bash shell using the public Ubuntu image shown in the following code. Then we’ll see what kinds of access we have:

$ sudo docker run -t -i ubuntu /bin/bash
root@808a2b8426d1:/# lsmod
Module                  Size  Used by
xt_nat                 12726  2
xt_tcpudp              12603  8
veth                   13244  0
xt_addrtype            12713  2
xt_conntrack           12760  1
iptable_filter         12810  1
acpiphp                24119  0
ipt_MASQUERADE         12759  4
aufs                  191008  14
iptable_nat            12909  1
nf_conntrack_ipv4      14538  2
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
nf_nat_ipv4            13316  1 iptable_nat
nf_nat                 26158  4 ipt_MASQUERADE,nf_nat_ipv4
nf_conntrack           83996  6 ipt_MASQUERADE,nf_nat
ip_tables              27473  2 iptable_filter,iptable_nat
x_tables               29938  7 ip_tables,xt_tcpudp
bridge                101039  0
floppy                 70206  0
...

We’ve cut the output down a bit, but what we’re looking at here is a new container that we started and we’ve just asked the kernel to tell us what modules are loaded. That’s not too surprising: a normal user can do that. But it does reinforce that we’re talking to the same Linux kernel. If you run this listing on the Docker server itself, it will be identical. So we can see the kernel modules; what happens if we try to unload the floppy module?

root@808a2b8426d1:/# rmmod floppy
rmmod: ERROR: ... kmod_module_remove_module() could not remove 'floppy': ...
rmmod: ERROR: could not remove module floppy: Operation not permitted

That’s the same error message we would get if we were a nonprivileged user telling the kernel what to do. This should give you a good sense that the kernel is doing its best to prevent us from doing things we shouldn’t. And because we’re in a limited namespace, we also can’t get the kernel to give us access to the top-level namespace either. We’re really relying on there being no bugs in the kernel that allow us to escalate that, however, because if we do, we’re root and can change things.

We can contrive a simple example of how things can go wrong by starting a bash shell in a container that has had the Docker server’s /etc bind mounted into the container’s namespace. Keep in mind that anyone who can start a container on your Docker server can do what we’re about to do any time they like because you can’t configure Docker to prevent it:

$ docker run -i -t -v /etc:/host_etc ubuntu /bin/bash
root@e674eb96bb74:/# more /host_etc/shadow
root:!:16230:0:99999:7:::
daemon:*:16230:0:99999:7:::
bin:*:16230:0:99999:7:::
sys:*:16230:0:99999:7:::
...
irc:*:16230:0:99999:7:::
nobody:*:16230:0:99999:7:::
libuuid:!:16230:0:99999:7:::
syslog:*:16230:0:99999:7:::
messagebus:*:16230:0:99999:7:::
kmatthias:$1$aTAYQT.j$3xamPL3dHGow4ITBdRh1:16230:0:99999:7:::
sshd:*:16230:0:99999:7:::
lxc-dnsmasq:!:16458:0:99999:7:::

Here we’ve used the -v switch to Docker to tell it to mount a host path into the container. The one we’ve chosen is /etc, which is a dangerous thing to do. But it serves to prove a point: we are root in the container and root has file permissions in this path. So we can look at the real /etc/shadow file any time we like. There are plenty of other things you could do here, but the point is that by default you’re only partly constrained.

Privileged containers

There are times when you need your container to have special kernel capabilities that would normally be denied to the container. This could include many things like mounting a USB drive, modifying the network configuration, or creating a new Unix device.

In the following code, we try to change the MAC address of our container:

$ docker run --rm -ti ubuntu /bin/bash
root@b328e3449da8:/# ip link ls
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state ...
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state ...
    link/ether 02:42:0a:00:00:04 brd ff:ff:ff:ff:ff:ff
root@b328e3449da8:/# ip link set eth0 address 02:0a:03:0b:04:0c
RTNETLINK answers: Operation not permitted
root@b328e3449da8:/# exit

As we can see, it doesn’t work. This is because the underlying Linux kernel blocks the nonprivileged container from doing this, which is exactly what we normally want. However, assuming that we need this functionality for our container to work as intended, the easiest way to significantly expand a container’s privileges is by launching it with the --privileged=true argument:

$ docker run -ti --rm --privileged=true ubuntu /bin/bash
root@88d9d17dc13c:/# ip link ls
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state ...
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state ...
    link/ether 02:42:0a:00:00:04 brd ff:ff:ff:ff:ff:ff
root@88d9d17dc13c:/# ip link set eth0 address 02:0a:03:0b:04:0c
root@88d9d17dc13c:/# ip link ls
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state ...
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state ...
    link/ether 02:0a:03:0b:04:0c brd ff:ff:ff:ff:ff:ff
root@88d9d17dc13c:/# exit

In the preceding output, you will notice that we no longer get the error and the link/ether entry for eth0 has been changed.

The problem with using the --privileged=true argument is that you are giving your container very broad privileges, and in most cases you likely only need one or two kernel capabilities to get the job done.

If we explore our privileged container some more, we will discover that we have capabilities that have nothing to do with changing the MAC address. I can even do things that could cause issue with both Docker and the host system. In the following code, we are going to create a memory swapfile¹ and enable it:

$ docker run -ti --rm --privileged=true ubuntu /bin/bash
root@0ffcdd8f7535:/# dd if=/dev/zero of=/swapfile1 bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.00046004 s, 223 MB/s
root@0ffcdd8f7535:/# mkswap /swapfile1
Setting up swapspace version 1, size = 96 KiB
no label, UUID=fc3d6118-83df-436e-867f-87e9fbce7692
root@0ffcdd8f7535:/# swapon /swapfile1
root@0ffcdd8f7535:/# swapoff /swapfile1
root@0ffcdd8f7535:/# exit
exit

FATAL [0049] Error response from daemon:
Cannot destroy container 0ff...670:
 Driver overlay failed to remove root filesystem 0ff...670:
 remove /var/lib/docker/overlay/0ff...670/upper/swapfile1:
 operation not permitted

$ sudo swapoff /var/lib/docker/overlay/0ff...670/upper/swapfile1

So as we’ve seen, it is possible for people to do bad things in a fully privileged container.

To change the MAC address, the only kernel capability we actually need is CAP_NET_ADMIN. Instead of giving our container the full set of privileges, we can give it this one privilege by launching our Docker container with the --cap-add argument, as shown here:

$ docker run -ti --rm --cap-add=NET_ADMIN ubuntu /bin/bash
root@852d18f5c38d:/# ip link set eth0 address 02:0a:03:0b:04:0c
root@852d18f5c38d:/# ip link ls
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state ...
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state ...
    link/ether 02:0a:03:0b:04:0c brd ff:ff:ff:ff:ff:ff
root@852d18f5c38d:/# exit

You should also notice that although we can change the MAC address, we can no longer use the swapon command inside our container.

$ docker run -ti --rm --cap-add=NET_ADMIN ubuntu /bin/bash
root@848aa7924594:/# dd if=/dev/zero of=/swapfile1 bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.000575541 s, 178 MB/s
root@848aa7924594:/# mkswap /swapfile1
Setting up swapspace version 1, size = 96 KiB
no label, UUID=3b365d90-8116-46ad-80c5-24341299dc41
root@848aa7924594:/# swapon /swapfile1
swapon: /swapfile1: swapon failed: Operation not permitted
root@848aa7924594:/# exit

By using both the --cap-add and --cap-drop arguments to docker run, you can finely control the Linux kernel capabilities that your container has.

SElinux, AppArmor

Earlier we talked about how containers are a combination of two or three things: cgroups, namespaces, and SELinux or AppArmor. We’re going to talk about the latter two systems now. They allow you to apply security controls that extend beyond those normally supported by Unix systems. SELinux originally came out of the US National Security Agency and supports very fine-grained control. AppArmor is an effort to achieve many of the same goals without the level of complication involved in SELinux. It actually predates SELinux, having first appeared in 1998 in the Immunix Linux distribution. Novell, SuSE, and Canonical have been some of its recent champions.

Docker ships by default with reasonable profiles enabled on platforms that support them. This means that on Ubuntu systems, AppArmor is enabled and configured, and on CentOS/RHEL/Fedora systems, SELinux is. You can further configure these profiles to prevent things like what we’ve just done in Chapter 9, and if you’re running Docker in production, you should do a risk analysis and determine if this is something you should look at. Here’s a quick outline of the benefits we’re getting from these systems.

They provide what is known as Mandatory Access Control. This is a class of security system where a system-wide security policy grants users (or “initiators”) access to a resource (or “target”). What this allows you to do is to prevent anyone, including root, from accessing a part of the system that they should not have access to. You can apply the policy to a whole container so that all processes are constrained. Many chapters would be required to give a very clear overview of how to configure these systems in detail. The default profiles are doing things like blocking access to parts of the /proc and /sys filesystems that would be dangerous to expose in the container, even though they show in the container’s namespace. They also provide more narrowly scoped mount access to prevent containers from getting ahold of mount points they should not see.

If you are considering using Docker containers in production, you should make certain that the systems you are running have AppArmor or SELinux enabled and running. For the most part, both systems are reasonably equivalent. But in the Docker context, one notable limitation of SELinux is that it only works fully on systems that support filesystem metadata, which means that it won’t work for you on BTRFS-backed Docker daemons, for example. Only the devicemapper backend currently fully supports SELinux. Unfortunately, that backend is also not currently very stable for production. The OverlayFS backend is going to support this shortly. AppArmor, on the other hand, does not use filesystem metadata and so works on all of the Docker backends. Which one you use is going to be somewhat distribution-centric, so you may be forced to choose a filesystem backend based on which distribution you run.