Container-building is an art form in and of itself. Ultimately, the goal of a container is to create an ultra-slim, single-task process .1 However, while containers look a lot like virtual machines, we have to take a different approach and mindset to building containers than we normally apply to building virtual and physical machines.
Containers are fundamentally different than virtual machines and should be treated as such.
Containers should be limited to a single, unified task.
Base images should be chosen with care.
Container images should include as little of the operating system as is possible.
All dependencies should be explicitly named in your Dockerfile.
Care needs to be taken to avoid image bloating.
Containers should be made to be easily configurable to the environment.
4.1 How Not to Build a Container
The best way to describe good container-building methodologies is to start by describing bad ones. The biggest (and probably most common) bad methodology for building containers is to treat a container as a slimmed-down version of a virtual machine (VM). This is a natural way of thinking because containers are indeed the successors of VMs in the data center. However, importing VM thinking into building a container will reduce your effectiveness considerably.
VMs are the successors of physical machines , and, in fact, they work essentially just like physical machines except for the ability to maintain hardware and software independently. Therefore, most of what worked when thinking about servers running on physical hardware transferred directly to thinking about servers running on VMs. It’s tempting to continue to bring that information forward to the next step, but, at heart, it’s a bad idea.
4.1.1 Don’t Make a Container Perform Multiple Tasks
Essentially, when managing servers, system administrators or developers typically “organized” the server—adding a caching server, adding a proxy service, running file synchronization jobs, running scheduled tasks, etc. However, in cloud native architectures, this organization is done at a higher level—at the cluster level. As we will be discussing in Part 2, the Kubernetes system will essentially be acting as your cluster’s operating system. You will organize Pods, Deployments, Services, and other Kubernetes workloads, which are all made up of containers. However, in order to enable this sort of orchestration to be maximally flexible and take advantage of all that Kubernetes has to offer, each container really needs to do only a single “thing.”
Now, I recognize that “thing” is inherently a fuzzy concept. However, the way that I think about it is this—if you would naturally manage it separately, it’s a separate thing. For instance, scheduled jobs are usually managed separately from the main service. Therefore, scheduled jobs are a separate “thing.” Caching servers have their own configuration, their own command-line options, run on their own port, and run in a separate process than your main task. It is a separate “thing.”
For larger systems such as the PostgreSQL database , even though it technically runs several very different processes under its main task (an autovacuumer, a stats collector, a log writer, etc.), it all follows from a single configuration file, all run under a single start/stop command that manages the whole thing. Therefore, since you are only managing one thing (the PostgreSQL configuration as a whole), you can consider larger systems like this a single “thing.”
4.1.2 Don’t Include an Entire Operating System
The second thing that is tempting is to build your container images with lots of additional tools. Some people take a base Linux distribution , add every package they can imagine, and then build all of their containers on top of that.
The more things you include in your container image, the more things an attacker can take advantage of. This is known as the “attack surface ” of your container image. The more things you install just give attackers one more potentially buggy tool that can be exploited.2
Bigger containers hide dependencies. One of the goals of containerization is to be more explicit about our operating system dependencies . If we throw the whole operating system into our container, then we may miss the fact that our container depends on having ImageMagick installed, or some other dependency. Containers usually come with only the bare minimum, and you should add each dependency explicitly so that you have documented what is needed to run your application. The “bare minimum” usually includes a shell, a few standard system utilities, package management tools, and the most common system libraries and configuration files.
Bigger containers tempt developers to add hidden dependencies. If it is already on the container, why not make use of it, right? Developers are usually quick to use anything and everything that is available to them. Making slim containers forces developers to make choices and explicitly log them in the Dockerfile. This means that all developers are more likely to use the same tools for similar tasks, because they will recognize which specific tools are already present (since they are there explicitly).
Bigger containers require more maintenance. One thing I’ve learned in life (both in computing and in my personal life) is that everything requires maintenance at some point. If you have a bigger house, you have more rooms to clean. Another bathroom also means more pipes that can break. The same is true of computers. Every package you add will have to be monitored and updated due to security vulnerabilities or will eventually stop working with something else you have installed. Overall, it’s best to go for the minimalist approach.
Bigger containers are, well, bigger. The nice thing about containers is that, even with a bigger container, running multiple of the same image doesn’t add additional disk space. However, it does use the space once, and that can be wasteful. This is especially true if you have different applications which don’t share much of the same base image running on the same host. Additionally, it takes time and network traffic to transfer images as well as disk space in the container registry to store them.
Bigger containers make it hard to switch base images. If you have a lot of hidden dependencies, switching Linux distributions can be quite a pain. The tool doesn’t work exactly the same, or the paths used are different. Having slim images with explicit dependencies means that you know everything that is in your image, so you also know what you will have to test or possibly modify if you switch Linux distributions.
A good, achievable goal for most containers is to keep them under 100MB. Most of my containers are significantly smaller.
Now that we know how not to build containers, let’s look at some good container practices we should be adopting.
4.2 Base Images
For most development platforms , there is a base image available that only contains (a) the minimal operating system install and (b) the minimum install of the language and platform needed to pull in other dependencies. For instance, the standard ruby image on Docker Hub contains the base operating system, Ruby, RubyGems (the package system), and Bundler (the dependency manager). Your container build script can then use this standard image to pull in additional dependencies, both from the operating system and from the platform.
I highly recommend using the standard base images wherever possible. Using standard images allows for better maintenance, because the actual platform maintenance will be handled by the platform itself. If you need to update the version of your platform, it usually just means updating the base image version at the top of your Dockerfile and moving on. The container images for the standard platforms are generally very high quality, and I highly recommend them.
Additionally, you should think about the version tags you are using for your base image. Using latest (or not specifying one at all) means that every time you rebuild your image, you will get an up-to-date version. The problem, however, is that sometimes your project may not run correctly on the latest upgrade. On the other hand, if you use a tag that is specific to the latest point release, you might find that this base image gets removed if a security bug is found.
You should basically balance two things—how specifically you are relying on version-specific features and how problematic a broken build process is. Most of the time I try to target only major-level releases. This gives me pretty good flexibility for upgrading easily just by rebuilding containers, but doesn’t run the risk of breaking the build when the project does a major overhaul.
Also remember that you don’t actually know exactly what is on someone else’s container image. Therefore, I would caution against using images except from known good sources. Just because some user has an image that they claim works well for a task does not guarantee that the image doesn’t also contain malware. I tend to stick to official Docker Hub images, Bitnami images, and ones from the official organizations that run the projects.
4.3 Alpine Distributions
The Alpine Linux distribution started life as the Linux Router Project (LRP)—a project whose goal was to get Linux to run on tiny Wi-Fi routers in order to pack in additional features. The LRP enabled you to take a cheap Wi-Fi router and turn it into a VPN server or any of a number of complicated network tools. They created a specialized Linux distribution because the devices they were trying to fit Linux onto had so little memory and disk space.
While the Linux Router Project no longer exists, the distribution they built eventually became the Alpine Linux distribution—an ultrasmall Linux distribution which works as a perfect base for containers. The base image clocks in at only five megabytes (megabytes, not gigabytes)! Yes, that’s the entire OS, including shell and all.
To fit in such a space, Alpine Linux has several differences from typical Linux distributions, however. First of all, most of the applications in Alpine are actually a single application called busybox. The actual application names, sh, vi, mv, rm, etc., are all symlinked to this one application. The application uses the name that it is called by in order to know what command to execute. That allows Alpine to fit its entire command structure in a single megabyte application. Second, rather than using the enormous GNU C Library, Alpine uses a smaller, stripped-down C library, known as musl.
Note that, on the whole, you won’t notice too many differences between Alpine Linux and any other distribution, but occasionally you may run into slight differences dealing from either differences in the commands themselves or small differences in the C Library, though these are usually very rare and technical differences.3
Most major container images are available as both a regular image (usually based off of Debian) and an Alpine version . The Alpine versions usually have -alpine in their image tag names. So, for instance, for a basic webserver, there is an official nginx image. If you want to run the Debian-based image for version 1.21 of nginx, you would use the image nginx:1.21, but for the Alpine build you would use nginx:1.21-alpine. The Debian-based one is about 56 megabytes, while the Alpine one is only 10!
Ultimately, the differences are not huge (remember, the container image is only stored once on any given host no matter how many containers are running it). However, the larger GNU C Library that comes standard in non-Alpine distributions uses more memory, which can cost money when running containers in the cloud.
4.4 Avoid Bloated Images from Deleted Files
The biggest surprise when building containers is that the RUN command builds a single layer which cannot be removed. That doesn’t mean the files can’t be removed—the next layer can remove them easily. However, the “deleted” files will still stay in your container image—forever.
A snippet of a Docker file illustrates the multistage build process. It includes the installation of the g c c.
Multistage Build: Dockerfile
The problem here is that each RUN layer gets a separate layer. That means that we will install gcc in one layer and use it in the next to generate myapplication, and then the final layer will remove gcc. However, even though the containers running our image will not be able to see gcc (because it got removed in higher layers), it still exists in the lower layers! This means it has to be stored in the registry, transferred over the Internet when pulled, and stored on the final host machine, even though it is never used! I’ve seen images bloat more than an extra gigabyte through such processes.
There are three primary ways to avoid this tragic fate. The first is to put everything on the same RUN command . That’s doable, but it makes the Dockerfile really messy. The second is to create a script in your build that contains (a) the setup installations, (b) the build, and (c) the teardown deinstallations all in one script. Either of these two methods will ensure that, by the end of the RUN command, only the files we wanted to add are in the layer.
The final mechanism is to use a separate container image for the build and then copy the specific, resulting files from the build into your main container image. This is known in the container world as a multistage build. Figure 4-1 shows an example Dockerfile implementing this. There, the build image is named builder, and we can then use the --from flag of the COPY command to load files from this build.
The different builds don’t need to use the same container base, you can have as many stages you want, and only the last image gets saved and tagged.
4.5 Make Your Containers Configurable
The best containers are highly configurable through environment variables. Environment variables are the best way to pass information to a container from the outside. They are supported on every application platform and have thorough support in Kubernetes.
Anything that you might consider configuring should be exposed (and documented) as an environment variable. The location of every other service that your application needs to talk to should be configurable as an environment variable.
The end goal is to be able to take the exact same container image and use it locally in development, in the staging cluster, and in the production cluster, without any changes to the container image at all. This way, not only is production the same code as staging, but it is literally the same server image. Everything that is different between the two should be encoded in those environment variables. This makes mistakes not only rare but also easy to spot and diagnose.
Many containers for standard services come with shell scripts that read the environment variables and then do a number of pre-startup tasks (such as writing configuration files) and then hand off the actual operation of the container to the main service. The official postgres image is an example of an ideal system. You can set the data directory, the username and password, the authentication mechanism, and more through environment variables. On startup, the entrypoint is a shell script which puts together the configuration for the server and then runs an exec to start up the server itself.
The exec shell command causes the current shell process to be replaced by the given command. When a shell just invokes a command, both the shell and the command are taking up memory, while the shell waits for the command to complete. By finishing the setup with an exec, however, the shell process is physically replaced by the executed command, thus freeing up some additional memory. This is a very common startup pattern for containers.
For example, let us say that you are building an application that connects to a databases, a caching server, and a payment system. At minimum, you would want the hostnames and credentials for each of these services to be configurable via environment variables or similar mechanisms. If they needed to be written to a configuration file, then your startup script can take the environment variables, create the configuration file, and then start the application.
4.6 Be Clear About Your Statefulness
Managing state is an important task within a cluster. It is mostly important to be clear which containers are maintaining state, how they are maintaining it, and what are the effects of restarting those containers. One of the goals of a containerized infrastructure is to have no (or at least very few) containers which you actually care about. You care about the system, and, ideally, any particular container should be replaceable.
The easiest to manage containers are those that don’t require any state. They don’t need special disks allocated to them. They don’t need any special setup or teardown. They just process, process, and process until we tell them to stop.
So, ideally, you want to not have state in your application. However, the fact is that the reason why someone uses your app at all is probably because it is holding some amount of state (data, files, etc.) that they care about. However, we should attempt to isolate the stateful aspects of our application as much as possible. In other words, we need to be clear about the statefulness of each container.
Can the whole thing be shut down and replaced without any loss of function? (ideal)
Does it need backing hard drive storage for its state?
Does the backing hard drive storage need to keep on living after the container dies?
Is there a good way for a new container to pick up where the previous container left off?
What sort of coordination is required by these stateful activities?
Is my application resilient enough to reconnect automatically if the stateful aspect goes offline, or will the whole cluster need to be individually restarted?
What are the ordering considerations for any stateful components? What needs to be booted first? Second?
How do you know when the stateful service is fully up and ready?
These are the types of things that a cluster administrator needs to keep in mind when thinking about stateful components. If a component is stateless, you don’t have to think about these things—which is why statelessness is preferred as much as possible.
4.7 Final Tips
You should always be sure to run tests on a container running the image. Remember, containers are there because we are packaging both the operating system and the application as a single package. Therefore, we need to test to be sure that they run well together. Running your tests in a built container means that, if an upstream modification to a container image breaks your build, you are likely to catch it before sending it to production.
Don’t forget that, with the advent of Apple Silicon, a lot of developers aren’t running the same CPU architecture anymore! Docker has the ability to create multi-architecture builds, or, more straightforwardly, you can specify --platform=linux/amd64 in the FROM line of your Dockerfile, which will generate a 64-bit x86-compatible Linux image from either Mac or PC.4
If, for some reason, you have a process that spawns children who don’t get taken care of (they become “zombies”), you may need an init process to help. Docker containers don’t have a standard init process to clean up zombie processes. However, there are packages such as dumb-init and tini that can help you out here. Normally, the presence of zombie processes is due to bad code or bad design, so always consider redesigning before switching to dumb-init or tini as a workaround.
Scan your images. Most registries have the ability to regularly scan your images for vulnerabilities. Be sure to take advantage of this easy security boost.
Get to know your distribution, and especially the package management tools. Most of your interaction with your distribution will be through its packaging tools, so be sure to get to know them well.
Don’t get too tricky. The Kubernetes infrastructure works best when everything works in an expected manner. The less “weird” your container image behaves, the more likely it will integrate into your cluster without causing headaches.
4.8 Summary
While containers are the next big advancement after virtual machines, there are a lot of differences between how containers and traditional servers are managed.
Container images should be oriented to one, single task.
Container images should be minimal, containing only the dependencies needed to run the application.
Choose the base image for your container wisely.
It is important to keep base images up to date with the latest security fixes, but you don’t want new releases to break your application.
Application tests should be run on a fully built container so that you test the complete operating system and application package.
Containers should be made to be largely configurable through environment variables.
Container images often contain an initial script which uses environment variables to create or modify configuration files and then runs exec to start the real service.