The first stage in our continuous deployment pipeline will contain quite a few steps. We'll need to check out the code, to run unit tests and any other static analysis, to build a Docker image, and to push it to the registry. If we define CI as a set of automated steps followed with manual operations and validations, we can say that the steps we are about to execute can be qualified as CI.

The only thing we truly need to make all those steps work is Docker client with the access to Docker server. One of the containers of the cd Pod already contains it. If you take another look at the definition, you'll see that we are mounting Docker socket so that the Docker client inside the container can issue commands to Docker server running on the host. Otherwise, we would be running Docker-in-Docker, and that is not a very good idea.

Now we can enter the docker container and check whether Docker client can indeed communicate with the server.

 1  kubectl -n go-demo-3-build 
 2      exec -it cd -c docker -- sh
 3
 4  docker container ls

Once inside the docker container, we executed docker container ls only as a proof that we are using a client inside the container which, in turn, uses Docker server running on the node. The output is the list of the containers running on top of one of our servers.

Let's get moving and execute the first step. We cannot do much without the code of our application, so the first step is to clone the repository.

Make sure that you replace [...] with your GitHub username in the command that follows.

 1  export GH_USER=[...]
 2    
 3  git clone 
 4      https://github.com/$GH_USER/go-demo-3.git 
 5      .

We cloned the repository into the workspace directory. That is the same folder we mounted as an emptyDir volume and is, therefore, available in all the containers of the cd Pod. Since that folder is set as workingDir of the container, we did not need to cd into it.

Please note that we cloned the whole repository and, as a result, we are having a local copy of the HEAD commit of the master branch. If this were a "real" pipeline, such a strategy would be unacceptable. Instead, we should have checked out a specific branch and a commit that initiated the process. However, we'll ignore those details for now, and assume that we'll solve them when we move the pipeline steps into Jenkins and other tools.

Next, we'll build an image and push it to Docker Hub. To do that, we'll need to login first.

Make sure that you replace [...] with your Docker Hub username in the command that follows.

 1  export DH_USER=[...]
 2
 3  docker login -u $DH_USER

Once you enter your password, you should see the Login Succeeded message.

We are about to execute the most critical step of this stage. We'll build an image.

At this moment you might be freaking out. You might be thinking that I went insane. A Pastafarian and a firm believer that nothing should be built without running tests first just told you to build an image as the first step after cloning the code. Sacrilege!

However, this Dockerfile is special, so let's take a look at it.

 1  cat Dockerfile

The output is as follows.

FROM golang:1.9 AS build
ADD . /src
WORKDIR /src
RUN go get -d -v -t
RUN go test --cover -v ./... --run UnitTest
RUN go build -v -o go-demo


FROM alpine:3.4
MAINTAINER  Viktor Farcic <[email protected]>
RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
EXPOSE 8080
ENV DB db
CMD ["go-demo"]
COPY --from=build /src/go-demo /usr/local/bin/go-demo
RUN chmod +x /usr/local/bin/go-demo

Normally, we'd run a container, in this case, based on the golang image, execute a few processes, store the binary into a directory that was mounted as a volume, exit the container, and build a new image using the binary created earlier. While that would work fairly well, multi-stage builds allow us to streamline the processes into a single docker image build command.

If you're not following Docker releases closely, you might be wondering what a multi-stage build is. It is a feature introduced in Docker 17.05 that allows us to specify multiple FROM statements in a Dockerfile. Each FROM instruction can use a different base, and each starts a new stage of the build process. Only the image created with the last FROM segment is kept. As a result, we can specify all the steps we need to execute before building the image without increasing its size.

In our example, we need to execute a few Go commands that will download all the dependencies, run unit tests, and compile a binary. Therefore, we specified golang as the base image followed with the RUN instruction that does all the heavy lifting. Please note that the first FROM statement is named build. We'll see why that matters soon.

Further down, we start over with a new FROM section that uses alpine. It is a very minimalist Linux distribution (a few MB in size) that guarantees that our final image is minimal and is not cluttered with unnecessary tools that are typically used in "traditional" Linux distros like ubuntu, Debian, and CentOS. Further down we are creating everything our application needs, like the DB environment variable used by the code to know where the database is, the command that should be executed when a container starts, and so on. The critical part is the COPY statement. It copies the binary we created in the build stage into the final image.

Let's build the image.

 1  docker image build 
 2      -t $DH_USER/go-demo-3:1.0-beta 
 3      .

We can see from the output that the steps of our multi-stage build were executed. We downloaded the dependencies, run unit tests, and built the go-demo binary. All those things were temporary, and we do not need them in the final image. There's no need to have a Go compiler, nor to keep the code. Therefore, once the first stage was finished, we can see the message "Removing intermediate container." Everything was discarded. We started over, and we built the production-ready image with the binary generated in the previous stage.

We have the whole continuous integration process reduced to a single command. Developers can run it on their laptops, and CI/CD tools can use it as part of their extended processes. Isn't that neat?

Let's take a quick look at the images on the node.

 1  docker image ls

The output, limited to the relevant parts, is as follows.

REPOSITORY        TAG      IMAGE ID CREATED            SIZE
vfarcic/go-demo-3 1.0-beta ...      54 seconds ago     25.8MB
<none>            <none>   ...      About a minute ago 779MB
...

The first two images are the result of our build. The final image (vfarcic/go-demo-3) is only 25 MB. It's that small because Docker discarded all but the last stage. If you'd like to know how big your image would be if everything was built in a single stage, please combine the size of the vfarcic/go-demo-3 image with the size of the temporary image used in the first stage (it's just below vfarcic/go-demo-3 1.0-beta).

The only thing missing is to push the image to the registry (for example, Docker Hub).

 1  docker image push 
 2      $DH_USER/go-demo-3:1.0-beta

The image is in the registry and ready for further deployments and testing. Mission accomplished. We're doing continuous integration manually. If we'd place those few commands into a CI/CD tool, we would have the first part of the process up and running.

We are still facing a few problems. Docker running in a Kubernetes cluster might be too old. It might not support all the features we need. As an example, most of the Kubernetes distributions before 1.10 supported Docker versions older than 17.05. If that's not enough, consider the possibility that you might not even use Docker in a Kubernetes cluster. It is very likely that ContainerD will be the preferable container engine in the future, and that is only one of many choices we can select. The point is that container engine in a Kubernetes cluster should be in charge of running container, and not much more. There should be no need for the nodes in a Kubernetes cluster to be able to build images.

Another issue is security. If we allow containers to mount Docker socket, we are effectively allowing them to control all the containers running on that node. That by itself makes security departments freak out, and for a very good reason. Also, don't forget that we logged into the registry. Anyone on that node could push images to the same registry without the need for credentials.

Even if we do log out, there was still a period when everyone could exploit the fact that Docker server is authenticated and authorized to push images.

Truth be told, we are not preventing anyone from mounting a Docker socket. At the moment, our policy is based on trust. That should change with PodSecurityPolicy. However, security is not the focus of this book, so I'll assume that you'll set up the policies yourself, if you deem them worthy of your time.

If that's not enough, there's also the issue of preventing Kubernetes to do its job. The moment we adopt container schedulers, we accept that they are in charge of scheduling all the processes running inside the cluster. If we start doing things behind their backs, we might end up messing with their scheduling capabilities. Everything we do without going through Kube API is unknown to Kubernetes.

We could use Docker inside Docker. That would allow us to build images inside containers without reaching out to Docker socket on the nodes. However, that requires privileged access which poses as much of a security risk as mounting a Docker socket. Actually, it is even riskier. So, we need to discard that option as well.

Another solution might be to use kaniko (https://github.com/GoogleContainerTools/kaniko). It allows us to build Docker images from inside Pods. The process is done without Docker so there is no dependency on Docker socket nor there is a need to run containers in privileged mode. However, at the time of this writing (May 2018) kaniko is still not ready. It is complicated to use, and it does not support everything Docker does (for example, multi-stage builds), it's not easy to decipher its logs (especially errors), and so on. The project will likely have a bright future, but it is still not ready for prime time.

Taking all this into consideration, the only viable option we have, for now, is to build our Docker images outside our cluster. The steps we should execute are the same as those we already run. The only thing missing is to figure out how to create a build server and hook it up to our CI/CD tool. We'll revisit this subject later on.

For now, we'll exit the container.

 1  exit

Let's move onto the next stage of our pipeline.