Now that we got a grip on ServiceAccounts, it should be relatively straightforward to correct the problem we experienced with Jenkins. As a reminder, we could not configure the Kubernetes plugin. We experienced the same forbidden message as when we tried to use kubectl container with the default ServiceAccount. Now that we know that ServiceAccounts provide permissions to processes running inside containers, all we have to do is to define one for Jenkins.

We'll spice it up a bit with a slightly more complicated use-case. We'll try to run Jenkins master in one Namespace and perform builds in another. That way we can have a clear separation between Jenkins and "random" stuff our builds might be doing. Through such separation, we can guarantee that Jenkins will (probably) not be affected if we do something wrong in our builds.

Let's take a look at yet another Jenkins definition.

 1  cat sa/jenkins.yml

The relevant parts of the output are as follows.

...
apiVersion: v1
kind: Namespace
metadata:
  name: build

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: jenkins

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
  namespace: build
rules:
- apiGroups: [""]
  resources: ["pods", "pods/exec", "pods/log"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
  namespace: build
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: jenkins
...
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: jenkins
  namespace: jenkins
spec:
  ...
  template:
    ...
    spec:
      serviceAccountName: jenkins
      ...

This time we are creating a second Namespace called build and a ServiceAccount jenkins in the jenkins namespace. Further on, we created a Role and a RoleBinding that provides required permissions in the build Namespace. As such, we bound the Role with the ServiceAccount in the jenkins Namespace. As a result, Jenkins should be able to create Pods in build, but it won't be able to do anything in its own Namespace jenkins. That way we can be relatively safe that a problem in our builds will not affect Jenkins. If we specified ResourceQuotas and LimitRanges for both Namespaces, our solution would be even more bulletproof. But, since I assume that you know how to do that, I excluded them in an attempt to simplify the definition.

Let's apply the config and observe the result.

 1  kubectl apply 
 2      -f sa/jenkins.yml 
 3      --record

We can see from the output that the resources were created. All that's left is to wait until Jenkins rolls out.

 1  kubectl -n jenkins 
 2      rollout status sts jenkins

Now that Jenkins is up-and-running, we should open it in a browser and repeat the same setup steps we did before.

 1  open "http://$CLUSTER_DNS/jenkins"

The first step is to get the initial admin password.

 1  kubectl -n jenkins 
 2      exec jenkins-0 -it -- 
 3      cat /var/jenkins_home/secrets/initialAdminPassword

Please copy the output and paste it into the Administrator password field. Click Continue, followed by the Install suggested plugins button. The rest of the setup requires you to Create First Admin User, so please go ahead. You don't need my help on that one.

Just as before, we'll need to add Kubernetes and BlueOcean plugins.

 1  open "http://$CLUSTER_DNS/jenkins/pluginManager/available"

You already know what to do. Once you're done installing the two plugins, we'll go to the configuration screen.

 1  open "http://$CLUSTER_DNS/jenkins/configure"

Please expand the Add a new cloud drop-down list in the Cloud section and select Kubernetes.

Now that we have the ServiceAccount that grants us the required permissions, we can click the Test Connection button and confirm that it works.

The output is as follows.

Error testing connection : Failure executing: GET at: 
 https://kubernetes.default.svc/api/v1/namespaces/jenkins/pods.
    Message: Forbidden!Configured service account doesn't have access.
    Service account may have been revoked. pods is forbidden: User 
 "system:serviceaccount:jenkins:jenkins" cannot list pods in the 
 namespace "jenkins".

Did we do something wrong? We didn't. That was the desired behavior. By not specifying a Namespace, Jenkins checked whether it has necessary permission in the Namespace where it runs. If we try to invoke Kube API from a container, it'll always use the same Namespace as the one where the container is. Jenkins is no exception. On the other hand, our YAML explicitly defined that we should have permissions to create Pods in the build Namespace. Let's fix that.

Please type build in the Kubernetes Namespace field and click the Test Connection button again.

This time the output shows that the connection test was successful. We managed to configure Jenkins' Kubernetes plugin to operate inside the build Namespace. Still, there is one more thing missing.

When we create a job that uses Kubernetes Pods, an additional container will be added. That container will use Java Network Launch Protocol (JNLP) to establish communication with the Jenkins master. We need to specify a valid address JNLP can use to connect to the master. Since the Pods will be in the build Namespace and the master is in jenkins, we need to use the longer DNS name that specifies both the name of the service (jenkins) as well as the Namespace (also jenkins). On top of all that, our master is configured to respond to requests with the root path /jenkins. All the all, the full address Pods can use to communicate with Jenkins master is should be http://[SERVICE_NAME].[NAMESPACE]/[PATH]. Since all three of those elements are jenkins, the "real" address is http://jenkins.jenkins/jenkins. Please type it inside the Jenkins URL field and click the Save button.

Now we're ready to create a job that'll test that everything works as expected.

Please click the New Item link from the left-hand menu to open a screen for creating jobs. Type my-k8s-job in the item name field, select Pipeline as the type, and click the OK button. Once inside the job configuration screen, click the Pipeline tab and write the script that follows inside the Pipeline Script field.

 1  podTemplate( 
 2      label: 'kubernetes', 
 3      containers: [ 
 4          containerTemplate(name: 'maven', image: 'maven:alpine',
    ttyEnabled: true, command: 'cat'),
 5          containerTemplate(name: 'golang', image: 'golang:alpine',
    ttyEnabled: true, command: 'cat') 
 6      ] 
 7  ) { 
 8      node('kubernetes') { 
 9          container('maven') { 
10              stage('build') { 
11                  sh 'mvn --version' 
12              } 
13              stage('unit-test') { 
14                  sh 'java -version' 
15              } 
16          } 
17          container('golang') { 
18              stage('deploy') { 
19                  sh 'go version' 
20              } 
21          } 
22      } 
23  }

The job is relatively simple. It uses podTemplate to define a node that will contain two containers. One of those is golang, and the other is maven. In both cases the command is cat. Without a long-running command (process 1), the container would exit immediately, Kubernetes would detect that and start another container based on the same image. It would fail again, and the loop would continue. Without the main process running, we'd enter into a never-ending loop.

Further on, we are defining that we want to use the podTemplate as a node and we start executing sh commands in different containers. Those commands only output software versions. The goal of this job is not to demonstrate a full CD pipeline (we'll do that later), but only to prove that integration with Kubernetes works and that we can use different containers that contain the tools we need.

Don't forget to click the Save button.

Now that we have a job, we should run it and validate that the integration with Kubernetes indeed works.

Please click the Open Blue Ocean link from the left-hand menu followed by the Run button.

We'll let Jenkins run the build and switch to Shell to observe what's happening.

 1  kubectl -n build get pods

After a while, the output should be as follows.

NAME              READY STATUS            RESTARTS AGE
jenkins-slave-... 0/3   ContainerCreating 0        11s

We can see that Jenkins created a Pod with three containers. At this moment in time, those containers are still not fully functional. Kubernetes is probably pulling them to the assigned node.

You might be wondering why are there three containers even though we specified two. Jenkins added the third to the Pod definition. It contains JNLP that is in charge of communication between Pods acting as nodes and Jenkins masters. From user's perspective, JNLP is non-existent. It is a transparent process we do not need to worry about.

Let's take another look at the Pods in the build Namespace.

 1  kubectl -n build get pods

The output is as follows.

NAME              READY STATUS  RESTARTS AGE
jenkins-slave-... 3/3   Running 0        5s

This time, if you are a fast reader, all the containers that form the Pod are running, and Jenkins is using them to execute the instructions we defined in the job.

Let's take another look at the Pods.

 1  kubectl -n build get pods

The output is as follows.

NAME              READY STATUS      RESTARTS AGE
jenkins-slave-... 3/3   Terminating 0        32s

Once Jenkins finished executing the instructions from the job, it issued a command to Kube API to terminate the Pod.

Jenkins nodes created through podTemplate are called on-shot agents. Instead of having long-running nodes, they are created when needed and destroyed when not in use. Since they are Pods, Kubernetes is scheduling them on the nodes that have enough resources. By combining one-shot agents with Kubernetes, we are distributing load and, at the same time, using only the resources we need. After all, there's no need to waste CPU and memory on non-existing processes.

We're done with Jenkins, so let's remove the Namespaces we created before we move into the next use-case.

 1  kubectl delete ns jenkins build