Red Hat’s CodeReady Containers

Red Hat (https://developers.redhat.com/products/codeready-containers/overview) provides a minimal, preconfigured OpenShift version 4 cluster that you can run on your laptop or desktop computer. The CodeReady OpenShift environment is intended to be used for development and testing purposes. CodeReady containers provide a fully functional cloud development environment on your local machine and contain all the tooling necessary for you to develop container-based applications.

IBM Cloud

IBM Cloud (https://cloud.ibm.com/) provides users with their choice of either a traditional Kubernetes cluster or a Red Hat OpenShift cluster. IBM Cloud’s Kubernetes offering is a managed Kubernetes service that brings all of the standard Kubernetes features including intelligent scheduling, self-healing, horizontal scaling, service discovery and load balancing, automated rollout and rollbacks, and secret and configuration management. In addition IBM Cloud Kubernetes Service includes automated operations for cluster deployment/updates/scaling, expert security, optimized configuration, and seamless integration with the IBM Cloud Infrastructure platform. It produces highly available multi-zone clusters across 6 regions and 35 data centers. IBM Cloud offers both a free Kubernetes cluster with over 40 free services as well as pay as you go options.

IBM Cloud also provides users with highly available, fully managed OpenShift clusters (https://www.ibm.com/cloud/openshift). IBM’s OpenShift offering implements unique security and productivity capabilities designed to eliminate substantial time spent on updating, scaling, and provisioning. Additionally, IBM Cloud’s OpenShift delivers the resiliency to handle unexpected surges and protects against attacks that can lead to financial and productivity losses. In addition to pay as you go and subscription options, IBM Cloud offers a free preconfigured OpenShift version 4.3 environment that is available for four hours at no charge.

OpenShift Deployment Options

There are several deployment options for OpenShift defined at https://www.openshift.com/try. The options described include installing OpenShift version 4 on your laptop, deploying it in your datacenter or public cloud, or having Red Hat manage OpenShift for you. In addition, Red Hat offers hands-on OpenShift tutorials and playground OpenShift environments for unstructured learning and experimentation. Figure 2-2 shows the myriad of OpenShift deployment options available.

Figure 2-2. OpenShift Deployment Options available at https://www.openshift.com/try

In the next section, we describe the command line tools used for interacting with these platforms.

Running the Samples Using the kubectl and oc Command Line Interfaces

After covering some core concepts in Kubernetes, the next sections provide several examples in the form of YAML files. For all Kubernetes environments, the samples included can be run using the standard Kubernetes command line tool known as kubectl. Many Kubernetes environments, including the ones mentioned earlier in this Chapter, describe how kubectl can be installed. Once you have your Kubernetes environment up and running and kubectl installed, all of the following YAML file samples in the next sections can be run by first saving the YAML to a file (e.g., kubesample1.yaml) and then by running the following kubectl command:

kubectl apply -f kubesample1.yaml

As previously discussed, the OpenShift distribution of Kubernetes adds several new enhancements and capabilities beyond those used by traditional Kubernetes. OpenShift provides access to these features by extending the capabilities of kubectl. To make it explicit that the OpenShift version of kubectl has extended functionality, OpenShift renamed its version of kubectl to be a new command line tool called oc. Thus, the following is equivalent to the kubectl command shown above:

oc apply -f kubesample1.yaml

For more information on the breadth of commands available from the OpenShift oc command line interface, please see https://docs.openshift.com/container-platform/4.4/cli_reference/openshift_cli/developer-cli-commands.html.

What’s a Pod?

Because Kubernetes provides support for the management and orchestration of containers, you would assume that the smallest deployable unit supported by Kubernetes would be a container. However, the designers of Kubernetes learned from experience³ that it was more optimal to have the smallest deployable unit be something that could hold multiple containers. In Kubernetes, this smallest deployable unit is called a Pod. A Pod can hold one or more application containers. The application containers that are in the same Pod have the following benefits:

• They share an IP address and port space

• They share the same hostname

• They can communicate with each other using native interprocess communication (IPC)

In contrast, application containers that run in separate pods are guaranteed to have different IP addresses and have different hostnames. Essentially, containers in different pods should be viewed as running on different servers even if they ended up on the same node.

Kubernetes contributes a robust list of features that make Pods easy to use:

Easy-to-use Pod management API

Kubernetes provides the kubectl command-line interface, which supports a variety of operations on Pods. The list of operations includes the creating, viewing, deleting, updating, interacting, and scaling of Pods.

File copy support

Kubernetes makes it very easy to copy files back and forth between your local host machine and your Pods running in the cluster.

Connectivity from your local machine to your Pod

In many cases, you will want to have network connectivity from your local host machine to your Pods running in the cluster. Kubernetes supports port forwarding whereby a network port on your local host machine is connected via a secure tunnel to a port of your Pod that is running in the cluster. This is an excellent feature to assist in debugging applications and services without having to expose them publicly.

Volume storage support

Kubernetes Pods support the attachment of remote network storage volumes to enable the containers in Pods to access persistent storage that remains long after the lifetime of the Pods and the containers that initially utilized it.

Probe-based health-check support

Kubernetes provides health checks in the form of probes to ensure the main processes of your containers are still running. In addition, Kubernetes also administers liveness checks that ensure the containers are actually functioning and capable of doing real work. With this health check support, Kubernetes can recognize if your containers have crashed or become non-functional and restart them on your behalf.

How Do I Describe What’s in My Pod?

Pods and all other resources managed by Kubernetes are described by using a YAML file. The following is a simple YAML file that describes a rudimentary Pod resource:

              apiVersion: v1  
kind: Pod  
metadata:  
 name: nginx
spec:  
 containers:
 - name: nginx
 image: nginx:1.7.9
 ports:
 - containerPort: 80

This field is used to declare which version of the Kubernetes API schema is being used. Kubernetes continues to experience a rapid growth in features and functionality. It manages the complexity that results from its growth in capabilities by supporting multiple versions of its API. By setting the apiVersion field, you can control the API version that your resource uses.

You use the kind field to identify the type of resource the YAML file is describing. In the preceding example, the YAML file declares that it is describing a Pod object.

The metadata section contains information about the resource that the YAML is defining. In the preceding example, the metadata contains a name field that declares the name of this Pod. The metadata section can contain other types of identifying information such as labels and annotations. We describe these in the next section.

The spec section provides a specification for what is the desired state for this resource. As shown in the example, the desired state for this Pod is to have a container with a name of nginx that is built from the Docker image that is identified as nginx:1.7.9. The container shares the IP address of the Pod it is contained in and the containerPort field is used to allocate this container a network port (in this case, 80) that it can use to send and receive network traffic.

To run the previous example, save the file as pod.yaml. You can now run it by doing the following:

$ kubectl apply -f pod.yaml

After running this command, you should see the following output:

pod/nginx created

To confirm that your Pod is actually running, us the kubectl get pods command to verify:

$ kubectl get pods

After running this command, you should see output similar to the following:

NAME  READY  STATUS RESTARTS AGE
nginx  1/1  Running 0   21s

If you need to debug your running container, you can create an interactive shell that runs within the container by using the following command:

$ kubectl exec -it nginx -- bash

This command instructs Kubernetes to run an interactive shell for the container that runs in the Pod named nginx. Because this Pod has only one container, Kubernetes knows which container you want to connect to without you specifying the container name, as well. Typically, accessing the container interactively to modify it at runtime is considered a bad practice. However, interactive shells can be useful as you are learning or debugging apps before deploying to production. After you run the preceding command, you can interact with the container’s runtime environment, as shown here:

root@nginx:/# ls
bin boot dev etc home lib lib64 media mnt opt proc root run sbin selinux srv sys tmp usr var
root@nginx:/# exit

If your Pod had multiple containers within it, you would need to include the container name as well in your kubectl exec command. To do this, you would use the -c option and include the container name in addition to the Pod name. Here is an example:

$ kubectl exec -it nginx -c nginx -- bash
root@nginx:/# exit
exit

To delete the Pod that you just created, run the following command:

$ kubectl delete pod nginx

You should see the following confirmation that the Pod has been deleted:

pod "nginx" deleted

When using Kubernetes you can expect to have large numbers of Pods running in a cluster. In the next section, we describe how labels and annotations are used to help you keep track of and identify your Pods.

Deployments

Deployments are a high-level Kubernetes abstraction that not only allow you to control the number of Pod replicas that are instantiated, but also provide support for rolling out new versions of the Pods. Deployments rely upon the previously described ReplicaSet resource to manage Pod Replicas and then add Pod version management support on top of this capability. Deployments also enable newly rolled out versions of Pods to be rolled back to previous versions if there is something wrong with the new version of the Pods. Furthermore, Deployments support two options for upgrading Pods, Recreate and RollingUpdate:

• Recreate: The Recreate Pod upgrade option is very straightforward. In this approach the Deployment resource modifies its associated ReplicaSet to point to the new version of the Pod. It then proceeds to terminate all the Pods. The ReplicaSet then notices that all the Pods have been terminated and thus spawns new Pods ensure the number of desired Replicas are up and running. The Recreate approach will typically result in your Pod application not being accessible for a period of time and thus it is not recommended for applications that need to always be available.

• RollingUpdate: Kubernetes Deployment resource also provides a RollingUpdate option. With the RollingUpdate option, your Pods are replaced with the newer version incrementally over time. This approach results in there being a mixture of both the old version of the Pod and the new version of the Pod running simultaneously and thus avoids having your Pod application unavailable during this maintenance period. The readiness of each pod is measured and used to inform kube-proxy and Ingress Controllers which Pod replicas are available to handle network requests to ensure that no requests are dropped during the update process.

The following is an example YAML specification for a Deployment that uses the RollingUpdate option:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx
 labels:
 app: webserver
 annotations:
 deployment.kubernetes.io/revision: "1"
spec:
 replicas: 3
 selector:
 matchLabels:
  app: webserver
 strategy:
 rollingUpdate:
  maxSurge: 1
  maxUnavailable: 1
 type: RollingUpdate
 template:
 metadata:
  labels:
  app: webserver
 spec:
  containers:
  - name: nginx
  image: nginx:1.7.9
  ports:
   - containerPort: 80

The above Deployment example encompasses many of the characteristics that we have seen in ReplicaSets and Pods. In its metadata it contains labels and annotations. For the Deployment, an annotation with deployment.kubernetes.io/revision as the key and 1 as its value provides information that this is the first revision of the contents in this Deployment. Similar to ReplicaSets, the Deployment declares the number of replicas it provides and uses a matchLabels field to declare what labels it uses to identify the Pods it manages. Also similar to ReplicaSets, the Deployment has both a spec section for the Deployment and a nested spec section inside a template that is used to describe the containers that comprise the Pod replicas managed by this Deployment.

The fields that are new and specific to a Deployment resource are the strategy field and its subfields of type and rollingUpdate. The type field is used to declare the Deployment strategy being utilized and can currently be set to Recreate or RollingUpdate.

If the RollingUpdate option is chosen, the subfields of maxSurge and maxUnavailable need to be set as well. The options are used as follows:

• maxSurge: The maxSurge RollingUpdate option enables extra resources to be allocated during a rollout. The value of this option can be set to a number or a percentage. As a simple example assume a Deployment is supporting 3 replicas and maxSurge is set to 2. In this scenario there will be a total of five replicas available during the RollingUpdate.

  At peak of the deployment, there will be three replicas with the old version of the Pods running and two with the new version of the Pods running. At this point one of the old version Pod Replicas will need to be terminated and then another replica of the new Pod version can then be created. At this stage there would be a total of 5 replicas, three of which have the new revision and two have the old version of the Pods. Finally, having reached a point of having the correct number of Pod replicas available with the new version, the two Pods with the old version can now be terminated.

• maxUnavailable: This RollingUpdate option is used to declare the number of the Deployment replica pods that may be unavailable during the update. It can either be set to a number or a percentage.

The following YAML example shows a Deployment that has been updated to initiate a rollout. Note that a new annotation label with a key of “kubernetes.op/change-cause” has been added with a value that denotes an update to the version of nginx running in the container. Also notice that the name of the image used by the container in the spec section has changed to “nginx:1.13.10”. This declaration is what actually drives the Pod replicas managed by the Deployment to now have a new version of the container images when the upgrade occurs:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx
 labels:
 app: webserver
 annotations:
 kubernetes.io/change-cause: "Update nginx to 1.13.10"
spec:
 replicas: 3
 selector:
 matchLabels:
  app: webserver
 strategy:
 rollingUpdate:
  maxSurge: 1
  maxUnavailable: 1
 type: RollingUpdate
 template:
 metadata:
  labels:
  app: webserver
 spec:
  containers:
  - name: nginx
  image: nginx:1.13.10
  ports:
   - containerPort: 80

To demonstrate the capabilities of Deployments, let’s run the two examples listed above. Save the first Deployment example as deploymentset.yaml and the second example as deploymentset2.yaml. You can now run the first deployment example by doing the following:

$ kubectl apply -f deploymentset.yaml

After running this command you should see the following output:

deployment.apps/nginx created

To confirm that your Pod replicas managed by the Deployment are actually running, use the kubectl get pods command to verify:

$ kubectl get pods

After running this command you should see output similar to the following:

NAME     READY STATUS RESTARTS AGE
nginx-758fbc45d-2znb7 1/1  Running 0   68s
nginx-758fbc45d-gxf2d 1/1  Running 0   68s
nginx-758fbc45d-s9f9t 1/1  Running 0   68s

With Deployments we have a new command called kubectl get deployments that provides us status on the Deployments as they update their images. We run this command as follows:

$ kubectl get deployments

After running this command you should see output similar to the following:

NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/3  3   3   2m6s

Now to make things interesting let’s update the image in our Deployment by applying our second Deployment example we saved in deploymentset2.yaml. Note that we could have just updated our original YAML we saved in deploymentset.yaml instead of using two separate files. We begin the update by doing the following:

$ kubectl apply -f deploymentset2.yaml

After running this command you should see the following output:

deployment.apps/nginx configured

Now, when we rerun the kubectl get deployments command which provides us status on the Deployments as they update their images, we see a much more interesting result:

$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 2/3  3   2   34s

As shown in the above output, the Deployment currently has three Pod replicas running. Three of the Pod replicas are up to date which means they are now running the updated nginx image. In addition, there are three Pod Replicas in total, and of these three replicas two are available to handle requests. After some amount of time, when the rolling image update is complete, we reach the desired state of having three updated Pod replicas available. We can confirm this by rerunning the kubectl get deployments command and viewing that the output now matches our desired state.

$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/3  3   3   46s

To delete the Deployment that was just created run the following command:

$ kubectl delete deployment nginx

You should get the following confirmation that the Deployment has been deleted.

deployment.apps "nginx" deleted

Deployments also include commands for pausing rollouts, resuming rollouts, and for rolling back the update of an image. The commands are quite helpful if you have some concerns about the new image being rolled out that merits investigation, or if you determine the updated image being rolled out is problematic and needs to be rolled back to a previous version. Please seehttps://kubernetes.io/docs/concepts/workloads/controllers/deployment/ for more information on how to use these Deployment capabilities.

Next, let’s introduce another fundamental Kubernetes concept which is used to provide authentication for key parts of the Kubernetes platform.

Service Accounts

When you interact with your cluster, you often represent yourself as a user identity. In the world of Kubernetes, we build intelligence into the system to help it interact with its world. Many times, Pods may use the Kubernetes API to interact with other parts of the system or to spawn work like Jobs. When we deploy a Pod, it may interact with volume storage, interact with the host file system, interact with the host networking, or be sensitive to what operating system user it is given access to use for file system access. In most cases you want to restrict the default permissions for a given Pod from doing anything more than the absolute basics. Basically, the less surface area which a Pod is given access to in the cluster, the host operating system, the networking layer, and your storage layer, the fewer attack vectors that can be exploited.

For a Pod to interact with the system, it is assigned a service account. Think of this like a functional identity. The service accounts are subjects that can authenticate with the kube-apiserver via tokens and are authorized for certain behaviors.

In some Kubernetes systems the service account projected into the pod can have identity outside of Kubernetes. A powerful use case is when using the open source Istio⁴ service mesh project with Kubernetes. In this scenario, the Istio identity is projected via the service account and this allows one pod to authenticate with another when making service requests. Some cloud providers and other security tools also allow for projection of a service account identity into the pod and this allows for authentication with these external platforms.

In OpenShift, service accounts are also used to associate a grouping of security privileges with each Pod. The object that OpenShift uses for creating specialized groupings of security privileges is called a security context constraint. In the next section, we provide a more detailed discussion of security context constraints as well as several other important enhancements that OpenShift delivers to supplement basic Kubernetes.

Authentication

Security is paramount to the OpenShift Kubernetes platform. As a result, all users must authenticate with the cluster in order to be able to access it. OpenShift supports a variety of common authentication methods including basic authentication with user name and password, OAuth Access Tokens, and X.509 Client Certificates.⁵ OpenShift provides the oc login command for performing authentication and this command is run by doing the following:

oc login

In a basic authentication use case, when the above command is run, the user will be asked to enter the OpenShift Platform Container server URL, whether or not secure connections are needed, and then the user will be asked to input their username and password. In addition, OpenShift’s configurable OAuth server allows for users to integrate OpenShift identity with external providers such as LDAP servers.

Projects

Standard Kubernetes provides the concept of a namespace,⁶ which allows you to define isolation for your Kubernetes resources. Namespaces enable cluster resources to be divided amongst a large number of users and the isolation that results from the scoping that they administer keeps users from accidentally using someone else’s resource due to a naming collision. Namespaces are incredibly useful and OpenShift has adapted namespaces for grouping applications. OpenShift accomplishes this by taking a Kubernetes namespace and adding a special standard list of annotations to the namespace. OpenShift refers to this specific type of namespace as a Project. OpenShift uses projects as its mechanism for grouping applications. Projects support the notion of access permissions. This enables you to add one or more users that have access to the project and role-based access control is used to set the permissions and capabilities that various users have when accessing a project.

Projects are created using the oc new-project command and by providing a project name, description, and display name as shown below:

oc new-project firstproject --description=”My first project” --display-name=”First Project”

OpenShift makes it easy to switch between projects by using the oc project command. Here we switch to a different project called secondproject:

oc project secondproject

To view the list of projects that you are authorized to access you can use the oc get projects command:

oc get projects 

For more information on the use of projects please see https://docs.openshift.com/container-platform/4.4/applications/projects/working-with-projects.html

Applications

When using a basic Kubernetes environment, one of the more tedious steps that needs to be performed by a cloud native application developer is the creation of their own container images. Typically, this involves finding the proper base image and creating a Dockerfile with all the necessary commands for taking a base image and adding in the developers code to create an assembled image that can be deployed by Kubernetes. OpenShift introduced the Application construct to greatly simplify the process of creating, deploying, and running container images in Kubernetes environments.⁷

Applications are created using the oc new-app command. This command supports a variety of options that enable container images to be built in a variety of ways. For example, with the new-app command, application images can be built from local or remote git repositories, or the application image can be pulled from a DockerHub or private image registry. In addition, the new-app command supports the creation of application images by inspecting the root directory of the repository to determine the proper way to create the application image. For example, the OpenShift new-app command will look for a JenkinsFile in the root directory of your repository and if it finds this file it will use it to create the application image. Furthermore, if the new-app command does not find a JenkinsFile, it will attempt to detect the programming language that your application is built in by looking at the files in your repository. If it is able to determine the programming language that was used, the new-app command will locate an acceptable base image for the programming language you are using and will use this to build your application image.

The following example illustrates using the the oc new-app command to create a new application image from an OpenShift example ruby hello world application:

oc new-app https://github.com/openshift/ruby-hello-world.git

The above command will create the application as part of whichever OpenShift project was most recently selected to be the current context for the user. For more information on the application image creation options supported by the new-app command, please see https://docs.openshift.com/container-platform/4.4/applications/application_life_cycle_management/creating-applications-using-cli.html.

Security Context Constraints

Security is always at the forefront in OpenShift. But with added security can come extra complexity and aggravation. If enhanced security is used and a container is not provided the proper security options, it will fail. If security is relaxed to avoid issues then vulnerabilities can result. In an effort to enable users to leverage enhanced security with less aggravation, OpenShift includes a security construct called Security Context Constraints.

The security context constraints identify a set of security privileges that a Pod’s container is guaranteed to execute with. Thus, before the Pod’s container begins execution, it knows what security privileges it will get. The following is a list of the common security privilege options that are provided by a security context constraints:

Allowing Pods to run privileged containers

Security context constraints can declare if a Pod is permitted to run privileged containers or if it can run only non-privileged containers.

Requiring Security Enhanced Linux (SELinux)

Security Enhanced Linux is a security architecture for Linux that defines access controls for applications, processes, and files on a system,⁸ SELinux presents extra protections beyond what is used by standard Linux. Security context constraints provide the MustRunAs attribute value for declaring if SELinux must be run by a Pod’s container and a RunAsAny attribute value for declaring if the Pod’s container can run either standard Linux or SELinux.

Run the Pod’s container as a specific user or as non-root

Containers running as root have a bigger vulnerability footprint than containers running as a non-root. Security context constraints provide a MustRunAsNonRoot attribute value to denote a Pod’s container is not permitted to run as root. Additionally, the security context constraints use a RunAsAny attribute value that permits a Pod’s container to run as either a root or non-root user. Finally, the security context constraint administers a MustRunAsRange attribute value that allows a Pod’s container to run if the user id is within a specific range of user ids.

Allow the Pod’s container access to File System Group block storage

Security context constraints can be used to limit the block storage that a Pod’s container has access to. Block storage portions are identified through the use of s File System Group Identifier. Security context constraints provide a RunAsAny attribute value that permits a Pod’s container to access any File System Group of block storage as well as a MustRunAs attribute value which is used to denote that the Pod’s block storage must be in the range of File System Group Ids listed in the security context constraint.

OpenShift includes several built in security context constraint profiles that can be reused. To view the list of projects that you are authorized to access you can use the oc get scc command:

$ oc get scc
NAME    AGE
anyuid    182d
hostaccess   182d
hostmount-anyuid 182d
hostnetwork  182d
node-exporter  182d
nonroot   182d
privileged   182d
restricted   182d

As shown above, OpenShift contributes security context constraint profiles for common scenarios such as privileged, or restricted, or running as nonroot. To see all the individual capability settings for the security constraint profile use the oc describe scc command and pass in the name of the profile that you want more details on. For example, if you wanted more details on how powerful the privileged constraint profile is, you would invoke the occ describe scc command as follows:

oc describe scc privileged

Running this command will list a large number constraint attributes associated with this profile. A few of the more interesting ones are listed below:

Settings:
 Allow Privileged:        true
 Allow Privilege Escalation:     true
 Default Add Capabilities:      <none>
 Required Drop Capabilities:     <none>
 Allowed Capabilities:       *
 Allowed Seccomp Profiles:      *
 Allowed Volume Types:       *
 Allowed Flexvolumes:       <all>
 Allowed Unsafe Sysctls:      *
 Forbidden Sysctls:       <none>
 Allow Host Network:       true
 Allow Host Ports:        true
 Allow Host PID:        true
 Allow Host IPC:        true
 Read Only Root Filesystem:     false
 Run As User Strategy: RunAsAny
 SELinux Context Strategy: RunAsAny
 FSGroup Strategy: RunAsAny
 Supplemental Groups Strategy: RunAsAny

For comparison purposes, we can run the same command for the restricted profile. As shown in the output below, the constraint attribute values are much more restrictive than those in the privileged profile:

oc describe scc restricted
Settings:
 Allow Privileged:        false
 Allow Privilege Escalation:     true
 Default Add Capabilities:      <none>
 Required Drop Capabilities:     KILL,MKNOD,SETUID,SETGID
 Allowed Capabilities:       <none>
 Allowed Seccomp Profiles:      <none>
 Allowed Volume Types:       configMap,downwardAPI,emptyDir,persistentVolumeClaim,projected,secret
 Allowed Flexvolumes:       <all>
 Allowed Unsafe Sysctls:      <none>
 Forbidden Sysctls:       <none>
 Allow Host Network:       false
 Allow Host Ports:        false
 Allow Host PID:        false
 Allow Host IPC:        false
 Read Only Root Filesystem:     false
 Run As User Strategy: MustRunAsRange
 SELinux Context Strategy: MustRunAs
 FSGroup Strategy: MustRunAs
 Supplemental Groups Strategy: RunAsAny

The key point here is that security context constraint profiles are able to group and encapsulate large groups of capability attributes and ensure all the attributes are met before a Pod is permitted to execute. This reduces the chance of improperly setting the capability attributes and reduces the chance of an unexpected Pod failure due to an incorrect security setting.

Security context constraint profiles are associated with Pods by using the Kubernetes Service Account object. This resource is covered in more detail later in this chapter. For more information on the use of security context constraints, please see https://docs.openshift.com/container-platform/4.4/authentication/managing-security-context-constraints.html.

Image Streams

One of the key steps in the deployment of a cloud native application is the retrieval of the correct container application image from a repository. When running in production, there are several pitfalls that can exist with this retrieval process. First, container images are retrieved by a tag identifier, but it is possible that container images can be overwritten and thus the image that is referenced by the tag can change. If this change goes unnoticed it could result in introducing unexpected errors into the cloud native application that is deployed. Second, when running in production the image retrieval process also needs to be supplemented with support for the automation of builds and deployments and many image repositories are limited in their ability to support this automation. Third, in some cases a container image needs to have multiple tags associated with it because the container image is used for different purposes in different environments. Unfortunately, many image repositories do not support the ability to associate multiple tags with a container application image.

To address all of these issues, OpenShift introduced the concept of image streams.⁹ Image streams are intended to provide a more stable pointer for tagged images. The image stream maintains a sha256 secure hash function to the image it points to in order to ensure the image is not mistakenly changed. Image streams also support multiple tags for images to better support using images in multiple environments. In addition, image streams include triggers that enable builds and deployments to be started automatically when the image stream is updated. Furthermore, image streams can not only reference container images from external repositories, but they can also be scheduled to periodically re-import the external container image to ensure they always have the most recently updated copy of the container image they are referencing in the external repository.

Creating and updating image streams is relatively straightforward. The oc import-image command is used to create an image stream. In the following example, the oc import-image command is used to create an initial image stream called nginx with an initial image stream tag for the imported image that has the value 1.12:

oc import-image nginx:1.12 --from=centos/nginx-112-centos7 --confirm

As shown in the above example, the initial container image that is being imported into the nginx image stream is the image that is located at centos/nginx-112-centos7. The confirm option states that the image stream should be created if it didn’t already exist.

Once the image stream is created we can examine it using the oc describe command. In the following example, the is value is the short name for an input stream resource. The specific input stream that we want described is the one with the name nginx:

oc describe is/nginx

The output from this command looks like the following:

$ oc describe is/nginx
Name:     nginx
Namespace:    default
Created:    52 seconds ago
Labels:     <none>
Annotations:   openshift.io/image.dockerRepositoryCheck=2020-06-12T20:16:15Z
Image Repository:  default-route-openshift-image-registry.apps-crc.testing/default/nginx
Image Lookup:   local=false
Unique Images:   1
Tags:     1
1.12
 tagged from centos/nginx-112-centos7
*centos/nginx-112-centos7@sha256:af171c38298e64664a9f999194480ce7e392858e773904df22f7585a1731ad0d

We can add an extra tag for this image by using the oc tag command. As shown in the example below, we add an nginx:latest tag to the existing nginx:1.12 tag by doing the following:

oc tag nginx:1.12 nginx:latest

Finally, we can tag an image from an external repository and schedule this image to be periodically re-imported by calling the oc tag command. As shown in the following example, we reference the image from the external repository, associate with an image stream tag, and then add the scheduled option to denote that the tag should be periodically updated:¹⁰

oc tag docker.io/nginx:1.14 nginx:1.14 --scheduled

For more information on the use of image streams, please see https://docs.openshift.com/container-platform/4.4/openshift_images/image-streams-manage.html

Webhooks

A Webhook is an HTTP callback. Essentially, a webhook enables information to be pushed to an external entity when an interesting event is occurring. Typically an HTTP Post operation is used to push the event information and the event information is most commonly represented as a JSON payload. In Kubernetes, webhooks are used for a variety of security related operations. For example, a webhook can be used by Kubernetes to query an external service to determine if a user has the correct privileges to perform a specific operation.

Webhooks are also used by OpenShift as a mechanism for triggering builds.¹¹ With webhooks, you can configure your GitHub repository to send an alert whenever there is a change in the repository. This alert can be used to kick off a new build, and if the build succeeds perform a deployment as well. For more details on how OpenShift uses webhooks for triggering builds please see https://docs.openshift.com/container-platform/4.4/builds/triggering-builds-build-hooks.html.

Webhooks are also used heavily by Kubernetes Admission Controllers which are described in the next section. For more information on the use of webhooks in Kubernetes please see https://kubernetes.io/docs/reference/access-authn-authz/webhook/.

Admission Controllers

The key to keeping your Kubernetes platform secure is by protecting it from requests that can cause harm. Admission Controllers are one of the mechanisms that Kubernetes uses to protect the platform from harmful requests. In some cases an admission controller will prevent a request from creating the Kubernetes object at all. In other cases, the admission controller will allow the request to be processed, but it will modify the request to make it safer. As an example, if a request comes in to start a Pod, and the request does not specify whether the Pod should be started in privileged or non-privileged mode, the admission controller could change the request such that in this situation the Pod is requested to be started in non-privileged mode.

A number of admission controllers are embedded in the kube-controller-manager and many are enabled in Kubernetes by default to keep the Kubernetes platform secure. In some cases there is enforcement the admin needs beyond the scope of the included admission controllers. Kubernetes allows the admin to add additional admission controllers via registration of webhooks to process requests on Kubernetes objects. We will go into more detail regarding admission controllers in Chapter 3 of this book.

Role Based Access Control

Authorization in Kubernetes is integrated into the platform. Kubernetes authorization uses a Role Based Access Control model and provides a fully featured authorization platform that allows operators to define various roles via Kubernetes objects, ClusterRole and Role, and bind them to users and groups using ClusterRoleBinding and RoleBinding. Think of RBAC as a way of setting permissions on a file system, but in the case of Kubernetes its setting permissions on the Kubernetes object model. We’ll cover the details of how to use RBAC and how best to build a multi-tenancy model around it in Chapter 4.

Operators

Kubernetes has built in abstractions such as Deployments that are extremely well-suited stateless applications. In addition, Kubernetes has a very elegant design based upon control loops that enables it to support a declarative programming model and allows the platform to execute robustly at large scale even when failures are common.

To support complex stateful applications, Kubernetes needed an extensibility model that would enable users to add custom resources and perform lifecycle management for those resources. Additionally, it would be ideal if the extensibility model could also support the control loop architecture that is used extensively inside the Kubernetes platform. Kubernetes includes the Operator pattern which provides an extensibility model for custom resources that meets all these requirements.¹²

Operators support the creation of custom resources. What this means is that you can define a new resource type in Kubernetes by creating a custom resource definition and this new resource can be stored in the Kubernetes etcd database just like any standard Kubernetes resource. Additionally, you can create a custom controller for your resource that performs the same type of control loop behavior that the standard Kubernetes controllers perform. The custom controller can then monitor the actual state of your stateful application and compare it to the desired state and then take actions to attempt to achieve the desired state for the application. For example, let’s say you create an operator for a special type of database which is a stateful application. The operator and its controller can make sure that the actual number of replicas of the database that are running matches the desired number of copies. Furthermore, since the operator has a custom controller, any custom lifecycle management code that is needed for starting up new copies of the database or updating existing copies of the database can be added to the controller.

The Operator pattern is well designed and a key advantage is that it is seamless. The custom resources associated with an operator are managed using the kubectl command line tool and look just like a standard Kubernetes resource from a management perspective. To ease the creation of operators, an operator SDK toolkit exists to generate the custom resource definitions and a large portion of the controller code required to run the operator’s control loop. As a result of the clean architectural design of the operator framework and also due to extensive tooling available, the creation of new operators as the means for adding stateful applications continues to grow in popularity. There is now an Operator Hub (https://operatorhub.io/) that hosts a large number of existing and reusable operators for managing a variety of applications for the Kubernetes platform. We will go into more detail about Operators and their consumption within Kubernetes later in this book.