Packaging: The Challenge

While Kubernetes defines resources with manifests, typically written in YAML, a low-level interface to declare the state of resources, these manifest files have shortcomings. Most importantly in the context of packaging containerized apps, the YAML manifests are static; that is, all values in a YAML manifest are fixed. This means that if you want to change the container image in a deployment manifest, for example, you have to create a new manifest.

Let’s look at a concrete example. Assume you have the following Kubernetes deployment encoded in a YAML manifest called mycontroller.yaml, representing the custom controller you’d like users to install:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: mycustomcontroller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: customcontroller
    spec:
      containers:
      - name: thecontroller
        image: example/controller:0.1.0
        ports:
        - containerPort: 9999
        env:
        - name: REGION
          value: eu-west-1

Imagine the environment variable REGION defines certain runtime properties of your controller, such as the availability of other services like a managed service mesh. In other words, while the default value of eu-west-1 might be a sensible one, users can and likely will overwrite it, based on their own preferences or policies.

Now, given that the YAML manifest mycontroller.yaml itself is a static file with all values defined at the time of writing—and clients such as kubectl don’t inherently support variable parts in the manifest—how do you enable users to supply variable values or overwrite existing values at runtime? That is, how in the preceding example can a user set REGION to, say, us-east-2 when they’re installing it, using (for example) kubectl apply?

To overcome these limitations of build-time, static YAML manifests in Kubernetes, there are a few options to templatize the manifests (Helm, for example) or otherwise enable variable input (Kustomize), depending on user-provided values or runtime properties.

Helm

Helm, which touts itself as the package manager for Kubernetes, was originally developed by Deis and is now a Cloud Native Computing Foundation (CNCF) project with major contributors from Microsoft, Google, and Bitnami (now part of VMware).

Helm helps you to install and upgrade Kubernetes applications by defining and applying so-called charts, effectively parameterized YAML manifests. Here is an excerpt of an example chart template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "flagger.fullname" . }}
...
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ template "flagger.name" . }}
      app.kubernetes.io/instance: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ template "flagger.name" . }}
        app.kubernetes.io/instance: {{ .Release.Name }}
    spec:
      serviceAccountName: {{ template "flagger.serviceAccountName" . }}
      containers:
        - name: flagger
          securityContext:
            readOnlyRootFilesystem: true
            runAsUser: 10001
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

As you can see, variables are encoded in {{ ._Some.value.here_ }} format, which happens to be Go templates.

To install a chart, you can run the helm install command. While Helm has several ways to find and install charts, the easiest is to use one of the official stable charts:

# get the latest list of charts:
$ helm repo update

# install MySQL:
$ helm install stable/mysql
Released smiling-penguin

# list running apps:
$ helm ls
NAME             VERSION   UPDATED                   STATUS    CHART
smiling-penguin  1         Wed Sep 28 12:59:46 2016  DEPLOYED  mysql-0.1.0

# remove it:
$ helm delete smiling-penguin
Removed smiling-penguin

In order to package your controller, you will need to create a Helm chart for it and publish it somewhere, by default to a public repository indexed and accessible through the Helm Hub, as depicted in Figure 7-1.

Figure 7-1. Helm Hub screenshot showing publicly available Helm charts

For further guidance on how to create Helm charts, peruse the following resources at your leisure:

• Bitnami’s excellent article “How to Create Your First Helm Chart”.

• “Using S3 as a Helm Repository”, if you want to keep the charts in your own organization.

• The official Helm docs: “The Chart Best Practices Guide”.

Helm is popular, partly because of its ease of use for end users. However, some argue that the current Helm architecture introduces security risks. The good news is that the community is actively working on addressing those.

Kustomize

Kustomize provides a declarative approach to configuration customization of Kubernetes manifest files, adhering to the familiar Kubernetes API. It was introduced in mid-2018 and is now a Kubernetes SIG CLI project.

You can install Kustomize on your machine, as a standalone, or, if you have a more recent kubectl version (newer than 1.14), it is shipped with kubectl and activated with the -k command-line flag.

So, Kustomize lets you customize the raw YAML manifest files, without touching the original manifest. But how does this work in practice? Let’s assume you want to package our cnat custom controller; you’d define a file called kustomize.yaml that looks something like:

imageTags:
  - name: quay.io/programming-kubernetes/cnat-operator
    newTag: 0.1.0
resources:
- cnat-controller.yaml

Now you can apply this to the cnat-controller.yaml file, say, with the following content:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: cnat-controller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: cnat
    spec:
      containers:
      - name: custom-controller
        image: quay.io/programming-kubernetes/cnat-operator

Use kustomize build and—leaving the cnat-controller.yaml file unchanged!—the output is then:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: cnat-controller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: cnat
    spec:
      containers:
      - name: custom-controller
        image: quay.io/programming-kubernetes/cnat-operator:0.1.0

The output of kustomize build can then, for example, be used in a kubectl apply command, with all the customizations applied for you, automatically.

For a more detailed walk-through of Kustomize and how to use it, check out the following resources:

• Sébastien Goasguen’s blog post “Configuring Kubernetes Applications with kustomize".

• Kevin Davin’s post “Kustomize—The right way to do templating in Kubernetes”.

• The video “TGI Kubernetes 072: Kustomize and friends”, where you can watch Joe Beda apply it.

Given the native support of Kustomize in kubectl, it’s likely that an increasing number of users will adopt it. Note that while it solves some problems (customization), there are other areas of the lifecycle management, such as validations and upgrades, that may require you to use Kustomize together with languages such as Google’s CUE.

To wrap up this packaging topic, let’s review some other solutions practitioners use.

Other Packaging Options

Some notable alternatives to the aforementioned packaging options—and the many others in the wild—are:

UNIX tooling

In order to customize values of raw Kubernetes manifests, you can use a range of CLI tools such as sed, awk, or jq in shell scripts. This is a popular solution and, at least until the arrival of Helm, likely the most widely used option—not least because it minimizes dependencies and is rather portable across *nix environments.

Traditional configuration management systems

You can use any of the traditional configuration management systems, such as Ansible, Puppet, Chef, or Salt, to package and deliver your operator.

Cloud-native languages

A new generation of so-called cloud-native programming languages, such as Pulumi and Ballerina, allows for, among other things, packaging and lifecycle management of Kubernetes-native apps.

ytt

With ytt you have another option for a YAML templating tool using a language that is itself a modified version of Google’s configuration language Starlark. It operates semantically on the YAML structures and focuses on reusability.

Ksonnet

A configuration management tool for Kubernetes manifests, originally developed by Heptio (now VMware), Ksonnet has been deprecated and is not actively worked on anymore, so use it at your own risk.

Read more about the options discussed here in Jesse Suen’s post “The State of Kubernetes Configuration Management: An Unsolved Problem”.

Now that we’ve discussed the packaging options in general, let’s look at best practices for packaging and shipping controllers and operators.

Packaging Best Practices

When packaging and publishing your operator, make sure you are aware of the following best practices. These apply regardless of which mechanism you choose (Helm, Kustomize, shell scripts, etc.):

• Provide a proper access control setup: this means defining a dedicated service account for the controller along with the RBAC permissions on a least-privileges basis; see “Getting the Permissions Right” for further details.

• Consider the scope of your custom controller: will it look after CRs in one namespace or more than one namespace? Check out Alex Ellis’s Twitter conversation about the pros and cons of the different approaches.

• Test and profile your controller so that you have an idea of its footprint and scalability. For example, Red Hat has put together a detailed set of requirements with instructions in the OperatorHub contribution guide.

• Make sure the CRDs and controller are well documented, ideally with the inline docs available on godoc.org and a set of usage examples; see Banzai Cloud’s bank-vaults operator for inspiration.

Lifecycle Management

A broader and more holistic approach, compared to package/ship, is that of lifecycle management. The basic idea is to consider the entire supply chain, from development to shipping to upgrades, and automate as much as possible. In this area, CoreOS (and later Red Hat) was again a trailblazer: applying the same logic that led to operators to their lifecycle management. In other words: in order to install and later upgrade the custom controller of an operator, you’d have a dedicated operator that knows how to, well, handle operators. And indeed, part of the Operator Framework—which also provides the Operator SDK, as discussed in “The Operator SDK”—is the so-called Operator Lifecycle Manager (OLM).

Jimmy Zelinskie, one of the main people behind OLM, phrased it as follows:

OLM does a lot for Operator authors, but it also solves an important problem that not many people have thought about yet: how do you effectively manage first-class extensions to Kubernetes over time?

In a nutshell, OLM provides a declarative way to install and upgrade operators and their dependencies, complementary packaging solutions such as Helm. It’s up to you if you want to buy into the full-blown OLM solution or create an ad hoc solution for the versioning and upgrading challenge; however, you should have some strategy in place here. For certain areas—for example, the certification process for the Operator Hub by Red Hat—it’s not only recommended but mandatory for any nontrivial deployment scenario, even if you don’t aim at the Hub.

Getting the Permissions Right

Your custom controller is part of the Kubernetes control plane. It needs to read the state of resources, create resources inside as well as (potentially) outside Kubernetes, and communicate the state of its own resources. For all of this, the custom controller needs the right set of permissions, expressed through a set of role-based access control (RBAC)–related settings. Getting this right is the topic of this section.

First things first: always create a dedicated service account to run your controller. In other words: never use the default service account in a namespace.¹

To make your life easier, you can define a ClusterRole with the necessary RBAC rules along with a RoleBinding to bind it to a specific namespace, effectively reusing the role across namespaces, as explained in the Using RBAC Authorization entry.

Following the least-privileges principle, assign only the permissions necessary for the controller to carry out its work. For example, if a controller only manages pods, there is no need to provide it with the permissions to list or create deployments or services. Also, make sure that the controller does not install the CRDs and/or the admission webhooks. In other words, the controller should not have permissions to manage CRDs and webhooks.

Common tooling for creating custom controllers, as discussed in Chapter 6, typically provides functionality for generating RBAC rules out-of-the-box. For example, Kubebuilder generates the following RBAC assets, along with an operator:

$ ls -al rbac/
total 40
drwx------  7 mhausenblas  staff   224 12 Apr 09:52 .
drwx------  7 mhausenblas  staff   224 12 Apr 09:55 ..
-rw-------  1 mhausenblas  staff   280 12 Apr 09:49 auth_proxy_role.yaml
-rw-------  1 mhausenblas  staff   257 12 Apr 09:49 auth_proxy_role_binding.yaml
-rw-------  1 mhausenblas  staff   449 12 Apr 09:49 auth_proxy_service.yaml
-rw-r--r--  1 mhausenblas  staff  1044 12 Apr 10:50 rbac_role.yaml
-rw-r--r--  1 mhausenblas  staff   287 12 Apr 10:50 rbac_role_binding.yaml

Looking at the autogenerated RBAC roles and bindings reveals a fine-grained setup. In rbac_role.yaml you can find:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: manager-role
rules:
- apiGroups:
  - apps
  resources:
  - deployments
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups:
  - apps
  resources:
  - deployments/status
  verbs: ["get", "update", "patch"]
- apiGroups:
  - cnat.programming-kubernetes.info
  resources:
  - ats
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups:
  - cnat.programming-kubernetes.info
  resources:
  - ats/status
  verbs: ["get", "update", "patch"]
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups:
  - ""
  resources:
  - secrets
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups:
  - ""
  resources:
  - services
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Looking at these permissions that Kubebuilder generates in v1, you’ll likely be a little taken aback.² We certainly were: best practice tells us that a controller, if it does not have very good reasons for doing so, should not be able to:

• Write resources that are only read in the code, generally. For example, if you only watch services and deployments, do remove the create, update, patch, and delete verbs in the role.

• Access all secrets; that is, always restrict this to the most minimal set of secrets necessary.

• Write MutatingWebhookConfigurations or ValidatingWebhookConfigurations. This is equivalent to getting access to any resource in the cluster.

• Write CustomResourceDefinitions. Note that this is not allowed in the cluster role just shown, but it’s important to mention here, nevertheless: CRD creation should be done by a separate process, not by the controller itself.

• Write the /status subresource (see “Subresources”) of foreign resources that it is not managing. For example, deployments here are not managed by the cnat controller and should not be in scope.

Kubebuilder, of course, is not really able to understand what your controller code is actually doing. So it’s not surprising that the generated RBAC rules are far too relaxed. We recommend double-checking the permissions and reducing them to the absolute minimum, following the preceding checklist.

From rbac_role_binding.yaml you can learn:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: null
  name: manager-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: manager-role
subjects:
- kind: ServiceAccount
  name: default
  namespace: system

For more best practices on RBAC and tooling around it, check out RBAC.dev, a website dedicated to RBAC in Kubernetes. Let’s move on now to testing and performance considerations for custom controllers.

Automated Builds and Testing

As a best practice in cloud-native land, consider an automated build of your custom controller. This is usually called continuous build or continuous integration (CI) and comprises unit tests, integration tests, building the container image, and potentially even sanity or smoke tests. The Cloud Native Computing Foundation (CNCF) maintains an interactive listing of the many open source CI tools available.

When building your controller, keep in mind that it should consume as few compute resources as possible, while at the same time serving as many clients as possible. Each CR, based on the CRD(s) you define, is a proxy for a client. But how do you know how much it consumes, if and where it leaks memory, and how well it scales?

You can and indeed should carry out a number of tests, once the development of your custom controller stabilizes. These can include the following, but may not be limited to them:

• Performance-related tests, as found in Kubernetes itself as well as the kboom tool, can provide you with data around scaling and resource footprints.

• Soak tests—for example, the ones used in Kubernetes—aim at long-term usage, from several hours to days, with the goal of unveiling any leaking of resources, like files or main memory.

As a best practice, these tests should be part of your CI pipeline. In other words, automate the building of the custom controller, testing, and packaging from day one. For a concrete example setup we encourage you to check out Marko Mudrinić’s excellent post “Spawning Kubernetes Clusters in CI for Integration and E2E tests”.

Next, we’ll look at best practices that provide the basis for effective troubleshooting: built-in support for observability.

Custom Controllers and Observability

In this section we look at observability aspects of your custom controllers, specifically logging and monitoring.

{ "level":"info",
  "ts":1555063927.492718,
  "logger":"controller",
  "msg":"=== Reconciling At" }
{ "level":"info",
  "ts":1555063927.49283,
  "logger":"controller",
  "msg":"Phase: PENDING" }
{ "level":"info",
  "ts":1555063927.492857,
  "logger":"controller",
  "msg":"Checking schedule" }
{ "level":"info",
  "ts":1555063927.492915,
  "logger":"controller",
  "msg":"Schedule parsing done" }

reqLogger := log.WithValues("namespace", request.Namespace, "at", request.Name)

case cnatv1alpha1.PhasePending:
  reqLogger.Info("Phase: PENDING")
  // As long as we haven't executed the command yet, we need to check if it's
  // already time to act:
  reqLogger.Info("Checking schedule", "Target", instance.Spec.Schedule)
  // Check if it's already time to execute the command with a tolerance of
  // 2 seconds:
  d, err := timeUntilSchedule(instance.Spec.Schedule)
  if err != nil {
    reqLogger.Error(err, "Schedule parsing failure")
    // Error reading the schedule. Wait until it is fixed.
    return reconcile.Result{}, err
  }
  reqLogger.Info("Schedule parsing done", "Result", "diff", fmt.Sprintf("%v", d))
  if d > 0 {
    // Not yet time to execute the command, wait until the scheduled time
    return reconcile.Result{RequeueAfter: d}, nil
  }
  reqLogger.Info("It's time!", "Ready to execute", instance.Spec.Command)
  instance.Status.Phase = cnatv1alpha1.PhaseRunning

Monitoring, instrumentation, and auditing

A great open source, container-ready monitoring solution you can use across environments (on-premises and in the cloud) is Prometheus. Alerting on each event is not practical, so you might want to think about who needs to be informed about what kind of event. For example, you could have a policy that node-related or namespace-related events are handled by infrastructure admins, and namespace admins or developers are paged for pod-level events. In this context, in order to visualize the metrics you’ve gathered, the most popular solution is certainly Grafana; see Figure 7-2 for an example of Prometheus metrics visualized in Grafana, taken from the Prometheus documentation.

If you are using a service mesh—for example, based on the Envoy proxy (like Istio or App Mesh), or Linkerd—then instrumentation typically comes for free or is achievable with minimal (configuration) effort. Otherwise, you will have to use the respective libraries, such as those provided by Prometheus, to expose the relevant metrics in your code yourself. In this context, you might also want to check out the fledgling Service Mesh Interface (SMI) project, introduced in early 2019, which aims to provide a standardized interface for service meshes, based on CRs and controllers.

Figure 7-2. Prometheus metrics visualized in Grafana

Another useful feature Kubernetes offers via the API server is auditing, which allows you to record a sequence of activities affecting the cluster. Different strategies are available in the auditing policy, from no logging to logging event metadata, request bodies, and response bodies. You can choose between a simple log backend and using a webhook for integrating with third-party systems.