Revising the Pizza Restaurant

To learn how CR conversion works, we’ll reimplement the pizza restaurant example from Chapter 8, this time purely with CRDs—that is, without the aggregated API server involved.

For conversion, we will concentrate on the Pizza resource:

apiVersion: restaurant.programming-kubernetes.info/v1alpha1
kind: Pizza
metadata:
  name: margherita
spec:
  toppings:
  - mozzarella
  - tomato

This object should have a different representation of the toppings slice in the v1beta1 version:

apiVersion: restaurant.programming-kubernetes.info/v1beta1
kind: Pizza
metadata:
  name: margherita
spec:
  toppings:
  - name: mozzarella
    quantity: 1
  - name: tomato
    quantity: 1

While in v1alpha1, repetition of toppings is used to represent an extra cheese pizza, we do this in v1beta1 by using a quantity field for each topping. The order of toppings does not matter.

We want to implement this translation—converting from v1alpha1 to v1beta1 and back. Before we do so, though, let’s define the API as a CRD. Note here that we cannot have an aggregated API server and CRDs of the same GroupVersion in the same cluster. So make sure that the APIServices from Chapter 8 are removed before continuing with the CRDs here.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: pizzas.restaurant.programming-kubernetes.info
spec:
  group: restaurant.programming-kubernetes.info
  names:
    kind: Pizza
    listKind: PizzaList
    plural: pizzas
    singular: pizza
  scope: Namespaced
  version: v1alpha1
  versions:
  - name: v1alpha1
    served: true
    storage: true
    schema: ...
  - name: v1beta1
    served: true
    storage: false
    schema: ...

The CRD defines two versions: v1alpha1 and v1beta1. We set the former as the storage version (see Figure 9-1), meaning every object to be stored in etcd is first converted to v1alpha1.

Figure 9-1. Conversion and storage version

As the CRD is defined currently, we can create an object as v1alpha1 and retrieve it as v1beta1, but both API endpoints return the same object. This is obviously not what we want. But we’ll improve this very soon.

But before we do that, we’ll set up the CRD in a cluster and create a margherita pizza:

apiVersion: restaurant.programming-kubernetes.info/v1alpha1
kind: Pizza
metadata:
  name: margherita
spec:
  toppings:
  - mozzarella
  - tomato

We register the preceding CRD and then create the margherita object:

$ kubectl create -f pizza-crd.yaml
$ kubectl create -f margherita-pizza.yaml

As expected, we get back the same object for both versions:

$ kubectl get pizza margherita -o yaml
apiVersion: restaurant.programming-kubernetes.info/v1beta1
kind: Pizza
metadata:
  creationTimestamp: "2019-04-14T11:39:20Z"
  generation: 1
  name: margherita
  namespace: pizza-apiserver
  resourceVersion: "47959"
  selfLink: /apis/restaurant.programming-kubernetes.info/v1beta1/namespaces/pizza-apiserver/
  pizzas/margherita
  uid: f18427f0-5ea9-11e9-8219-124e4d2dc074
spec:
  toppings:
  - mozzarella
  - tomato

Kubernetes uses the canonical version order; that is:

v1alpha1

Unstable: might go away or change any time and often disabled by default.

v1beta1

Towards stable: exists at least in one release in parallel to v1; contract: no incompatible API changes.

v1

Stable or generally available (GA): will stay for good, and will be compatible.

The GA versions come first in that order, then the betas, and then the alphas, with the major version ordered from high to low and the same for the minor version. Every CRD version not fitting into this pattern comes last, ordered alphabetically.

In our case, the preceding kubectl get pizza therefore returns v1beta1, although the created object was in version v1alpha1.

Conversion Webhook Architecture

Now let’s add the conversion from v1alpha1 to v1beta1 and back. CRD conversions are implemented via webhooks in Kubernetes. Figure 9-2 shows the flow:

1. The client (e.g., our kubectl get pizza margherita) requests a version.

2. etcd has stored the object in some version.

3. If the versions do not match, the storage object is sent to the webhook server for conversion. The webhook returns a response with the converted object.

4. The converted object is sent back to the client.

Figure 9-2. Conversion webhook

We have to implement this webhook server. Before doing so, let’s look at the webhook API. The Kubernetes API server sends a ConversionReview object in the API group apiextensions.k8s.io/v1beta1:

type ConversionReview struct {
    metav1.TypeMeta `json:",inline"`
    Request *ConversionRequest
    Response *ConversionResponse
}

The request field is set in the payload sent to the webhook. The response field is set in the response.

The request looks like this:

type ConversionRequest struct {
    ...

    // `desiredAPIVersion` is the version to convert given objects to.
    // For example, "myapi.example.com/v1."
    DesiredAPIVersion string

    // `objects` is the list of CR objects to be converted.
    Objects []runtime.RawExtension
}

The DesiredAPIVersion string has the usual apiVersion format we know from TypeMeta: group/version.

The objects field has a number of objects. It is a slice because for one list request for pizzas, the webhook will receive one conversion request, with this slice being all objects for the list request.

The webhook converts and sets the response:

type ConversionResponse struct {
    ...

    // `convertedObjects` is the list of converted versions of `request.objects`
    // if the `result` is successful otherwise empty. The webhook is expected to
    // set apiVersion of these objects to the ConversionRequest.desiredAPIVersion.
    // The list must also have the same size as input list with the same objects
    // in the same order (i.e. equal UIDs and object meta).
    ConvertedObjects []runtime.RawExtension

    // `result` contains the result of conversion with extra details if the
    // conversion failed. `result.status` determines if the conversion failed
    // or succeeded. The `result.status` field is required and represents the
    // success or failure of the conversion. A successful conversion must set
    // `result.status` to `Success`. A failed conversion must set `result.status`
    // to `Failure` and provide more details in `result.message` and return http
    // status 200. The `result.message` will be used to construct an error
    // message for the end user.
    Result metav1.Status
}

The result status tells the Kubernetes API server whether the conversion was successful.

But when in the request pipeline is our conversion webhook actually called? What kind of input object can we expect? To understand this better, take a look at the general request pipeline in Figure 9-3: all those solid and striped circles are where conversion takes place in the k8s.io/apiserver code.

Figure 9-3. Conversion webhook calls for CRs

In contrast to aggregated custom API servers (see “Internal Types and Conversion”), CRs do not use internal types but convert directly between the external API versions. Hence, only those yellow circles are actually doing conversions in Figure 9-4; the solid circles are NOOPs for CRDs. In other words: CRD conversion takes place only from and to etcd.

Figure 9-4. Where conversion takes place for CRs

Therefore, we can assume our webhook will be called from those two places in the request pipeline (refer to Figure 9-3).

Also note that patch requests do automatic retries on conflict (updates cannot retry, and they respond with errors directly to the caller). Each retry consists of a read and write to etcd (the yellow circles in Figure 9-3) and therefore leads to two calls to the webhook per iteration.

Before we start implementing the webhook, some final words about what the webhook can do and must avoid:

• The order of the objects in request and response must not change.

• ObjectMeta with the exception of labels and annotation must not be mutated.

• Conversion is all or nothing: either all objects are successfully converted or all fail.

Conversion Webhook Implementation

With the theory behind us, we are ready to start the implementation of the webhook project. You can find the source at the repository, which includes:

• A webhook implementation as an HTTPS web server

• A number of endpoints:

  • /convert/v1beta1/pizza converts a pizza object between v1alpha1 and v1beta1.

  • /admit/v1beta1/pizza defaults the spec.toppings field to mozzarella, tomato, salami.

  • /validate/v1beta1/pizza verifies that each specified topping has a corresponding toppings object.

The last two endpoints are admission webhooks, which will be discussed in detail in “Admission Webhooks”. The same webhook binary will serve both admission and conversion.

The v1beta1 in these paths should not be confused with v1beta1 of our restaurant API group, but it is meant as the apiextensions.k8s.io API group version we support as a webhook. Someday v1 of that webhook API will be supported,¹ at which time we’ll add the corresponding v1 as another endpoint, in order to support old (as of today) and new Kubernetes clusters. It is possible to specify inside the CRD manifest which versions a webhook supports.

Let’s look into how this conversion webhook actually works. Afterwards we will take a deeper dive into how to deploy the webhook into a real cluster. Note again that webhook conversion is still alpha in 1.14 and must be enabled manually using the CustomResourceWebhookConversion feature gate, but it is available as beta in 1.15.

Setting Up the HTTPS Server

The first step is to start a web server with support for transport layer security, or TLS (i.e., HTTPS). Webhooks in Kubernetes require HTTPS. The conversion webhook even requires certificates that are successfully checked by the Kubernetes API server against the CA bundle provided in the CRD object.

In the example project, we make use of the secure serving library that is part of the k8s.io/apiserver. It provides all TLS flags and behavior you might be used to from deploying a kube-apiserver or an aggregated API server binary.

The k8s.io/apiserver secure serving code follows the options-config pattern (see “Options and Config Pattern and Startup Plumbing”). It is very easy to embed that code into your own binary:

func NewDefaultOptions() *Options {
    o := &Options{
        *options.NewSecureServingOptions(),
    }
    o.SecureServing.ServerCert.PairName = "pizza-crd-webhook"
    return o
}

type Options struct {
    SecureServing options.SecureServingOptions
}

type Config struct {
    SecureServing *server.SecureServingInfo
}

func (o *Options) AddFlags(fs *pflag.FlagSet) {
    o.SecureServing.AddFlags(fs)
}

func (o *Options) Config() (*Config, error) {
    err := o.SecureServing.MaybeDefaultWithSelfSignedCerts("0.0.0.0", nil, nil)
    if err != nil {
        return nil, err
    }

    c := &Config{}

    if err := o.SecureServing.ApplyTo(&c.SecureServing); err != nil {
        return nil, err
    }

    return c, nil
}

In the main function of the binary, this Options struct is instantiated and wired to a flag set:

opt := NewDefaultOptions()
fs := pflag.NewFlagSet("pizza-crd-webhook", pflag.ExitOnError)
globalflag.AddGlobalFlags(fs, "pizza-crd-webhook")
opt.AddFlags(fs)
if err := fs.Parse(os.Args); err != nil {
    panic(err)
}

// create runtime config
cfg, err := opt.Config()
if err != nil {
    panic(err)
}

stopCh := server.SetupSignalHandler()

...

// run server
restaurantInformers.Start(stopCh)
if doneCh, err := cfg.SecureServing.Serve(
    handlers.LoggingHandler(os.Stdout, mux),
    time.Second * 30, stopCh,
); err != nil {
    panic(err)
} else {
    <-doneCh
}

In place of the three dots, we set up the HTTP multiplexer with our three paths as follows:

// register handlers
restaurantInformers := restaurantinformers.NewSharedInformerFactory(
    clientset, time.Minute * 5,
)
mux := http.NewServeMux()
mux.Handle("/convert/v1beta1/pizza", http.HandlerFunc(conversion.Serve))
mux.Handle("/admit/v1beta1/pizza", http.HandlerFunc(admission.ServePizzaAdmit))
mux.Handle("/validate/v1beta1/pizza",
    http.HandlerFunc(admission.ServePizzaValidation(restaurantInformers)))
restaurantInformers.Start(stopCh)

As the pizza validation webhook at the path /validate/v1beta1/pizza has to know the existing topping objects in the cluster, we instantiate a shared informer factory for the restaurant.programming-kubernetes.info API group.

Now we’ll look at the actual conversion webhook implementation behind conversion.Serve. It is a normal Golang HTTP handler function, meaning it gets a request and a response writer as arguments.

The request body contains a ConversionReview object from the API group apiextensions.k8s.io/v1beta1. Hence, we have to first read the body from the request, and then decode the byte slice. We do this by using a deserializer from API Machinery:

func Serve(w http.ResponseWriter, req *http.Request) {
    // read body
    body, err := ioutil.ReadAll(req.Body)
    if err != nil {
        responsewriters.InternalError(w, req,
          fmt.Errorf("failed to read body: %v", err))
        return
    }

    // decode body as conversion review
    gv := apiextensionsv1beta1.SchemeGroupVersion
    reviewGVK := gv.WithKind("ConversionReview")
    obj, gvk, err := codecs.UniversalDeserializer().Decode(body, &reviewGVK,
        &apiextensionsv1beta1.ConversionReview{})
    if err != nil {
        responsewriters.InternalError(w, req,
          fmt.Errorf("failed to decode body: %v", err))
        return
    }
    review, ok := obj.(*apiextensionsv1beta1.ConversionReview)
    if !ok {
        responsewriters.InternalError(w, req,
          fmt.Errorf("unexpected GroupVersionKind: %s", gvk))
        return
    }
    if review.Request == nil {
        responsewriters.InternalError(w, req,
          fmt.Errorf("unexpected nil request"))
        return
    }

    ...
}

This code makes use of the codec factory codecs, which is derived from a scheme. This scheme has to include the types of apiextensions.k8s.io/v1beta1. We also add the types of our restaurant API group. The passed ConversionReview object will have our pizza type embedded in a runtime.RawExtension type—more about that in a second.

First let’s create our scheme and the codec factory:

import (
    apiextensionsv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
    "github.com/programming-kubernetes/pizza-crd/pkg/apis/restaurant/install"
    ...
)

var (
    scheme = runtime.NewScheme()
    codecs = serializer.NewCodecFactory(scheme)
)

func init() {
    utilruntime.Must(apiextensionsv1beta1.AddToScheme(scheme))
    install.Install(scheme)
}

A runtime.RawExtension is a wrapper for Kubernetes-like objects embedded in a field of another object. Its structure is actually very simple:

type RawExtension struct {
    // Raw is the underlying serialization of this object.
    Raw []byte `protobuf:"bytes,1,opt,name=raw"`
    // Object can hold a representation of this extension - useful for working
    // with versioned structs.
    Object Object `json:"-"`
}

In addition, runtime.RawExtension has special JSON and protobuf marshaling two methods. Moreover, there is special logic around the conversion to runtime.Object on the fly, when converting to internal types—that is, automatic encoding and decoding.

In this case of CRDs, we don’t have internal types, and therefore that conversion magic does not play a role. Only RawExtension.Raw is filled with a JSON byte slice of the pizza object sent to the webhook for conversion. Thus, we will have to decode this byte slice. Note again that one ConversionReview potentially carries a number of objects, such that we have to loop over all of them:

// convert objects
review.Response = &apiextensionsv1beta1.ConversionResponse{
    UID: review.Request.UID,
    Result:  metav1.Status{
        Status: metav1.StatusSuccess,
    },
}
var objs []runtime.Object
for _, in := range review.Request.Objects {
    if in.Object == nil {
        var err error
        in.Object, _, err = codecs.UniversalDeserializer().Decode(
            in.Raw, nil, nil,
        )
        if err != nil {
            review.Response.Result = metav1.Status{
                Message: err.Error(),
                Status:  metav1.StatusFailure,
            }
            break
        }
    }

    obj, err := convert(in.Object, review.Request.DesiredAPIVersion)
    if err != nil {
        review.Response.Result = metav1.Status{
            Message: err.Error(),
            Status:  metav1.StatusFailure,
        }
        break
    }
    objs = append(objs, obj)
}

The convert call does the actual conversion of in.Object, with the desired API version as the target version. Note here that we break the loop immediately when the first error occurs.

Finally, we set the Response field in the ConversionReview object and write it back as the response body of the request using API Machinery’s response writer, which again uses our codec factory to create a serializer:

if review.Response.Result.Status == metav1.StatusSuccess {
    for _, obj = range objs {
        review.Response.ConvertedObjects =
          append(review.Response.ConvertedObjects,
            runtime.RawExtension{Object: obj},
          )
    }
}

// write negotiated response
responsewriters.WriteObject(
    http.StatusOK, gvk.GroupVersion(), codecs, review, w, req,
)

Now, we have to implement the actual pizza conversion. After all this plumbing above, the conversion algorithm is the easiest part. It just checks that we actually got a pizza object of the known versions and then does the conversion from v1beta1 to v1alpha1 and vice versa:

func convert(in runtime.Object, apiVersion string) (runtime.Object, error) {
    switch in := in.(type) {
    case *v1alpha1.Pizza:
        if apiVersion != v1beta1.SchemeGroupVersion.String() {
            return nil, fmt.Errorf("cannot convert %s to %s",
              v1alpha1.SchemeGroupVersion, apiVersion)
        }
        klog.V(2).Infof("Converting %s/%s from %s to %s", in.Namespace, in.Name,
            v1alpha1.SchemeGroupVersion, apiVersion)

        out := &v1beta1.Pizza{
            TypeMeta: in.TypeMeta,
            ObjectMeta: in.ObjectMeta,
            Status: v1beta1.PizzaStatus{
                Cost: in.Status.Cost,
            },
        }
        out.TypeMeta.APIVersion = apiVersion

        idx := map[string]int{}
        for _, top := range in.Spec.Toppings {
            if i, duplicate := idx[top]; duplicate {
                out.Spec.Toppings[i].Quantity++
                continue
            }
            idx[top] = len(out.Spec.Toppings)
            out.Spec.Toppings = append(out.Spec.Toppings, v1beta1.PizzaTopping{
                Name: top,
                Quantity: 1,
            })
        }

        return out, nil

    case *v1beta1.Pizza:
        if apiVersion != v1alpha1.SchemeGroupVersion.String() {
            return nil, fmt.Errorf("cannot convert %s to %s",
              v1beta1.SchemeGroupVersion, apiVersion)
        }
        klog.V(2).Infof("Converting %s/%s from %s to %s",
          in.Namespace, in.Name, v1alpha1.SchemeGroupVersion, apiVersion)

        out := &v1alpha1.Pizza{
            TypeMeta: in.TypeMeta,
            ObjectMeta: in.ObjectMeta,
            Status: v1alpha1.PizzaStatus{
                Cost: in.Status.Cost,
            },
        }
        out.TypeMeta.APIVersion = apiVersion

        for i := range in.Spec.Toppings {
            for j := 0; j < in.Spec.Toppings[i].Quantity; j++ {
                out.Spec.Toppings = append(
                  out.Spec.Toppings, in.Spec.Toppings[i].Name)
            }
        }

        return out, nil

    default:
    }
    klog.V(2).Infof("Unknown type %T", in)
    return nil, fmt.Errorf("unknown type %T", in)
}

Note that in both directions of the conversion, we just copy TypeMeta and ObjectMeta, change the API version to the desired one, and then convert the toppings slice, which is actually the only part of the objects which structurally differs.

If there are more versions, another two-way conversion is necessary between all of them. Alternatively, of course, we could use a hub version as in aggregated API servers (see “Internal Types and Conversion”), instead of implementing conversions from and to all supported external versions.

Deploying the Conversion Webhook

We now want to deploy the conversion webhook. You can find all the manifests on GitHub.

Conversion webhooks for CRDs are launched in the cluster and put behind a service object, and that service object is referenced by the conversion webhook specification in the CRD manifest:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: pizzas.restaurant.programming-kubernetes.info
spec:
  ...
  conversion:
    strategy: Webhook
    webhookClientConfig:
      caBundle: BASE64-CA-BUNDLE
      service:
        namespace: pizza-crd
        name: webhook
        path: /convert/v1beta1/pizza

The CA bundle must match the serving certificate used by the webhook. In our example project, we use a Makefile to generate certificates using OpenSSL and plug them into the manifests using text replacement.

Note here that the Kubernetes API server assumes that the webhook supports all specified versions of the CRD. There is also only one such webhook possible per CRD. But as CRDs and conversion webhooks are usually owned by the same team, this should be enough.

Also note that the service port must be 443 in the current apiextensions.k8s.io/v1beta1 API. The service can map this, however, to any port used by the webhook pods. In our example, we map 443 to 8443, served by the webhook binary.

Seeing Conversion in Action

Now that we understand how the conversion webhook works and how it is wired into the cluster, let’s see it in action.

We assume you’ve checked out the example project. In addition, we assume that you have a cluster with webhook conversion enabled (either via feature gate in a 1.14 cluster or through a 1.15+ cluster, which has webhook conversion enabled by default). One way to get such a cluster is via the kind project, which provides support for Kubernetes 1.14.1 and a local kind-config.yaml file to enable the alpha feature gate for webhook conversion (“What Does Programming Kubernetes Mean?” linked a number of other options for development clusters):

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatchesJson6902:
- group: kubeadm.k8s.io
  version: v1beta1
  kind: ClusterConfiguration
  patch: |
    - op: add
      path: /apiServer/extraArgs
      value: {}
    - op: add
      path: /apiServer/extraArgs/feature-gates
      value: CustomResourceWebhookConversion=true

Then we can create a cluster:

$ kind create cluster --image kindest/node-images:v1.14.1 --config kind-config.yaml
$ export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"

Now we can deploy our manifests:

$ cd pizza-crd
$ cd manifest/deployment
$ make
$ kubectl create -f ns.yaml
$ kubectl create -f pizza-crd.yaml
$ kubectl create -f topping-crd.yaml
$ kubectl create -f sa.yaml
$ kubectl create -f rbac.yaml
$ kubectl create -f rbac-bind.yaml
$ kubectl create -f service.yaml
$ kubectl create -f serving-cert-secret.yaml
$ kubectl create -f deployment.yaml

These manifests contain the following files:

ns.yaml

Creates the pizza-crd namespace.

pizza-crd.yaml

Specifies the pizza resource in the restaurant.programming-kubernetes.info API group, with the v1alpha1 and v1beta1 versions, and the webhook conversion configuration as shown previously.

topping-crd.yaml

Specifies the toppings CR in the same API group, but only in the v1alpha1 version.

sa.yaml

Introduces the webhook service account.

rbac.yaml

Defines a role to read, list, and watch toppings.

rbac-bind.yaml

Binds the earlier RBAC role to the webhook service account.

service.yaml

Defines the webhook services, mapping port 443 to 8443 of the webhook pods.

serving-cert-secret.yaml

Contains the serving certificate and private key to be used by the webhook pods. The certificate is also used directly as the CA bundle in the preceding pizza CRD manifest.

deployment.yaml

Launches webhook pods, passing --tls-cert-file and --tls-private-key the serving certificate secret.

After this we can create a margherita pizza finally:

$ cat  ../examples/margherita-pizza.yaml
apiVersion: restaurant.programming-kubernetes.info/v1alpha1
kind: Pizza
metadata:
  name: margherita
spec:
  toppings:
  - mozzarella
  - tomato
$ kubectl create ../examples/margherita-pizza.yaml
pizza.restaurant.programming-kubernetes.info/margherita created

Now, with the conversion webhook in place, we can retrieve the same object in both versions. First explicitly in the v1alpha1 version:

$ kubectl get pizzas.v1alpha1.restaurant.programming-kubernetes.info 
    margherita -o yaml
apiVersion: restaurant.programming-kubernetes.info/v1alpha1
kind: Pizza
metadata:
  creationTimestamp: "2019-04-14T21:41:39Z"
  generation: 1
  name: margherita
  namespace: pizza-crd
  resourceVersion: "18296"
  pizzas/margherita
  uid: 15c1c06a-5efe-11e9-9230-0242f24ba99c
spec:
  toppings:
  - mozzarella
  - tomato
status: {}

Then the same object as v1beta1 shows the different toppings structure:

$ kubectl get pizzas.v1beta1.restaurant.programming-kubernetes.info 
    margherita -o yaml
apiVersion: restaurant.programming-kubernetes.info/v1beta1
kind: Pizza
metadata:
  creationTimestamp: "2019-04-14T21:41:39Z"
  generation: 1
  name: margherita
  namespace: pizza-crd
  resourceVersion: "18296"
  pizzas/margherita
  uid: 15c1c06a-5efe-11e9-9230-0242f24ba99c
spec:
  toppings:
  - name: mozzarella
    quantity: 1
  - name: tomato
    quantity: 1
status: {}

Meanwhile, in the log of the webhook pod we see this conversion call:

I0414 21:46:28.639707       1 convert.go:35] Converting pizza-crd/margherita
  from restaurant.programming-kubernetes.info/v1alpha1
  to restaurant.programming-kubernetes.info/v1beta1
10.32.0.1 - - [14/Apr/2019:21:46:28 +0000]
  "POST /convert/v1beta1/pizza?timeout=30s HTTP/2.0" 200 968

Hence, the webhook is doing its job as expected.

Admission Requirements in the Restaurant Example

The restaurant example uses admission for multiple things:

• spec.toppings defaults if it is nil or empty to mozzarella, tomato, and salami.

• Unknown fields should be dropped from the CR JSON and not persisted in etcd.

• spec.toppings must contain only toppings that have a corresponding topping object.

The first two use cases are mutating; the third use case is purely validating. Therefore, we will use one mutating webhook and one validating webhook to implement those steps.

Admission Webhook Architecture

Admission webhooks are structurally very similar to the conversion webhooks we saw earlier in the chapter.

They are deployed in the cluster, put behind a service mapping port 443 to some port of the pods, and called using a review object, AdmissionReview in the API group admission.k8s.io/v1beta1:

---
// AdmissionReview describes an admission review request/response.
type AdmissionReview struct {
    metav1.TypeMeta `json:",inline"`
    // Request describes the attributes for the admission request.
    // +optional
    Request *AdmissionRequest `json:"request,omitempty"`
    // Response describes the attributes for the admission response.
    // +optional
    Response *AdmissionResponse `json:"response,omitempty"`
}
---

The AdmissionRequest contains all the information we are used to from the admission attributes (see “Implementation”):

// AdmissionRequest describes the admission.Attributes for the admission request.
type AdmissionRequest struct {
    // UID is an identifier for the individual request/response. It allows us to
    // distinguish instances of requests which are otherwise identical (parallel
    // requests, requests when earlier requests did not modify etc). The UID is
    // meant to track the round trip (request/response) between the KAS and the
    // WebHook, not the user request. It is suitable for correlating log entries
    // between the webhook and apiserver, for either auditing or debugging.
    UID types.UID `json:"uid"`
    // Kind is the type of object being manipulated.  For example: Pod
    Kind metav1.GroupVersionKind `json:"kind"`
    // Resource is the name of the resource being requested.  This is not the
    // kind.  For example: pods
    Resource metav1.GroupVersionResource `json:"resource"`
    // SubResource is the name of the subresource being requested.  This is a
    // different resource, scoped to the parent resource, but it may have a
    // different kind. For instance, /pods has the resource "pods" and the kind
    // "Pod", while /pods/foo/status has the resource "pods", the sub resource
    // "status", and the kind "Pod" (because status operates on pods). The
    // binding resource for a pod though may be /pods/foo/binding, which has
    // resource "pods", subresource "binding", and kind "Binding".
    // +optional
    SubResource string `json:"subResource,omitempty"`
    // Name is the name of the object as presented in the request.  On a CREATE
    // operation, the client may omit name and rely on the server to generate
    // the name.  If that is the case, this method will return the empty string.
    // +optional
    Name string `json:"name,omitempty"`
    // Namespace is the namespace associated with the request (if any).
    // +optional
    Namespace string `json:"namespace,omitempty"`
    // Operation is the operation being performed
    Operation Operation `json:"operation"`
    // UserInfo is information about the requesting user
    UserInfo authenticationv1.UserInfo `json:"userInfo"`
    // Object is the object from the incoming request prior to default values
    // being applied
    // +optional
    Object runtime.RawExtension `json:"object,omitempty"`
    // OldObject is the existing object. Only populated for UPDATE requests.
    // +optional
    OldObject runtime.RawExtension `json:"oldObject,omitempty"`
    // DryRun indicates that modifications will definitely not be persisted
    // for this request.
    // Defaults to false.
    // +optional
    DryRun *bool `json:"dryRun,omitempty"`
}

The same AdmissionReview object is used for both mutating and validating admission webhooks. The only difference is that in the mutating case, the AdmissionResponse can have a field patch and patchType, to be applied inside the Kubernetes API server after the webhook response has been received there. In the validating case, these two fields are kept empty on response.

The most important field for our purposes here is the Object field, which—as in the preceding conversion webhook—uses the runtime.RawExtension type to store a pizza object.

We also get the old object for update requests and could, say, check for fields that are meant to be read-only but are changed in a request. We don’t do this here in our example. But you will encounter many cases in Kubernetes where such logic is implemented—for example, for most fields of a pod, as you can’t change the command of a pod after it is created.

The patch returned by the mutating webhook must be of type JSON Patch (see RFC 6902) in Kubernetes 1.14. This patch describes how the object should be modified to fulfill the required invariant.

Note that it is best practice to validate every mutating webhook change in a validating webhook at the very end, at least if those enforced properties are significant for the behavior. Imagine some other mutating webhook touches the same fields in an object. Then you cannot be sure that the mutating changes will survive until the end of the mutating admission chain.

There is no order currently in mutating webhooks other than alphabetic order. There are ongoing discussions to change this in one way or another in the future.

For validating webhooks the order does not matter, obviously, and the Kubernetes API server even calls validating webhooks in parallel to reduce latency. In contrast, mutating webhooks add latency to every request that passes through them, as they are called sequentially.

Common latencies—of course heavily depending on the environment—are around 100ms. So running many webhooks in sequence leads to considerable latencies that the user will experience when creating or updating objects.

Registering Admission Webhooks

Admission webhooks are not registered in the CRD manifest. The reason is that they apply not only to CRDs, but to any kind of resource. You can even add custom admission webhooks to standard Kubernetes resources.

Instead there are registration objects: MutatingWebhookRegistration and ValidatingWebhookRegistration. They differ only in the kind name:

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: restaurant.programming-kubernetes.info
webhooks:
- name: restaurant.programming-kubernetes.info
  failurePolicy: Fail
  sideEffects: None
  admissionReviewVersions:
  - v1beta1
  rules:
  - apiGroups:
    - "restaurant.programming-kubernetes.info"
    apiVersions:
    - v1alpha1
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - pizzas
  clientConfig:
    service:
      namespace: pizza-crd
      name: webhook
      path: /admit/v1beta1/pizza
    caBundle: CA-BUNDLE

This registers our pizza-crd webhook from the beginning of the chapter for mutating admission for our two versions of the resource pizza, the API group restaurant.programming-kubernetes.info, and the HTTP verbs CREATE and UPDATE (which includes patches as well).

There are further ways in webhook configurations to restrict the matching resources—for example, a namespace selector (to exclude, e.g., a control plane namespace to avoid bootstrapping issues) and more advanced resource patterns with wildcards and subresources.

Last but not least is a failure mode, which can be either Fail or Ignore. It specifies what to do if the webhook cannot be reached or fails for other reasons.

Implementing an Admission Webhook

With the work we’ve done on the conversion webhook in the beginning of the chapter, it is not hard to add admission capabilities. We also saw that the paths /admit/v1beta1/pizza and /validate/v1beta1/pizza are registered in the main function of the pizza-crd-webhook binary:

mux.Handle("/admit/v1beta1/pizza", http.HandlerFunc(admission.ServePizzaAdmit))
mux.Handle("/validate/v1beta1/pizza", http.HandlerFunc(
admission.ServePizzaValidation(restaurantInformers)))

The first part of the two HTTP handler implementations looks nearly the same as for the conversion webhook:

func ServePizzaAdmit(w http.ResponseWriter, req *http.Request) {
    // read body
    body, err := ioutil.ReadAll(req.Body)
    if err != nil {
        responsewriters.InternalError(w, req,
          fmt.Errorf("failed to read body: %v", err))
        return
    }

    // decode body as admission review
    reviewGVK := admissionv1beta1.SchemeGroupVersion.WithKind("AdmissionReview")
    decoder := codecs.UniversalDeserializer()
    into := &admissionv1beta1.AdmissionReview{}
    obj, gvk, err := decoder.Decode(body, &reviewGVK, into)
    if err != nil {
        responsewriters.InternalError(w, req,
          fmt.Errorf("failed to decode body: %v", err))
        return
    }
    review, ok := obj.(*admissionv1beta1.AdmissionReview)
    if !ok {
        responsewriters.InternalError(w, req,
          fmt.Errorf("unexpected GroupVersionKind: %s", gvk))
        return
    }
    if review.Request == nil {
        responsewriters.InternalError(w, req,
          fmt.Errorf("unexpected nil request"))
        return
    }

    ...
}

In the case of the validating webhook, we have to wire the informer (used to check that toppings exist in the cluster). We return an internal error as long as the informer is not synced. An informer that is not synced has incomplete data, so the toppings might not be known and the pizza would be rejected although they are valid:

func ServePizzaValidation(informers restaurantinformers.SharedInformerFactory)
    func (http.ResponseWriter, *http.Request)
{
    toppingInformer := informers.Restaurant().V1alpha1().Toppings().Informer()
    toppingLister := informers.Restaurant().V1alpha1().Toppings().Lister()

    return func(w http.ResponseWriter, req *http.Request) {
        if !toppingInformer.HasSynced() {
            responsewriters.InternalError(w, req,
              fmt.Errorf("informers not ready"))
            return
        }

        // read body
        body, err := ioutil.ReadAll(req.Body)
        if err != nil {
            responsewriters.InternalError(w, req,
              fmt.Errorf("failed to read body: %v", err))
            return
        }

        // decode body as admission review
        gv := admissionv1beta1.SchemeGroupVersion
        reviewGVK := gv.WithKind("AdmissionReview")
        obj, gvk, err := codecs.UniversalDeserializer().Decode(body, &reviewGVK,
            &admissionv1beta1.AdmissionReview{})
        if err != nil {
            responsewriters.InternalError(w, req,
              fmt.Errorf("failed to decode body: %v", err))
            return
        }
        review, ok := obj.(*admissionv1beta1.AdmissionReview)
        if !ok {
            responsewriters.InternalError(w, req,
              fmt.Errorf("unexpected GroupVersionKind: %s", gvk))
            return
        }
        if review.Request == nil {
            responsewriters.InternalError(w, req,
              fmt.Errorf("unexpected nil request"))
            return
        }

        ...
    }
}

As in the webhook conversion case, we have set up the scheme and the codec factory with the admission API group and our restaurant API group:

var (
    scheme = runtime.NewScheme()
    codecs = serializer.NewCodecFactory(scheme)
)

func init() {
    utilruntime.Must(admissionv1beta1.AddToScheme(scheme))
    install.Install(scheme)
}

With these two, we decode the embedded pizza object (this time only one, no slice) from the AdmissionReview:

// decode object
if review.Request.Object.Object == nil {
    var err error
    review.Request.Object.Object, _, err =
      codecs.UniversalDeserializer().Decode(review.Request.Object.Raw, nil, nil)
    if err != nil {
        review.Response.Result = &metav1.Status{
            Message: err.Error(),
            Status:  metav1.StatusFailure,
        }
        responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(),
          codecs, review, w, req)
        return
    }
}

Then we can do the actual mutating admission (the defaulting of spec.toppings for both API versions):

orig := review.Request.Object.Raw
var bs []byte
switch pizza := review.Request.Object.Object.(type) {
case *v1alpha1.Pizza:
    // default toppings
    if len(pizza.Spec.Toppings) == 0 {
        pizza.Spec.Toppings = []string{"tomato", "mozzarella", "salami"}
    }
    bs, err = json.Marshal(pizza)
    if err != nil {
        responsewriters.InternalError(w, req,
          fmt.Errorf"unexpected encoding error: %v", err))
        return
    }

case *v1beta1.Pizza:
    // default toppings
    if len(pizza.Spec.Toppings) == 0 {
        pizza.Spec.Toppings = []v1beta1.PizzaTopping{
            {"tomato", 1},
            {"mozzarella", 1},
            {"salami", 1},
        }
    }
    bs, err = json.Marshal(pizza)
    if err != nil {
        responsewriters.InternalError(w, req,
          fmt.Errorf("unexpected encoding error: %v", err))
        return
    }

default:
    review.Response.Result = &metav1.Status{
        Message: fmt.Sprintf("unexpected type %T", review.Request.Object.Object),
        Status:  metav1.StatusFailure,
    }
    responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(),
      codecs, review, w, req)
    return
}

Alternatively, we could use the conversion algorithms from the conversion webhook and then implement defaulting only for one of the versions. Both approaches are possible, and which one makes more sense depends on the context. Here, the defaulting is simple enough to implement it twice.

The final step is to compute the patch—the difference between the original object (stored in orig as JSON) and the new defaulted one:

// compare original and defaulted version
ops, err := jsonpatch.CreatePatch(orig, bs)
if err != nil {
    responsewriters.InternalError(w, req,
        fmt.Errorf("unexpected diff error: %v", err))
    return
}
review.Response.Patch, err = json.Marshal(ops)
if err != nil {
    responsewriters.InternalError(w, req,
    fmt.Errorf("unexpected patch encoding error: %v", err))
    return
}
typ := admissionv1beta1.PatchTypeJSONPatch
review.Response.PatchType = &typ
review.Response.Allowed = true

We use the JSON-Patch library (a fork of Matt Baird’s with critical fixes) to derive the patch from the original object orig and the modified object bs, both passed as JSON byte slices. Alternatively, we could operate directly on untyped JSON data and create the JSON-Patch manually. Again, it depends on the context. Using a diff library is convenient.

Then, as in the webhook conversion, we conclude by writing the response to the response writer, using the codec factory created previously:

responsewriters.WriteObject(
    http.StatusOK, gvk.GroupVersion(), codecs, review, w, req,
)

The validating webhook is very similar, but it uses the toppings lister from the shared informer to check for the existence of the topping objects:

switch pizza := review.Request.Object.Object.(type) {
case *v1alpha1.Pizza:
    for _, topping := range pizza.Spec.Toppings {
        _, err := toppingLister.Get(topping)
        if err != nil && !errors.IsNotFound(err) {
            responsewriters.InternalError(w, req,
              fmt.Errorf("failed to lookup topping %q: %v", topping, err))
            return
        } else if errors.IsNotFound(err) {
            review.Response.Result = &metav1.Status{
                Message: fmt.Sprintf("topping %q not known", topping),
                Status:  metav1.StatusFailure,
            }
            responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(),
              codecs, review, w, req)
            return
        }
    }
    review.Response.Allowed = true
case *v1beta1.Pizza:
    for _, topping := range pizza.Spec.Toppings {
        _, err := toppingLister.Get(topping.Name)
        if err != nil && !errors.IsNotFound(err) {
            responsewriters.InternalError(w, req,
              fmt.Errorf("failed to lookup topping %q: %v", topping, err))
            return
        } else if errors.IsNotFound(err) {
            review.Response.Result = &metav1.Status{
                Message: fmt.Sprintf("topping %q not known", topping),
                Status:  metav1.StatusFailure,
            }
            responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(),
              codecs, review, w, req)
            return
        }
    }
    review.Response.Allowed = true
default:
    review.Response.Result = &metav1.Status{
        Message: fmt.Sprintf("unexpected type %T", review.Request.Object.Object),
        Status:  metav1.StatusFailure,
    }
}
responsewriters.WriteObject(http.StatusOK, gvk.GroupVersion(),
      codecs, review, w, req)

Admission Webhook in Action

We deploy the two admission webhooks by creating the two registration objects in the cluster:

$ kubectl create -f validatingadmissionregistration.yaml
$ kubectl create -f mutatingadmissionregistration.yaml

After this, we can’t create pizzas with unknown toppings anymore:

$ kubectl create -f ../examples/margherita-pizza.yaml
Error from server: error when creating "../examples/margherita-pizza.yaml":
  admission webhook "restaurant.programming-kubernetes.info" denied the request:
    topping "tomato" not known

Meanwhile, in the webhook log we see:

I0414 22:45:46.873541       1 pizzamutation.go:115] Defaulting pizza-crd/ in
  version admission.k8s.io/v1beta1, Kind=AdmissionReview
10.32.0.1 - - [14/Apr/2019:22:45:46 +0000]
  "POST /admit/v1beta1/pizza?timeout=30s HTTP/2.0" 200 871
10.32.0.1 - - [14/Apr/2019:22:45:46 +0000]
  "POST /validate/v1beta1/pizza?timeout=30s HTTP/2.0" 200 956

After creating the toppings in the example folder, we can create the margherita pizza again:

$ kubectl create -f ../examples/topping-tomato.yaml
$ kubectl create -f ../examples/topping-salami.yaml
$ kubectl create -f ../examples/topping-mozzarella.yaml
$ kubectl create -f ../examples/margherita-pizza.yaml
pizza.restaurant.programming-kubernetes.info/margherita created

Last but not least, let’s check that defaulting works as expected. We want to create an empty pizza:

apiVersion: restaurant.programming-kubernetes.info/v1alpha1
kind: Pizza
metadata:
  name: salami
spec:

This is supposed to be defaulted to a salami pizza, and it is:

$ kubectl create -f ../examples/empty-pizza.yaml
pizza.restaurant.programming-kubernetes.info/salami created
$ kubectl get pizza salami -o yaml
apiVersion: restaurant.programming-kubernetes.info/v1beta1
kind: Pizza
metadata:
  creationTimestamp: "2019-04-14T22:49:40Z"
  generation: 1
  name: salami
  namespace: pizza-crd
  resourceVersion: "23227"
  uid: 962e2dda-5f07-11e9-9230-0242f24ba99c
spec:
  toppings:
  - name: tomato
    quantity: 1
  - name: mozzarella
    quantity: 1
  - name: salami
    quantity: 1
status: {}

Voilà, a salami pizza with all the toppings that we expect. Enjoy!

Before concluding the chapter, we want to look toward an apiextensions.k8s.io/v1 API group version (i.e., nonbeta, general availability) of CRDs—namely, the introduction of structural schemas.

Pruning Versus Preserving Unknown Fields

CRDs traditionally store any (possibly validated) JSON as is in etcd. This means that unspecified fields (if there is an OpenAPI v3 validation schema at all) will be persisted. This is in contrast to native Kubernetes resources like a pod. If the user specifies a field spec.randomField, this will be accepted by the API server HTTPS endpoint but dropped (we call this pruning) before writing that pod to etcd.

If a structural OpenAPI v3 validation schema is defined (either in the global spec.validation.openAPIV3Schema or for each version), we can enable pruning (which drops unspecified fields on creation and on update) by setting spec.preserveUnknownFields to false.

Let’s look at the cnat example.² With a Kubernetes 1.15 cluster at hand, we enable pruning:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ats.cnat.programming-kubernetes.info
spec:
  ...
  preserveUnknownFields: false

Then we try to create an instance with an unknown field:

apiVersion: cnat.programming-kubernetes.info/v1alpha1
kind: At
metadata:
  name: example-at
spec:
  schedule: "2019-07-03T02:00:00Z"
  command: echo "Hello, world!"
  someGarbage: 42

If we retrieve this object with kubectl get at example-at, we see that the someGarbage value is dropped:

apiVersion: cnat.programming-kubernetes.info/v1alpha1
kind: At
metadata:
  name: example-at
spec:
  schedule: "2019-07-03T02:00:00Z"
  command: echo "Hello, world!"

We say that someGarbage has been pruned.

As of Kubernetes 1.15, pruning is available in apiextensions/v1beta1, but it defaults to off; that is, spec.preserveUnknownFields defaults to true. In apiextensions/v1, no new CRD with spec.preserveUnknownFields: true will be allowed to be created.

Controlling Pruning

With spec.preserveUnknownField: false in the CRD, pruning is enabled for all CRs of that type and in all versions. It is possible, though, to opt out of pruning for a JSON subtree via x-kubernetes-preserve-unknown-fields: true in the OpenAPI v3 validation schema:

type: object
properties:
  json:
    x-kubernetes-preserve-unknown-fields: true

The field json can store any JSON value, without anything being pruned.

It is possible to partially specify the permitted JSON:

type: object
properties:
  json:
    x-kubernetes-preserve-unknown-fields: true
    type: object
    description: this is arbitrary JSON

With this approach, only object type values are allowed.

Pruning is enabled again for each specified property (or additionalProperties):

type: object
properties:
  json:
    x-kubernetes-preserve-unknown-fields: true
    type: object
    properties:
      spec:
        type: object
        properties:
          foo:
            type: string
          bar:
            type: string

With this, the value:

json:
  spec:
    foo: abc
    bar: def
    something: x
  status:
    something: x

will be pruned to:

json:
  spec:
    foo: abc
    bar: def
  status:
    something: x

This means that the something field in the specified spec object is pruned (because “spec” is specified), but everything outside is not. status is not specified such that status.something is not pruned.

IntOrString and RawExtensions

There are situations where structural schemas are not expressive enough. One of those is a polymorphic field—one that can be of different types. We know IntOrString from native Kubernetes API types.

It is possible to have IntOrString in CRDs using the x-kubernetes-int-or-string: true directive inside the schema. Similarly, runtime.RawExtensions can be declared using the x-kubernetes-embedded-object: true.

For example:

type: object
properties:
  intorstr:
    type: object
    x-kubernetes-int-or-string: true
  embedded:
    x-kubernetes-embedded-object: true
    x-kubernetes-preserve-unknown-fields: true

This declares:

• A field called intorstr that holds either an integer or a string

• A field called embedded that holds a Kubernetes-like object such as a complete pod specification

Refer to the official CRD documentation for all the details about these directives.

The last topic we want to talk about that depends on structural schemas is defaulting.

Default Values

In native Kubernetes types, it is common to default certain values. Defaulting used to be possible for CRDs only by way of mutating admission webhooks (see “Admission Webhooks”). As of Kubernetes 1.15, however, defaulting support is added (see the design document) to CRDs directly via the OpenAPI v3 schema described in the previous section.

In order to default certain fields, just specify the default value via the default keyword in the OpenAPI v3 schema. This is very useful when you are adding new fields to a type.

Starting with the schema of the cnat example from “Validating Custom Resources”, let’s assume we want to make the container image customizable, but default to a busybox image. For that we add the image field of string type to the OpenAPI v3 schema and set the default to busybox:

type: object
properties:
  apiVersion:
    type: string
  kind:
    type: string
  metadata:
    type: object
  spec:
    type: object
    properties:
      schedule:
        type: string
        pattern: "^d{4}-([0]d|1[0-2])-([0-2]d|3[01])..."
      command:
        type: string
      image:
        type: string
        default: "busybox"
    required:
    - schedule
    - command
  status:
    type: object
    properties:
      phase:
        type: string
required:
- metadata
- apiVersion
- kind
- spec

If the user creates an instance without specifying the image, the value is automatically set:

apiVersion: cnat.programming-kubernetes.info/v1alpha1
kind: At
metadata:
  name: example-at
spec:
  schedule: "2019-07-03T02:00:00Z"
  command: echo "hello world!"

On creation, this turns automatically into:

apiVersion: cnat.programming-kubernetes.info/v1alpha1
kind: At
metadata:
  name: example-at
spec:
  schedule: "2019-07-03T02:00:00Z"
  command: echo "hello world!"
  image: busybox

This looks super convenient and significantly improves the user experience of CRDs. What’s more, all old objects persisted in etcd will automatically inherit the new field when read from the API server.³

Note that persisted objects in etcd will not be rewritten (i.e., migrated automatically). In other words, on read the default values are only added on the fly and are only persisted when the object is updated for another reason.