The Client Library

The Kubernetes programming interface in Go mainly consists of the k8s.io/client-go library (for brevity we will just call it client-go going forward). client-go is a typical web service client library that supports all API types that are officially part of Kubernetes. It can be used to execute the usual REST verbs:

• Create

• Get

• List

• Update

• Delete

• Patch

Each of these REST verbs are implemented using the “The HTTP Interface of the API Server”. Furthermore, the verb Watch is supported, which is special for Kubernetes-like APIs, and one of the main differentiators compared to other APIs.

client-go is available on GitHub (see Figure 3-1), and used in Go code with the k8s.io/client-go package name. It is shipped in parallel to Kubernetes itself; that is, for each Kubernetes 1.x.y release, there is a client-go release with a matching tag kubernetes-1.x.y.

Figure 3-1. The client-go repository on GitHub

In addition, there is a semantic versioning scheme. For example, client-go 9.0.0 matches the Kubernetes 1.12 release, client-go 10.0.0 matches Kubernetes 1.13, and so on. There may be more fine-grained releases in the future. Besides the client code for Kubernetes API objects, client-go also contains a lot of generic library code. This is also used for user-defined API objects in Chapter 4. See Figure 3-1 for a list of packages.

While all packages have their use, most of your code that speaks to Kubernetes APIs will use tools/clientcmd/ to set up a client from a kubeconfig file and kubernetes/ for the actual Kubernetes API clients. We will see code doing this very soon. Before that, let’s finish a quick walk through with other relevant repositories and packages.

Kubernetes API Types

As we have seen, client-go holds the client interfaces. The Kubernetes API Go types for objects like pods, services, and deployments are located in their own repository. It is accessed as k8s.io/api in Go code.

Pods are part of the legacy API group (often also called the “core” group) version v1. Hence, the Pod Go type is found in k8s.io/api/core/v1, and similarly for all other API types in Kubernetes. See Figure 3-2 for a list of packages, most of which correspond to Kubernetes API groups and their versions.

The actual Go types are contained in a types.go file (e.g., k8s.io/api/core/v1/types.go). In addition, there are other files, most of them automatically generated by a code generator.

Figure 3-2. The API repository on GitHub

API Machinery

Last but not least, there is a third repository called API Machinery, which is used as k8s.io/apimachinery in Go. It includes all the generic building blocks to implement a Kubernetes-like API. API Machinery is not restricted to container management, so, for example, it could be used to build APIs for an online shop or any other business-specific domain.

Nevertheless, you’ll meet a lot of API Machinery packages in Kubernetes-native Go code. An important one is k8s.io/apimachinery/pkg/apis/meta/v1. It contains many of the generic API types such as ObjectMeta, TypeMeta, GetOptions, and ListOptions (see Figure 3-3).

Figure 3-3. The API Machinery repository on GitHub

Creating and Using a Client

Now we know all the building blocks to create a Kubernetes client object, which means we can access resources in a Kubernetes cluster. Assuming you have access to a cluster in your local environment (i.e., kubectl is properly set up and credentials are configured), the following code illustrates how you can use client-go in a Go project:

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/kubernetes"
)

kubeconfig = flag.String("kubeconfig", "~/.kube/config", "kubeconfig file")
flag.Parse()
config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
clientset, err := kubernetes.NewForConfig(config)

pod, err := clientset.CoreV1().Pods("book").Get("example", metav1.GetOptions{})

The code imports the meta/v1 package to get access to metav1.GetOptions. Furthermore, it imports clientcmd from client-go in order to read and parse the kubeconfig (i.e., the client configuration with server name, credentials, etc.). Then it imports the client-go kubernetes package with the client sets for Kubernetes resources.

The default location for the kubeconfig file is in .kube/config in the user’s home directory. This is also where kubectl gets the credentials for the Kubernetes clusters.

That kubeconfig is then read and parsed using clientcmd.BuildConfigFromFlags. We omitted the mandatory error handling throughout this code, but the err variable would normally contain, for example, the syntax error if a kubeconfig is not well formed. As syntax errors are common in Go code, such an error ought to be checked for properly, like so:

config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
if err != nil {
    fmt.Printf("The kubeconfig cannot be loaded: %v
", err
    os.Exit(1)
}

From clientcmd.BuildConfigFromFlags we get a rest.Config, which you can find in the k8s.io/client-go/rest package). This is passed to kubernetes.NewForConfig in order to create the actual Kubernetes client set. It’s called a client set because it contains multiple clients for all native Kubernetes resources.

When running a binary inside of a pod in a cluster, the kubelet will automatically mount a service account into the container at /var/run/secrets/kubernetes.io/serviceaccount. It replaces the kubeconfig file just mentioned and can easily be turned into a rest.Config via the rest.InClusterConfig() method. You’ll often find the following combination of rest.InClusterConfig() and clientcmd.BuildConfigFromFlags(), including support for the KUBECONFIG environment variable:

config, err := rest.InClusterConfig()
if err != nil {
    // fallback to kubeconfig
    kubeconfig := filepath.Join("~", ".kube", "config")
    if envvar := os.Getenv("KUBECONFIG"); len(envvar) >0 {
        kubeconfig = envvar
    }
    config, err = clientcmd.BuildConfigFromFlags("", kubeconfig)
    if err != nil {
        fmt.Printf("The kubeconfig cannot be loaded: %v
", err
        os.Exit(1)
    }
}

In the following example code we select the core group in v1 with clientset.CoreV1() and then access the pod "example" in the "book" namespace:

 pod, err := clientset.CoreV1().Pods("book").Get("example", metav1.GetOptions{})

Note that only the last function call, Get, actually accesses the server. Both CoreV1 and Pods select the client and set the namespace only for the following Get call (this is often called the builder pattern, in this case to build a request).

The Get call sends an HTTP GET request to /api/v1/namespaces/book/pods/example on the server, which is set in the kubeconfig. If the Kubernetes API server answers with HTTP code 200, the body of the response will carry the encoded pod objects, either as JSON—which is the default wire format of client-go—or as protocol buffers.

cfg, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
cfg.AcceptContentTypes = "application/vnd.kubernetes.protobuf,
                          application/json"
cfg.ContentType = "application/vnd.kubernetes.protobuf"
clientset, err := kubernetes.NewForConfig(cfg)

Versioning and Compatibility

Kubernetes APIs are versioned. We have seen in the previous section that pods are in v1 of the core group. The core group actually exists in only one version today. There are other groups, though—for example, the apps group, which exists in v1, v1beta2, and v1beta1 (as of this writing). If you look into the k8s.io/api/apps package, you will find all the API objects of these versions. In the k8s.io/client-go/kubernetes/typed/apps package, you’ll see the client implementations for all of these versions.

All of this is only the client side. It does not say anything about the Kubernetes cluster and its API server. Using a client with a version of an API group that the API server does not support will fail. Clients are hardcoded to a version, and the application developer has to select the right API group version in order to speak to the cluster at hand. See “API Versions and Compatibility Guarantees” for more on API group compatibility guarantees.

A second aspect of compatibility is the meta API features of the API server that client-go is speaking to. For example, there are option structs for CRUD verbs, like CreateOptions, GetOptions, UpdateOptions, and DeleteOptions. Another important one is ObjectMeta (discussed in detail in “ObjectMeta”), which is part of every kind. All of these are frequently extended with new features; we usually call them API machinery features. In the Go documentation of their fields, comments specify when features are considered alpha or beta. The same API compatibility guarantees apply as for any other API fields.

In the example that follows, the DeleteOptions struct is defined in the package k8s.io/apimachinery/pkg/apis/meta/v1/types.go:

// DeleteOptions may be provided when deleting an API object.
type DeleteOptions struct {
    TypeMeta `json:",inline"`

    GracePeriodSeconds *int64 `json:"gracePeriodSeconds,omitempty"`
    Preconditions *Preconditions `json:"preconditions,omitempty"`
    OrphanDependents *bool `json:"orphanDependents,omitempty"`
    PropagationPolicy *DeletionPropagation `json:"propagationPolicy,omitempty"`

    // When present, indicates that modifications should not be
    // persisted. An invalid or unrecognized dryRun directive will
    // result in an error response and no further processing of the
    // request. Valid values are:
    // - All: all dry run stages will be processed
    // +optional
    DryRun []string `json:"dryRun,omitempty" protobuf:"bytes,5,rep,name=dryRun"`
}

The last field, DryRun, was added in Kubernetes 1.12 as alpha and in 1.13 as beta (enabled by default). It is not understood by the API server in earlier versions. Depending on the feature, passing such an option might simply be ignored or even rejected. So it is important to have a client-go version that is not too far off from the cluster version.

The formally guaranteed support matrix between cluster and client-go versions is published in the client-go README (see Table 3-1).

• ✓: both client-go and the Kubernetes version have the same features and the same API group versions.

• +: client-go has features or API group versions that may be absent from the Kubernetes cluster. This may be because of added functionality in client-go or because Kubernetes removed old, deprecated functionality. However, everything they have in common (i.e., most APIs) will work.

• –: client-go is knowingly incompatible with the Kubernetes cluster.

The takeaway from Table 3-1 is that the client-go library is supported with its corresponding cluster version. In case of version skew, developers have to carefully consider which features and which API groups they use and whether these are supported in the cluster version the application speaks to.

In Table 3-1, the client-go versions are listed. We briefly mentioned in “The Client Library” that client-go uses semantic versioning (semver) formally, though by increasing the major version of client-go each time the minor version of Kubernetes (the 13 in 1.13.2) is increased. With client-go 1.0 being released for Kubernetes 1.4, we are now at client-go 12.0 (at the time of this writing) for Kubernetes 1.15.

This semver applies only to client-go itself, not to API Machinery or the API repository. Instead, the latter are tagged using Kubernetes versions, as seen in Figure 3-4. See “Vendoring” to see what this means for vendoring k8s.io/client-go, k8s.io/apimachinery, and k8s.io/api in your project.

Figure 3-4. client-go versioning

API Versions and Compatibility Guarantees

As seen in the previous section, the selection of the right API group versions may be crucial if you target different cluster versions with your code. Kubernetes versions all API groups. A common Kubernetes-style versioning scheme is used, which consists of alpha, beta, and GA (general availability) versions.

The pattern is:

• v1alpha1, v1alpha2, v2alpha1, and so on are called alpha versions and considered unstable. This means:

  • They might go away or change at any time, in any incompatible way.

  • Data might be dropped, get lost, or become inaccessible from Kubernetes version to version.

  • They are often disabled by default, if the administrator does not opt in manually.

• v1beta1, v1beta2, v2beta1, and so on, are called beta versions. They are on the way to stability, which means:

  • They will still exist for at least one Kubernetes release in parallel to the corresponding stable API version.

  • They will usually not change in incompatible ways, but there is no strict guarantee of that.

  • Objects stored in a beta version will not be dropped or become inaccessible.

  • Beta versions are often enabled in clusters by default. But this might depend on the Kubernetes distribution or cloud provider used.

• v1, v2, and so on are stable, generally available APIs; that is:

  • They will stay.

  • They will be compatible.

In connection to API group versions, there are two important points to keep in mind:

• API group versions apply to API resources as a whole, like the format of pods or services. In addition to API group versions, API resources might have single fields that are versioned orthogonally; for example, fields in stable APIs might be marked as alpha quality in their Go inline code documentation. The same rules as those just listed for API groups will apply to those fields. For example:

  • An alpha field in a stable API could become incompatible, lose data, or go away at any time. For example, the ObjectMeta.Initializers field, which was never promoted beyond alpha, will go away in the near future (it is deprecated in 1.14):

    // DEPRECATED - initializers are an alpha field and will be removed
    // in v1.15.
    Initializers *Initializers `json:"initializers,omitempty"

  • It usually will be disabled by default and must be enabled with an API server feature gate, like so:

    type JobSpec struct {
        ...
        // This field is alpha-level and is only honored by servers that
        // enable the TTLAfterFinished feature.
        TTLSecondsAfterFinished *int32 `json:"ttlSecondsAfterFinished,omitempty"
    }

  • The behavior of the API server will differ from field to field. Some alpha fields will be rejected and some will be ignored if the corresponding feature gate is not enabled. This is documented in the field description (see TTLSecondsAfterFinished in the previous example).

• Furthermore, API group versions play a role in accessing the API. Between different versions of the same resource, there is an on-the-fly conversion done by the API server. That is, you can access objects created in one version (e.g., v1beta1) in any of the other supported versions (e.g., v1) without any further work in your application. This is very convenient for building backward- and forward-compatible applications.

  • Each object stored in etcd is stored in a specific version. By default, this is called the storage version of that resource. While the storage version can change from Kubernetes version to version, the object stored in etcd will not automatically be updated as of this writing. Hence, the cluster administrator has to make sure migration happens in time when Kubernetes clusters are updated, before old version support is dropped. There is no generic migration mechanism for that, and migration differs from Kubernetes distribution to distribution.

  • For the application developer, though, this operational work should not matter at all. On-the-fly conversion will make sure the application has a unified picture of the objects in the cluster. The application will not even notice which storage version is in use. Storage versioning will be transparent to the written Go code.

TypeMeta

While runtime.Object is only an interface, we want to know how it is actually implemented. Kubernetes objects from k8s.io/api implement the type getter and setter of schema.ObjectKind by embedding the metav1.TypeMeta struct from the package k8s.io/apimachinery/meta/v1:

// TypeMeta describes an individual object in an API response or request
// with strings representing the type of the object and its API schema version.
// Structures that are versioned or persisted should inline TypeMeta.
//
// +k8s:deepcopy-gen=false
type TypeMeta struct {
    // Kind is a string value representing the REST resource this object
    // represents. Servers may infer this from the endpoint the client submits
    // requests to.
    // Cannot be updated.
    // In CamelCase.
    // +optional
    Kind string `json:"kind,omitempty" protobuf:"bytes,1,opt,name=kind"`

    // APIVersion defines the versioned schema of this representation of an
    // object. Servers should convert recognized schemas to the latest internal
    // value, and may reject unrecognized values.
    // +optional
    APIVersion string `json:"apiVersion,omitempty"`
}

With this, a pod declaration in Go looks like this:

// Pod is a collection of containers that can run on a host. This resource is
// created by clients and scheduled onto hosts.
type Pod struct {
    metav1.TypeMeta `json:",inline"`
    // Standard object's metadata.
    // +optional
    metav1.ObjectMeta `json:"metadata,omitempty"`

    // Specification of the desired behavior of the pod.
    // +optional
    Spec PodSpec `json:"spec,omitempty"`

    // Most recently observed status of the pod.
    // This data may not be up to date.
    // Populated by the system.
    // Read-only.
    // +optional
    Status PodStatus `json:"status,omitempty"`
}

As you can see, TypeMeta is embedded. Moreover, the pod type has JSON tags that also declare TypeMeta as being inlined.

This matches the YAML representation of a pod, which all Kubernetes users know:²

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: example
spec:
  containers:
  - name: hello
    image: debian:latest
    command:
    - /bin/sh
    args:
    - -c
    - echo "hello world"; sleep 10000

The version is stored in TypeMeta.APIVersion, the kind in TypeMeta.Kind.

When running the example in “Creating and Using a Client” to get a pod from the cluster, notice that the pod object returned by the client does not actually have the kind and the version set. The convention in client-go–based applications is that these fields are empty in memory, and they are filled with the actual values on the wire only when they’re marshaled to JSON or protobuf. This is done automatically by the client, however, or, more precisely, by a versioning serializer.

In other words, client-go–based applications check the Golang type of objects to determine the object at hand. This might differ in other frameworks, like the Operator SDK (see “The Operator SDK”).

ObjectMeta

In addition to TypeMeta, most top-level objects have a field of type metav1.ObjectMeta, again from the k8s.io/apimachinery/pkg/meta/v1 package:

type ObjectMeta struct {
    Name string `json:"name,omitempty"`
    Namespace string `json:"namespace,omitempty"`
    UID types.UID `json:"uid,omitempty"`
    ResourceVersion string `json:"resourceVersion,omitempty"`
    CreationTimestamp Time `json:"creationTimestamp,omitempty"`
    DeletionTimestamp *Time `json:"deletionTimestamp,omitempty"`
    Labels map[string]string `json:"labels,omitempty"`
    Annotations map[string]string `json:"annotations,omitempty"`
    ...
}

In JSON or YAML these fields are under metadata. For example, for the previous pod, metav1.ObjectMeta stores:

metadata:
  namespace: default
  name: example

In general, it contains all metalevel information like name, namespace, resource version (not to be confused with the API group version), several timestamps, and the well-known labels and annotations is part of ObjectMeta. See “Anatomy of a type” for a deeper discussion of ObjectMeta fields.

The resource version was discussed earlier in “Optimistic Concurrency”. It is hardly ever read or written from client-go code. But it is one of the fields in Kubernetes that makes the whole system work. resourceVersion is part of ObjectMeta because each object with embedded ObjectMeta corresponds to a key in etcd where the resourceVersion value originated.

spec and status

Finally, nearly every top-level object has a spec and a status section. This convention comes from the declarative nature of the Kubernetes API: spec is the user desire, and status is the outcome of that desire, usually filled by a controller in the system. See “Controllers and Operators” for a detailed discussion of controllers in Kubernetes.

There are only a few exceptions to the spec and status convention in the system—for example, endpoints in the core group, or RBAC objects like ClusterRole.

Status Subresources: UpdateStatus

Deployments have a so-called status subresource. This means that UpdateStatus uses an additional HTTP endpoint suffixed with /status. While updates on the /apis/apps/v1beta1/namespaces/ns/deployments/name endpoint can change only the spec of the deployment, the endpoint /apis/apps/v1beta1/namespaces/ns/deployments/name/status can change only the status of the object. This is useful in order to set different permissions for spec updates (done by a human) and status updates (done by a controller).

By default the client-gen (see “client-gen Tags”) generates the UpdateStatus() method. The existence of the method does not guarantee that the resource actually supports the subresource. This will be important when we’re working with CRDs in “Subresources”.

Listings and Deletions

DeleteCollection allows us to delete multiple objects of a namespace at once. The ListOptions parameter allows us to define which objects should be deleted using a field or label selector:

type ListOptions struct {
    ...

    // A selector to restrict the list of returned objects by their labels.
    // Defaults to everything.
    // +optional
    LabelSelector string `json:"labelSelector,omitempty"`
    // A selector to restrict the list of returned objects by their fields.
    // Defaults to everything.
    // +optional
    FieldSelector string `json:"fieldSelector,omitempty"`

    ...
}

Watches

Watch gives an event interface for all changes (adds, removes, and updates) to objects. The returned watch.Interface from k8s.io/apimachinery/pkg/watch looks like this:

// Interface can be implemented by anything that knows how to watch and
// report changes.
type Interface interface {
    // Stops watching. Will close the channel returned by ResultChan(). Releases
    // any resources used by the watch.
    Stop()

    // Returns a chan which will receive all the events. If an error occurs
    // or Stop() is called, this channel will be closed, in which case the
    // watch should be completely cleaned up.
    ResultChan() <-chan Event
}

The result channel of the watch interface returns three kinds of events:

// EventType defines the possible types of events.
type EventType string

const (
    Added    EventType = "ADDED"
    Modified EventType = "MODIFIED"
    Deleted  EventType = "DELETED"
    Error    EventType = "ERROR"
)

// Event represents a single event to a watched resource.
// +k8s:deepcopy-gen=true
type Event struct {
    Type EventType

    // Object is:
    //  * If Type is Added or Modified: the new state of the object.
    //  * If Type is Deleted: the state of the object immediately before
    //    deletion.
    //  * If Type is Error: *api.Status is recommended; other types may
    //    make sense depending on context.
    Object runtime.Object
}

While it is tempting to use this interface directly, in practice it is actually discouraged in favor of informers (see “Informers and Caching”). Informers are a combination of this event interface and an in-memory cache with indexed lookup. This is by far the most common use case for watches. Under the hood informers first call List on the client to get the set of all objects (as a baseline for the cache) and then Watch to update the cache. They handle error conditions correctly—that is, recover from network issues or other cluster problems.

Client Expansion

DeploymentExpansion is actually an empty interface. It is used to add custom client behavior, but it’s hardly used in Kubernetes nowadays. Instead, the client generator allows us to add custom methods in a declarative way (see “client-gen Tags”).

Note again that all of those methods in DeploymentInterface neither expect valid information in the TypeMeta fields Kind and APIVersion, nor set those fields on Get() and List() (see also “TypeMeta”). These fields are filled with real values only on the wire.

Client Options

It is worth looking at the different options we can set when creating a client set. In the note before “Versioning and Compatibility”, we saw that we can switch to the protobuf wire format for native Kubernetes types. Protobuf is more efficient than JSON (both spacewise and for the CPU load of the client and server) and therefore preferable.

For debugging purposes and readability of metrics, it is often helpful to differentiate between different clients accessing the API server. To do so, we can set the user agent field in the REST config. The default value is binary/version (os/arch) kubernetes/commit; for example, kubectl will use a user agent like kubectl/v1.14.0 (darwin/amd64) kubernetes/d654b49. If that pattern does not suffice for the setup, it can be customized like so:

cfg, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
cfg.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"
cfg.UserAgent = fmt.Sprintf(
    "book-example/v1.0 (%s/%s) kubernetes/v1.0",
    runtime.GOOS, runtime.GOARCH
)
clientset, err := kubernetes.NewForConfig(cfg)

Other values often overridden in the REST config are those for client-side rate limiting and timeouts:

// Config holds the common attributes that can be passed to a Kubernetes
// client on initialization.
type Config struct {
    ...

    // QPS indicates the maximum QPS to the master from this client.
    // If it's zero, the created RESTClient will use DefaultQPS: 5
    QPS float32

    // Maximum burst for throttle.
    // If it's zero, the created RESTClient will use DefaultBurst: 10.
    Burst int

    // The maximum length of time to wait before giving up on a server request.
    // A value of zero means no timeout.
    Timeout time.Duration

    ...
}

The QPS value defaults to 5 requests per second, with a burst of 10.

The timeout has no default value, at least not in the client REST config. By default the Kubernetes API server will timeout every request that is not a long-running request after 60 seconds. A long-running request can be a watch request or unbounded requests to subresources like /exec, /portforward, or /proxy.

Work Queue

A work queue is a data structure. You can add elements and take elements out of the queue, in an order predefined by the queue. Formally, this kind of queue is called a priority queue. client-go provides a powerful implementation for the purpose of building controllers in k8s.io/client-go/util/workqueue.

More precisely, the package contains a number of variants for different purposes. The base interface implemented by all variants looks like this:

type Interface interface {
    Add(item interface{})
    Len() int
    Get() (item interface{}, shutdown bool)
    Done(item interface{})
    ShutDown()
    ShuttingDown() bool
}

Here Add(item) adds an item, Len() gives the length, and Get() returns an item with the highest priority (and it blocks until one is available). Every item returned by Get() needs a Done(item) call when the controller has finished processing it. Meanwhile, a repeated Add(item) will only mark the item as dirty such that it is readded when Done(item) has been called.

The following queue types are derived from this generic interface:

• DelayingInterface can add an item at a later time. This makes it easier to requeue items after failures without ending up in a hot-loop:

  type DelayingInterface interface {
      Interface
      // AddAfter adds an item to the workqueue after the
      // indicated duration has passed.
      AddAfter(item interface{}, duration time.Duration)
  }

• RateLimitingInterface rate-limits items being added to the queue. It extends the DelayingInterface:

  type RateLimitingInterface interface {
      DelayingInterface
  
      // AddRateLimited adds an item to the workqueue after the rate
      // limiter says it's OK.
      AddRateLimited(item interface{})
  
      // Forget indicates that an item is finished being retried.
      // It doesn't matter whether it's for perm failing or success;
      // we'll stop the rate limiter from tracking it. This only clears
      // the `rateLimiter`; you still have to call `Done` on the queue.
      Forget(item interface{})
  
      // NumRequeues returns back how many times the item was requeued.
      NumRequeues(item interface{}) int
  }

  Most interesting here is the Forget(item) method: it resets the back-off of the given item. Usually, it will be called when an item has been processed successfully.

  The rate limiting algorithm can be passed to the constructor NewRateLimitingQueue. There are several rate limiters defined in the same package, such as the BucketRateLimiter, the ItemExponentialFailureRateLimiter, the ItemFastSlowRateLimiter, and the MaxOfRateLimiter. For more details, you can refer to the package documentation. Most controllers will just use the DefaultControllerRateLimiter() *RateLimiter functions, which gives:

  • An exponential back-off starting at 5 ms and going up to 1,000 seconds, doubling the delay on each error

  • A maximal rate of 10 items per second and 100 items burst

Depending on the context, you might want to customize the values. A 1,000 seconds maximal back-off per item is a lot for certain controller applications.

Kinds

Kinds are divided into API groups and are versioned, as we already have seen in “API Terminology”. Therefore, a core term in the API Machinery repository is GroupVersionKind, or GVK for short.

In Go, each GVK corresponds to one Go type. In contrast, a Go type can belong to multiple GVKs.

Kinds do not formally map one-to-one to HTTP paths. Many kinds have HTTP REST endpoints that are used to access objects of the given kind. But there are also kinds without any HTTP endpoint (e.g., admission.k8s.io/v1beta1.AdmissionReview, which is used to call out to a webhook). There are also kinds that are returned from many endpoints—for example, meta.k8s.io/v1.Status, which is returned by all endpoints to report a nonobject status like an error.

By convention, kinds are formatted in CamelCase like words and are usually singular. Depending on the context, their concrete format differs. For CustomResourceDefinition kinds, it must be a DNS path label (RFC 1035).

Resources

In parallel to kinds, as we saw in “API Terminology”, there is the concept of a resource. Resources are again grouped and versioned, leading to the term GroupVersionResource, or GVR for short.

Each GVR corresponds to one HTTP (base) path. GVRs are used to identify REST endpoints of the Kubernetes API. For example, the GVR apps/v1.deployments maps to /apis/apps/v1/namespaces/namespace/deployments.

Client libraries use this mapping to construct the HTTP path to access a GVR.

By convention, resources are lowercase and plural, usually corresponding to the plural words of the parallel kind. They must conform to the DNS path label format (RFC 1025). As resources map directly to HTTP paths, this is not surprising.

REST Mapping

The mapping of a GVK to a GVR is called REST mapping.

A RESTMapper is the Golang interface that enables us to request the GVR for a GVK:

RESTMapping(gk schema.GroupKind, versions ...string) (*RESTMapping, error)

where the type RESTMapping on the right looks like this:

type RESTMapping struct {
    // Resource is the GroupVersionResource (location) for this endpoint.
    Resource schema.GroupVersionResource.

    // GroupVersionKind is the GroupVersionKind (data format) to submit
    // to this endpoint.
    GroupVersionKind schema.GroupVersionKind

    // Scope contains the information needed to deal with REST Resources
    // that are in a resource hierarchy.
    Scope RESTScope
}

In addition, a RESTMapper provides a number of convenience functions:

// KindFor takes a partial resource and returns the single match.
// Returns an error if there are multiple matches.
KindFor(resource schema.GroupVersionResource) (schema.GroupVersionKind, error)

// KindsFor takes a partial resource and returns the list of potential
// kinds in priority order.
KindsFor(resource schema.GroupVersionResource) ([]schema.GroupVersionKind, error)

// ResourceFor takes a partial resource and returns the single match.
// Returns an error if there are multiple matches.
ResourceFor(input schema.GroupVersionResource) (schema.GroupVersionResource, error)

// ResourcesFor takes a partial resource and returns the list of potential
// resource in priority order.
ResourcesFor(input schema.GroupVersionResource) ([]schema.GroupVersionResource, error)

// RESTMappings returns all resource mappings for the provided group kind
// if no version search is provided. Otherwise identifies a preferred resource
// mapping for the provided version(s).
RESTMappings(gk schema.GroupKind, versions ...string) ([]*RESTMapping, error)

Here, a partial GVR means that not all fields are set. For example, imagine you type kubectl get pods. In that case, the group and the version are missing. A RESTMapper with enough information might still manage to map it to the v1 Pods kind.

For the preceding deployment example, a RESTMapper that knows about deployments (more about what this means in a bit) will map apps/v1.Deployment to apps/v1.deployments as a namespaced resource.

There are multiple different implementations of the RESTMapper interface. The most important one for client applications is the discovery-based DeferredDiscoveryRESTMapper in the package k8s.io/client-go/restmapper: it uses discovery information from the Kubernetes API server to dynamically build up the REST mapping. It will also work with non-core resources like custom resources.

Scheme

The final core concept we want to present here in the context of the Kubernetes type system is the scheme in the package k8s.io/apimachinery/pkg/runtime.

A scheme connects the world of Golang with the implementation-independent world of GVKs. The main feature of a scheme is the mapping of Golang types to possible GVKs:

func (s *Scheme) ObjectKinds(obj Object) ([]schema.GroupVersionKind, bool, error)

As we saw in “Kubernetes Objects in Go”, an object can return its group and kind via the GetObjectKind() schema.ObjectKind method. However, these values are empty most of the time and are therefore pretty useless for identification.

Instead, the scheme takes the Golang type of the given object via reflection and maps it to the registered GVK(s) of that Golang type. For that to work, of course, the Golang types have to be registered into the scheme like this:

scheme.AddKnownTypes(schema.GroupVersionKind{"", "v1", "Pod"}, &Pod{})

The scheme is used not only to register the Golang types and their GVK, but also to store a list of conversion functions and defaulters (see Figure 3-6). We’ll discuss conversions and defaulters in more detail in Chapter 8. It is the data source to implement encoders and decoders as well.

Figure 3-6. The scheme, connecting Golang data types with the GVK, conversions, and defaulters

For Kubernetes core types there is a predefined scheme in the client-go client set in the package k8s.io/client-go/kubernetes/scheme, with all the types preregistered. Actually, every client set generated by the client-gen code generator (see Chapter 5) has the subpackage scheme with all types in all groups and versions in the client set.

With the scheme we conclude our deep dive into API Machinery concepts. If you only remember one thing about these concepts, let it be Figure 3-7.

Figure 3-7. From Golang types to GVKs to GVRs to an HTTP path—API Machinery in a nutshell

glide

Projects using glide can use its ability to read the Godeps/Godeps.json file on any dependency change. This has proven to work pretty reliably: the developer has only to declare the right k8s.io/client-go version, and glide will select the right version of k8s.io/apimachinery, k8s.io/api, and other dependencies.

For some projects on GitHub, the glide.yaml file might look like this:

package: github.com/book/example
import:
- package: k8s.io/client-go
  version: v10.0.0
...

With that, glide install -v will download k8s.io/client-go and its dependencies into the local vendor/ package. Here, -v means to drop vendor/ packages from vendored libraries. This is required for our purposes.

If you update to a new version of client-go by editing glide.yaml, glide update -v will download the new dependencies, again in the right versions.

dep

dep is often considered more powerful and advanced than glide. For a long time it was seen as the successor to glide in the ecosystem and seemed destined to be the Go vendoring tool. At the time of this writing, its future is not clear, and Go modules seem to be the path forward.

In the context of client-go, it is very important to be aware of a couple of restrictions of dep:

• dep does read Godeps/Godeps.json on the first run of dep init.

• dep does not read Godeps/Godeps.json on later dep ensure -update calls.

This means that the resolution for dependencies of client-go is most probably wrong when the client-go version is updated in Godep.toml. This is unfortunate, because it requires the developer to explicitly and usually manually declare all dependencies.

A working and consistent Godep.toml file looks like this:

[[constraint]]
  name = "k8s.io/api"
  version = "kubernetes-1.13.0"

[[constraint]]
  name = "k8s.io/apimachinery"
  version = "kubernetes-1.13.0"

[[constraint]]
  name = "k8s.io/client-go"
  version = "10.0.0"

[prune]
  go-tests = true
  unused-packages = true

# the following overrides are necessary to enforce
# the given version, even though our
# code does not import the packages directly.
[[override]]
  name = "k8s.io/api"
  version = "kubernetes-1.13.0"

[[override]]
  name = "k8s.io/apimachinery"
  version = "kubernetes-1.13.0"

[[override]]
  name = "k8s.io/client-go"
  version = "10.0.0"

Go Modules

Go modules are the future of dependency management in Golang. They were introduced in Go 1.11 with preliminary support and were further stabilized in 1.12. A number of commands, like go run and go get, work with Go modules by setting the GO111MODULE=on environment variable. In Go 1.13 this will be the default setting.

Go modules are driven by a go.mod file in the root of a project. Here is an excerpt of the go.mod file for our github.com/programming-kubernetes/pizza-apiserver project in Chapter 8:

module github.com/programming-kubernetes/pizza-apiserver

require (
    ...
    k8s.io/api v0.0.0-20190222213804-5cb15d344471 // indirect
    k8s.io/apimachinery v0.0.0-20190221213512-86fb29eff628
    k8s.io/apiserver v0.0.0-20190319190228-a4358799e4fe
    k8s.io/client-go v2.0.0-alpha.0.0.20190307161346-7621a5ebb88b+incompatible
    k8s.io/klog v0.2.1-0.20190311220638-291f19f84ceb
    k8s.io/kube-openapi v0.0.0-20190320154901-c59034cc13d5 // indirect
    k8s.io/utils v0.0.0-20190308190857-21c4ce38f2a7 // indirect
    sigs.k8s.io/yaml v1.1.0 // indirect
)

client-go v11.0.0—matching Kubernetes 1.14—and older versions do not have explicit support for Go modules. Still, it is possible to use Go modules with the Kubernetes libraries, as you see in the preceding example.

As long as client-go and the other Kubernetes repositories do not ship a go.mod file, though (at least until Kubernetes 1.15), the right versions must be selected manually. That is, you’ll need a complete list of all dependencies matching the revisions of dependencies of the Godeps/Godeps.json in client-go.

Also note the not-very-readable revisions in the previous example. They are pseudo-versions derived from existing tags, or using v0.0.0 as the prefix if there are no tags. Even worse, you can reference tagged versions in that file, but the Go module commands will replace those on the next run with the pseudo-versions.

With client-go v12.0.0—matching Kubernetes 1.15—we ship a go.mod file and deprecate support for all other vendoring tools (see the corresponding proposal document). The shipped go.mod file includes all dependencies, and your project go.mod file no longer has to list all transitive dependencies manually. In later releases, it’s also possible that the tagging scheme will be changed to fix the ugly pseudo-revisions and replace them with proper semver tags. But at the time of this writing, this is still not fully implemented or decided.