© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2023
P. MartinKubernetes Programming with Gohttps://doi.org/10.1007/978-1-4842-9026-2_6

6. The Client-go Library

Philippe Martin1  
(1)
Blanquefort, France
 

The previous chapters explored the Kubernetes API Library, a collection of Go structures to work with the objects of the Kubernetes API, and the API Machinery Library, which provides utilities for working with the API objects that follow the Kubernetes API object conventions. Specifically, you have seen that the API Machinery provides Scheme and RESTMapper abstractions.

This chapter explores the Client-go Library, which is a high-level library that can be used by developers to interact with the Kubernetes API using the Go language. The Client-go Library brings together the Kubernetes API and the API Machinery libraries, providing a Scheme preconfigured with Kubernetes API’s objects and a RESTMapper implementation for the Kubernetes API. It also provides a set of clients to use to execute operations on the resources of the Kubernetes API in a simple way.

To use this library, you will need to import packages from it with the prefix k8s.io/client-go. For example, to use the package kubernetes, let’s use the following:
import (
     "k8s.io/client-go/kubernetes"
)
You also need to download a version of the Client-go Library. For this you can employ the go get command to obtain the version you want to use:
$ go get k8s.io/[email protected]

The version of the Client-go Library is aligned with the version of Kubernetes—version 0.24.4 corresponds to version 1.24.4 of the server.

Kubernetes is backward-compatible so you can use older versions of Client-go with newer versions of clusters, but you may well want to get a recent version to be able to use a current feature, because only bug fixes are backported to previous client-go releases, not new features.

Connecting to the Cluster

The first step before connecting to the Kubernetes API Server is to have the configuration connect to it—that is, the address of the server, its credentials, the connection parameters, and so on.

The rest package provides a rest.Config structure, which contains all the configuration information necessary for an application to connect to a REST API Server.

In-cluster Configuration

By default, a container running on a Kubernetes Pod contains all the information needed to connect to the API Server:

  • A token and the root certificate, provided by the ServiceAccount used for the Pod, are available in this directory: /var/run/secrets/kubernetes.io/serviceaccount/.

  • Note that it is possible to disable this behavior by setting automountServiceAccountToken: false in the ServiceAccount used by the Pod, or in the specifications of the Pod directly

  • The environment variables, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT, defined in the container environment, added by kubelet, define the host and port to which to contact the API Server.

When an application is dedicated to run inside a Pod’s container, you can use the following function to create an appropriate rest.Config structure, leveraging the information just described:
import "k8s.io/client-go/rest"
func InClusterConfig() (*Config, error)

Out-of-Cluster Configuration

Kubernetes tools generally rely on the kubeconfig file—that is, a file that contains connection configuration for one or several Kubernetes clusters.

You can build a rest.Config structure based on the content of this kubeconfig file by using one of the following functions from the clientcmd package.

From kubeconfig in Memory

The RESTConfigFromKubeConfig function can be used to build a rest.Config structure from the content of a kubeconfig file as an array of bytes.
func RESTConfigFromKubeConfig(
     configBytes []byte,
) (*rest.Config, error)
If the kubeconfig file contains several contexts, the current context will be used, and the other contexts will be ignored. For example, you can read the content of a kubeconfig file first, then use the following function:
import "k8s.io/client-go/tools/clientcmd"
configBytes, err := os.ReadFile(
     "/home/user/.kube/config",
)
if err != nil {
     return err
}
config, err := clientcmd.RESTConfigFromKubeConfig(
     configBytes,
)
if err != nil {
     return err
}

From a kubeconfig on Disk

The BuildConfigFromFlags function can be used to build a rest.Config structure either from the URL of the API Server, or based on a kubeconfig file given its path, or both.
func BuildConfigFromFlags(
     masterUrl,
     kubeconfigPath string,
) (*rest.Config, error)
The following code allows you to get a rest.Config structure:
import "k8s.io/client-go/tools/clientcmd"
config, err := clientcmd.BuildConfigFromFlags(
     "",
     "/home/user/.kube/config",
)
The following code gets the configuration from the kubeconfig, and overrides the URL of the API Server:
config, err := clientcmd.BuildConfigFromFlags(
     "https://192.168.1.10:6443",
     "/home/user/.kube/config",
)

From a Personalized kubeconfig

The previous functions use an api.Config structure internally, representing the data in the kubeconfig file (not to be confused with the rest.Config structure that contains the parameters for the REST HTTP connection).

If you need to manipulate this intermediary data, you can use the BuildConfigFromKubeconfigGetter function accepting a kubeconfigGetter function as an argument, which itself will return an api.Config structure.
BuildConfigFromKubeconfigGetter(
     masterUrl string,
     kubeconfigGetter KubeconfigGetter,
) (*rest.Config, error)
type KubeconfigGetter
     func() (*api.Config, error)
For example, the following code will load the kubeconfig file with the clientcmd.Load or clientcmd.LoadFromFile functions from the kubeconfigGetter function:
import (
     "k8s.io/client-go/tools/clientcmd"
     "k8s.io/client-go/tools/clientcmd/api"
)
config, err :=
clientcmd.BuildConfigFromKubeconfigGetter(
     "",
     func() (*api.Config, error) {
          apiConfig, err := clientcmd.LoadFromFile(
               "/home/user/.kube/config",
          )
          if err != nil {
               return nil, nil
          }
          // TODO: manipulate apiConfig
          return apiConfig, nil
     },
)

From Several kubeconfig Files

The kubectl tool uses by default the $HOME/.kube/config kubeconfig file, and you can specify another kubeconfig file path using the KUBECONFIG environment variable.

More than that, you can specify a list of kubeconfig file paths in this environment variable, and the kubeconfig files will be merged into just one before being used. You can obtain the same behavior with this function: NewNonInteractiveDeferredLoadingClientConfig.
func NewNonInteractiveDeferredLoadingClientConfig(
     loader ClientConfigLoader,
     overrides *ConfigOverrides,
) ClientConfig
The type clientcmd.ClientConfigLoadingRules implements the ClientConfigLoader interface, and you can get a value for this type using the following function:
func NewDefaultClientConfigLoadingRules()
 *ClientConfigLoadingRules

This function will get the value of the KUBECONFIG environment variable, if it exists, to obtain the list of kubeconfig files to merge, or will fallback on using the default kubeconfig file located in $HOME/.kube/config.

Using the following code to create the rest.Config structure, your program will have the same behavior as kubectl, as previously described:
import (
     "k8s.io/client-go/tools/clientcmd"
)
config, err :=
clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
     clientcmd.NewDefaultClientConfigLoadingRules(),
     nil,
).ClientConfig()

Overriding kubeconfig with CLI Flags

It has been shown that the second parameter of this function, NewNonInteractiveDeferredLoadingClientConfig, is a ConfigOverrides structure. This structure contains values to override some fields of the result of merging the kubeconfig files.

You can set specific values in this structure yourself, or, if you are creating a CLI using the spf13/pflag library (i.e., github.com/spf13/pflag), you can use the following code to automatically declare default flags for your CLI and bind them to the ConfigOverrides structure:
import (
     "github.com/spf13/pflag"
     "k8s.io/client-go/tools/clientcmd"
)
var (
     flags pflag.FlagSet
     overrides clientcmd.ConfigOverrides
     of = clientcmd.RecommendedConfigOverrideFlags("")
)
clientcmd.BindOverrideFlags(&overrides, &flags, of)
flags.Parse(os.Args[1:])
config, err :=
clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
     clientcmd.NewDefaultClientConfigLoadingRules(),
     &overrides,
).ClientConfig()

Note that you can declare a prefix for the added flags when calling the function RecommendedConfigOverrideFlags.

Getting a Clientset

The Kubernetes package provides functions to create a clientset of the type kubernetes.Clientset.

  • func NewForConfig(c *rest.Config) (*Clientset, error) – The NewForConfig function returns a Clientset, using the provided rest.Config built with one of the methods seen in the previous section.

  • func NewForConfigOrDie(c *rest.Config) *Clientset – this function is like the previous one, but panics in case of error, instead of returning the error. This function can be used with a hard-coded config for which you will want to assert its validity.

• NewForConfigAndClient(
     c *rest.Config,
     httpClient *http.Client,
) (*Clientset, error)

  • – this NewForConfigAndClient function returns a Clientset, using the provided rest.Config, and the provided http.Client.

  • The previous function NewForConfig uses a default HTTP Client built with the function rest.HTTPClientFor. If you want to personalize the HTTP Client before building the Clientset, you can use this function instead.

Using the Clientset

The kubernetes.Clientset type implements the interface kubernetes.Interface, defined as follows:
type Interface interface {
     Discovery() discovery.DiscoveryInterface
     [...]
     AppsV1()          appsv1.AppsV1Interface
     AppsV1beta1()     appsv1beta1.AppsV1beta1Interface
     AppsV1beta2()     appsv1beta2.AppsV1beta2Interface
     [...]
     CoreV1()           corev1.CoreV1Interface
     [...]
}

The first method Discovery() gives access to an interface that provides methods to discover the groups, versions, and resources available in the cluster, as well as preferred versions for resources. This interface also provides access to the server version and the OpenAPI v2 and v3 definitions. This is examined in detail in the Discovery client section.

Apart from the Discovery() method, the kubernetes.Interface is composed of a series of methods, one for each Group/Version defined by the Kubernetes API. When you see the definition of this interface, it is possible to understand that the Clientset is a set of clients, and each client is dedicated to its own Group/Version.

Each method returns a value that implements an interface specific to the Group/Version. For example, the CoreV1() method of kubernetes.Interface returns a value, implementing the interface corev1.CoreV1Interface, defined as follows:
type CoreV1Interface interface {
     RESTClient() rest.Interface
     ComponentStatusesGetter
     ConfigMapsGetter
     EndpointsGetter
     [...]
}

The first method in this CoreV1Interface interface is RESTClient() rest.Interface, which is a method used to get a REST client for the specific Group/Version. This low-level client will be used internally by the Group/Version client, and you can use this REST client to build requests not provided natively by the other methods of this interface: CoreV1Interface.

The interface rest.Interface implemented by the REST client is defined as follows:
type Interface interface {
     GetRateLimiter()            flowcontrol.RateLimiter
     Verb(verb string)           *Request
     Post()                      *Request
     Put()                       *Request
     Patch(pt types.PatchType)   *Request
     Get()                       *Request
     Delete()                    *Request
     APIVersion()                schema.GroupVersion
}

As you can see, this interface provides a series of methods—Verb, Post, Put, Patch, Get, and Delete—that return a Request object with a specific HTTP Verb. This is examined further in the “How to Use These Request Objects to Complete Operations” section.

The other methods in the CoreV1Interface are used to get specific methods for each Resource of the Group/Version. For example, the ConfigMapsGetter embedded interface is defined as follows:
type ConfigMapsGetter interface {
     ConfigMaps(namespace string) ConfigMapInterface
}
Then, the interface ConfigMapInterface is returned by the method ConfigMaps and is defined as follows:
type ConfigMapInterface interface {
     Create(
          ctx context.Context,
          configMap *v1.ConfigMap,
          opts metav1.CreateOptions,
     ) (*v1.ConfigMap, error)
     Update(
          ctx context.Context,
          configMap *v1.ConfigMap,
          opts metav1.UpdateOptions,
     ) (*v1.ConfigMap, error)
     Delete(
          ctx context.Context,
          name string,
          opts metav1.DeleteOptions,
     ) error
     [...]
}

You can see that this interface provides a series of methods, one for each Kubernetes API Verb.

Each method related to an operation takes as a parameter an Option structure, named after the name of the operation: CreateOptions, UpdateOptions, DeleteOptions, and so on. These structures, and the related constants, are defined in this package: k8s.io/apimachinery/pkg/apis/meta/v1.

Finally, to make an operation on a resource of a Group-Version, you can chain the calls following this pattern for namespaced resources, where namespace can be the empty string to indicate a cluster-wide operation:
clientset.
     GroupVersion().
     NamespacedResource(namespace).
     Operation(ctx, options)
Then, the following is the pattern for non-namespaced resources:
clientset.
    GroupVersion().
     NonNamespacedResource().
     Operation(ctx, options)
For example, use the following to List the Pods of the core/v1 Group/Version in namespace project1:
podList, err := clientset.
     CoreV1().
     Pods("project1").
     List(ctx, metav1.ListOptions{})
To get the list of pods in all namespaces, you need to specify an empty namespace name:
podList, err := clientset.
     CoreV1().
     Pods("").
     List(ctx, metav1.ListOptions{})
To get the list of nodes (which are non-namespaced resources) use this:
nodesList, err := clientset.
     CoreV1().
     Nodes().
     List(ctx, metav1.ListOptions{})

The following sections describe in detail the various operations using the Pod resource. You can apply the same examples by removing the namespace parameter when working with non-namespaced resources.

Examining the Requests

If you want to know which HTTP requests are executed when calling client-go methods, you can enable logging for your program. The Client-go Library uses the klog library (https://github.com/kubernetes/klog), and you can enable the log flags for your command with the following code:
import (
     "flag"
     "k8s.io/klog/v2"
)
func main() {
     klog.InitFlags(nil)
     flag.Parse()
     [...]
}

Now, you can run your program with the flag -v <level>—for example, -v 6 to get the URL called for every request. You can find more detail about the defined log levels in Table 2-1.

Creating a Resource

To create a new resource in the cluster, you first need to declare this resource in memory using the dedicated Kind structure, then use the Create method for the resource you want to create. For example, use the following to create a Pod named nginx-pod in the project1 namespace:
wantedPod := corev1.Pod{
     Spec: corev1.PodSpec{
          Containers: []corev1.Container{
               {
                    Name:  "nginx",
                    Image: "nginx",
               },
          },
     },
}
wantedPod.SetName("nginx-pod")
createdPod, err := clientset.
     CoreV1().
     Pods("project1").
     Create(ctx, &wantedPod, v1.CreateOptions{})
The various options used to declare the CreateOptions structure, when creating a resource, are:

  • DryRun – this indicates which operations on the API server-side should be executed. The only available value is metav1.DryRunAll, indicating execution of all the operations except persisting the resource to storage.

  • Using this option, you can get, as result of the command, the exact object that would have been created in the cluster without really creating it, and check whether an error would occur during this creation.

  • FieldManager – this indicates the name of the field manager for this operation. This information will be used for future server-side Apply operations.

  • FieldValidation – this indicates how the server should react when duplicate or unknown fields are present in the structure. The following are the possible values:

    • metav1.FieldValidationIgnore to ignore all duplicate or unknown fields

    • metav1.FieldValidationWarn to warn when duplicate or unknown fields are present

    • metav1.FieldValidationStrict to fail when duplicate or unknown fields are present

  • Note that using this method, you will not be able to define duplicate or unknown fields because you are using a structure to define the object.

In case of error, you can test its type with the functions defined in the package k8s.io/apimachinery/pkg/api/errors. All the possible errors are defined in section “Errors and Statuses”, and here are the possible errors specific to the Create operation:

  • IsAlreadyExists – this function indicates whether the request failed because a resource with the same name already exists in the cluster:

if errors.IsAlreadyExists(err) {
     // ...
}

  • IsNotFound – this function indicates whether the namespace you specified in the request does not exist.

  • IsInvalid – this function indicates whether the data passed into the structure is invalid.

Getting Information About a Resource

To get information about a specific resource in the cluster, you can use the Get method for the resource you want to get information from. For example, to get information about the pod named nginx-pod in the project1 namespace:
pod, err := clientset.
     CoreV1().
     Pods("project1").
     Get(ctx, "nginx-pod", metav1.GetOptions{})
The various options to declare it into the GetOptions structure, when getting information about a resource are:

  • ResourceVersion – to request a version of the resource not older than the specified version.

  • If ResourceVersion is “0,” indicates to return any version of the resource. You will generally receive the latest version of the resource, but this is not guaranteed; receiving an older version can happen on high availability clusters due to partitioning or stale cache.

  • If the option is not set, you are guaranteed to receive the most recent version of the resource.

The possible error specific to the Get operation is:

  • IsNotFound – this function indicates that the namespace you specified in the request does not exist, or that the resource with the specified name does not exist.

Getting List of Resources

To get a list of resources in the cluster, you can use the List method for the resource you want to list. For example, use the following to list the pods in the project1 namespace:
podList, err := clientset.
     CoreV1().
     Pods("project1").
     List(ctx, metav1.ListOptions{})
Or, to get the list of pods in all namespaces, use:
podList, err := clientset.
     CoreV1().
     Pods("").
     List(ctx, metav1.ListOptions{})
The various options to declare into the ListOptions structure, when listing resources are the following:

  • LabelSelector, FieldSelector – this is used to filter the list by label or by field. These options are detailed in the “Filtering the Result of a List” section.

  • Watch, AllowWatchBookmarks – this is used to run a Watch operation. These options are detailed in the “Watching Resources” section.

  • ResourceVersion, ResourceVersionMatch – this indicates which version of the List of resources you want to obtain.

  • Note that, when receiving a response of a List operation, a ResourceVersion value is indicated for the List element itself, as well as ResourceVersion values for each element of the list. The ResourceVersion to indicate in the Options refers to the ResourceVersion of the List.

  • For a List operation without pagination (you can refer to the “Paginating Results” and “Watching Resources” sections for the behavior of these options in other circumstances):

    • When ResourceVersionMatch is not set, the behavior is the same as for a Get operation:

    • ResourceVersion indicates that you should return a list that is not older than the specified version.

    • If ResourceVersion is “0,” this indicates that it is necessary to return to any version of the list. You generally will receive the latest version of it, but this is not guaranteed; receiving an older version can happen on high-availability clusters because of a partitioning or a stale cache.

    • If the option is not set, you are guaranteed to receive the most recent version of the list.

    • When ResourceVersionMatch is set to metav1.ResourceVersionMatchExact, the ResourceVersion value indicates the exact version of the list you want to obtain.

    • Setting ResourceVersion to “0,” or not defining it, is invalid.

    • When ResourceVersionMatch is set to metav1.ResourceVersionMatchNotOlderThan, ResourceVersion indicates you will obtain a list that is not older than the specified version.

    • If ResourceVersion is “0,” this indicates a return any version of the list. You generally will receive the latest version of the list, but this is not guaranteed; receiving an older version can happen on high-availability clusters because of a partitioning or a stale cache.

    • Not defining ResourceVersion is invalid.

  • TimeoutSeconds – this limits the duration of the request to the indicated number of seconds.

  • Limit, Continue – this is used for paginating the result of the list. These options are detailed in Chapter 2’s “Paginating Results” section.

The following are the possible errors specific to the List operation:

  • IsResourceExpired – this function indicates that the specified ResourceVersion with a ResourceVersionMatch, set to metav1.ResourceVersionMatchExact, is expired.

Note that, if you specify a nonexisting namespace for a List operation, you will not receive a NotFound error.

Filtering the Result of a List

As described in Chapter 2’s “Filtering the Result of a List” section, it is possible to filter the result of a List operation with labels selectors and field selectors. This section shows how to use the fields and labels packages of the API Machinery Library to create a string applicable to the LabelSelector and FieldSelector options.

Setting LabelSelector Using the Labels Package

Here is the necessary import information to use the labels package from the API Machinery Library.
import (
     "k8s.io/apimachinery/pkg/labels"
)

The package provides several methods for building and validating a LabelsSelector string: using Requirements, parsing a labelSelector string, or using a set of key-value pairs.

Using Requirements

You first need to create a labels.Selector object using the following code:
labelsSelector := labels.NewSelector()
Then, you can create Requirement objects using the labels.NewRequirement function:
func NewRequirement(
     key string,
     op selection.Operator,
     vals []string,
     opts ...field.PathOption,
) (*Requirement, error)
Constants for the possible values of op are defined in the selection package (i.e., k8s.io/apimachinery/pkg/selection). The number of values in the vals array of strings depends on the operation:

  • selection.In; selection.NotIn – the value attached to key must equal one of (In)/must not equal one of (NotIn) the values defined of vals.

  • vals must be non-empty.

  • selection.Equals; selection.DoubleEquals; selection.NotEquals – the value attached to key must equal (Equals, DoubleEquals) or must not equal (NotEquals) the value defined in vals.

  • vals must contain a single value.

  • selection.Exists; selection.DoesNotExist – the key must be defined (Exists) or must not be defined (DoesNotExist).

  • vals must be empty.

  • selection.Gt; selection.Lt – the value attached to a key must be greater than (Gt) or less than (Lt) the value defined in vals.

  • vals must contain a single value, representing an integer.

For example, to require that the value of the key mykey equals value1, you can declare a Requirement with:
req1, err := labels.NewRequirement(
     "mykey",
     selection.Equals,
     []string{"value1"},
)
After defining the Requirement, you can add the requirements to the selector using the Add method on the selector:
labelsSelector = labelsSelector.Add(*req1, *req2)
Finally, you can obtain the String to be passed for the LabelSelector option with:
s := labelsSelector.String()

Parsing a LabelSelector String

If you already have a string describing the label selector, you can check its validity with the Parse function. The Parse function will validate the string and return a LabelSelector object. You can use the String method on this LabelSelector object to obtain the string as validated by the Parse function.

As an example, the following code will parse, validate, and return the canonical form of the label selector, “mykey = value1, count < 5”:
selector, err := labels.Parse(
     "mykey = value1, count < 5",
)
if err != nil {
     return err
}
s := selector.String()
// s = "mykey=value1,count<5"

Using a Set of Key-value Pairs

The function ValidatedSelectorFromSet can be used when you only want to use the Equal operation, for one or several requirements:
func ValidatedSelectorFromSet(
     ls Set
) (Selector, error)

In this case, the Set will define the set of key-value pairs you want to check for equality.

As an example, the following code will declare a label selector that requires the key, key1, to equal value1 and the key, key2, to equal value2:
set := labels.Set{
     "key1": "value1",
     "key2": "value2",
}
selector, err = labels.ValidatedSelectorFromSet(set)
s = selector.String()
// s = "key1=value1,key2=value2"

Setting Fieldselector Using the Fields Package

Here is the necessary code to use to import the fields package from the API Machinery Library.
import (
     "k8s.io/apimachinery/pkg/fields"
)

The package provides several methods for building and validating a FieldSelector string: assembling one term selectors, parsing a fieldSelector string, or using a set of key-value pairs.

Assembling One Term Selectors

You can create one term selectors with the functions OneTermEqualSelector and OneTermNotEqualSelector, then assemble the selectors to build a complete field selector with the function AndSelectors.
func OneTermEqualSelector(
     k, v string,
) Selector
func OneTermNotEqualSelector(
     k, v string,
) Selector
func AndSelectors(
     selectors ...Selector,
) Selector
For example, this code builds a field selector with an Equal condition on the field status.Phase and a NotEqual condition on the field spec.restartPolicy:
fselector = fields.AndSelectors(
     fields.OneTermEqualSelector(
          "status.Phase",
          "Running",
     ),
     fields.OneTermNotEqualSelector(
          "spec.restartPolicy",
          "Always",
     ),
)
fs = fselector.String()

Parsing a FieldSelector String

If you already have a string describing the field selector, you can check its validity with the ParseSelector or ParseSelectorOrDie functions. The ParseSelector function will validate the string and return a fields.Selector object. You can use the String method on this fields.Selector object to obtain the string, as validated by the ParseSelector function.

As an example, this code will parse, validate, and return the canonical form of the field selector “status.Phase = Running, spec.restartPolicy != Always”:
selector, err := fields.ParseSelector(
     "status.Phase=Running, spec.restartPolicy!=Always",
)
if err != nil {
     return err
}
s := selector.String()
// s = "spec.restartPolicy!=Always,status.Phase=Running"

Using a Set of Key-Value Pairs

The function SelectorFromSet can be used when you want to use only the Equal operation, for one or several single selectors.
func SelectorFromSet(ls Set) Selector

In this case, the Set will define the set of key-value pairs you want to check for equality.

As an example, the following code will declare a field selector that requires the key, key1, to equal value1 and the key, key2, to equal value2:
set := fields.Set{
     "field1": "value1",
     "field2": "value2",
}
selector = fields.SelectorFromSet(set)
s = selector.String()
// s = "key1=value1,key2=value2"

Deleting a Resource

To delete a resource from the cluster, you can use the Delete method for the resource you want to delete. For example, to delete a Pod named nginx-pod from the project1 namespace use:
err = clientset.
     CoreV1().
     Pods("project1").
     Delete(ctx, "nginx-pod", metav1.DeleteOptions{})

Note that it is not guaranteed that the resource is deleted when the operation terminates. The Delete operation will not effectively delete the resource, but mark the resource to be deleted (by setting the field .metadata.deletionTimestamp), and the deletion will happen asynchronously.

The different options, to declare into the DeleteOptions structure, when deleting a resource are:

  • DryRun – this indicates which operations on the API server-side should be executed. The only available value is metav1.DryRunAll, indicating that it is to execute all the operations except (the operation of) persisting the resource to storage. Using this option, you can get the result of the command, without really deleting the resource, and check whether an error would occur during this deletion.

  • GracePeriodSeconds – this value is useful when deleting pods only. This indicates the duration in seconds before the pod should be deleted.

  • The value must be a pointer to a non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the pod will be used, as indicated in the TerminationGracePeriodSeconds field of the pod specification.

  • You can use the metav1.NewDeleteOptions function to create a DeleteOptions structure with the GracePeriodSeconds defined:
    err = clientset.
         CoreV1().
         Pods("project1").
         Delete(ctx,
              "nginx-pod",
              *metav1.NewDeleteOptions(5),
         )
  • Preconditions – When you delete an object, you may want to be sure to delete the expected one. The Preconditions field lets you indicate which resource you expect to delete, either by:
    • Indicating the UID, so if the expected resource is deleted and another resource is created with the same name, the deletion will fail, producing a Conflict error. You can use the metav1.NewPreconditionDeleteOptions function to create a DeleteOptions structure with the UID of the Preconditions set:
      uid := createdPod.GetUID()
      err = clientset.
           CoreV1().
           Pods("project1").
           Delete(ctx,
                "nginx-pod",
                *metav1.NewPreconditionDeleteOptions(
                     string(uid),
                ),
           )
      if errors.IsConflict(err) {
         [...]
      }
  • Indicating the ResourceVersion, so if the resource is updated in the meantime, the deletion will fail, with a Conflict error. You can use the metav1.NewRVDeletionPrecondition function to create a DeleteOptions structure with the ResourceVersion of the Preconditions set:
    rv := createdPod.GetResourceVersion()
    err = clientset.
         CoreV1().
         Pods("project1").
         Delete(ctx,
              "nginx-pod",
              *metav1.NewRVDeletionPrecondition(
                   rv,
              ),
         )
    if errors.IsConflict(err) {
       [...]
    }
  • OrphanDependents – this field is deprecated in favor of PropagationPolicy. PropagationPolicy – this indicates whether and how garbage collection will be performed. See also Chapter 3’s “OwnerReferences” section. The acceptable values are:

    • metav1.DeletePropagationOrphan – to indicate to the Kubernetes API to orphan the resources owned by the resource you are deleting, so they will not be deleted by the garbage collector.

    • metav1.DeletePropagationBackground – to indicate to the Kubernetes API to return from the Delete operation immediately after the owner resource is marked for deletion, not to wait for owned resources to be deleted by the garbage collector.

    • metav1.DeletePropagationForeground – to indicate to the Kubernetes API to return from the Delete operation after the owner and the owned resources with BlockOwnerDeletion set to true are deleted. The Kubernetes API will not wait for other owned resources to be deleted.

The following are the possible errors specific to the Delete operation:

  • IsNotFound – this function indicates that the resource or the namespace you specified in the request does not exist.

  • IsConflict – this function indicates that the request failed because a precondition is not respected (either UID or ResourceVersion)

Deleting a Collection of Resources

To delete a collection of resources from the cluster, you can use the DeleteCollection method for the resource you want to delete. For example, to delete a collection of Pods from the project1 namespace:
err = clientset.
     CoreV1().
     Pods("project1").
     DeleteCollection(
          ctx,
          metav1.DeleteOptions{},
          metav1.ListOptions{},
     )
Two sets of options must be provided to the function:

  • The DeleteOptions, indicating the options for the Delete operation on each object, as described in the “Deleting a Resource” section.

  • The ListOptions, refining the collection of resources to delete, as described in the “Getting List of Resources” section.

Updating a Resource

To update a resource in the cluster, you can use the Update method for the resource you want to update. For example, use the following to update a Deployment in the project1 namespace:
updatedDep, err := clientset.
     AppsV1().
     Deployments("project1").
     Update(
          ctx,
          myDep,
          metav1.UpdateOptions{},
     )

The various options, to declare into the UpdateOptions structure when updating a resource, are the same as the options in CreateOptions described in the “Creating a Resource” section.

The possible errors specific to the Update operation are:

  • IsInvalid – this function indicates that the data passed into the structure is invalid.

  • IsConflict – this function indicates that the ResourceVersion incorporated into the structure (here myDep) is a version older than the one in the cluster. More information is available to in Chapter 2’s “Updating a Resource Managing Conflicts” section.

Using a Strategic Merge Patch to Update a Resource

You have seen in Chapter 2’s “Using a Strategic Merge Patch to Update a Resource” section how patching a resource with a strategic merge patch works. To sum up, you need to:

  • Use the Patch operation

  • Specify a specific value for the content-type header

  • Pass into the body the only fields you want to modify

Using the Client-go Library, you can use the Patch method for the resource you want to patch.
Patch(
     ctx context.Context,
     name string,
     pt types.PatchType,
     data []byte,
     opts metav1.PatchOptions,
     subresources ...string,
     ) (result *v1.Deployment, err error)

The patch type indicates whether you want to use a StrategicMerge patch (types.StrategicMergePatchType) or a merge patch (types.MergePatchType). These constants are defined in the k8s.io/apimachinery/pkg/types package.

The data field contains the patch you want to apply to the resource. You could write this patch data directly, like was done in Chapter 2, or you can use the following functions of the controller-runtime library to help you build this patch. This library is explored in more depth in Chapter 10).
import "sigs.k8s.io/controller-runtime/pkg/client"
func StrategicMergeFrom(
     obj Object,
     opts ...MergeFromOption,
) Patch

The StrategicMergeFrom function accepts a first parameter of type Object, representing any Kubernetes object. You will pass by this parameter the object you want to patch, before any change.

The function then accepts a series of options. The only accepted option at this time is the client.MergeFromWithOptimisticLock{} value. This value asks the library to add the ResourceVersion to the patch data, so the server will be able to check whether the resource version you want to update is the last one.

After you have created a Patch object by using the StrategicMergeFrom function, you can create a deep copy of the object you want to patch, then modify it. Then, when you are done updating the object, you can build the data for the patch with the dedicated Data method of the Patch object.

As an example, to build patch data for a Deployment, containing the ResourceVersion for optimistic lock, you can use the following code (createdDep is a Deployment structure that reflects a Deployment created in the cluster):
patch := client.StrategicMergeFrom(
     createdDep,
     pkgclient.MergeFromWithOptimisticLock{},
)
updatedDep := createdDep.DeepCopy()
updatedDep.Spec.Replicas = pointer.Int32(2)
patchData, err := patch.Data(updatedDep)
// patchData = []byte(`{
//   "metadata":{"resourceVersion":"4807923"},
//   "spec":{"replicas":2}
// }`)
patchedDep, err := clientset.
     AppsV1().Deployments("project1").Patch(
          ctx,
          "dep1",
          patch.Type(),
          patchData,
          metav1.PatchOptions{},
     )

Note that the MergeFrom and MergeFromWithOptions functions are also available, if you prefer to execute a Merge Patch instead.

The Type method of the Patch object can be used to retrieve the patch type, instead of using the constants in the type package. You can pass PatchOptions when calling the patch operation. The possible options are:

  • DryRun – this indicates which operations on the API server -side should be executed. The only available value is metav1.DryRunAll, indicating execution of all the operations except persisting the resource to storage.

  • Force – this option can be used only for Apply patch requests and must be unset when working with StrategicMergePatch or MergePatch requests.

  • FieldManager – this indicates the name of the field manager for this operation. This information will be used for future server-side Apply operations. This option is optional for StrategicMergePatch or MergePatch requests.

  • FieldValidation – this indicates how the server should react when duplicate or unknown fields are present in the structure. The following are the possible values:

    • metav1.FieldValidationIgnore – to ignore all duplicate or unknown fields

    • metav1.FieldValidationWarn – to warn when duplicate or unknown fields are present

    • metav1.FieldValidationStrict – to fail when duplicate or unknown fields are present

Note that the Patch operation accepts a subresources parameter. This parameter can be used to patch a subresource of the resource on which the Patch method is applied. For example, to patch the Status of a Deployment, you can use the value “status” for the subresources parameter.

The possible errors specific to the MergePatch operation are:

  • IsInvalid – this function indicates whether the data passed as a patch is invalid.

  • IsConflict – this function indicates whether the ResourceVersion incorporated into the patch (if you are using the Optimistic lock when building the patch data) is a version older than the one in the cluster. More information is available in Chapter 2’s “Updating a Resource Managing Conflicts” section.

Applying Resources Server-side with Patch

Chapter 2’s “Applying Resources Server-side” section described how a Server-side Apply patch works. To sum up, wee need to:

  • Use the Patch operation

  • Specify a specific value for the content-type header

  • Pass into the body the only fields you want to modify

  • Provide a fieldManager name

Using the Client-go Library, you can use the Patch method for the resource you want to patch. Note that you also can use the Apply method; see the next section, “Applying Resources Server-side with Apply.”
Patch(
     ctx context.Context,
     name string,
     pt types.PatchType,
     data []byte,
     opts metav1.PatchOptions,
     subresources ...string,
) (result *v1.Deployment, err error)

The Patch type indicates the type of patch, types.ApplyPatchType in this case, defined in the k8s.io/apimachinery/pkg/types package.

The data field contains the patch you want to apply to the resource. You can use the client.Apply value to build this data. This value implements the client.Patch interface, providing the Type and Data methods.

Note that you need to set the APIVersion and Kind fields in the structure of the resource you want to patch. Also note that this Apply operation also can be used to create the resource.

The Patch operation accepts a subresources parameter. This parameter can be used to patch a subresource of the resource on which the Patch method is applied. For example, to patch the Status of a Deployment, you can use the value “status” for the subresources parameter.
import "sigs.k8s.io/controller-runtime/pkg/client"
wantedDep := appsv1.Deployment{
     Spec: appsv1.DeploymentSpec{
          Replicas: pointer.Int32(1),
     [...]
}
wantedDep.SetName("dep1")
wantedDep.APIVersion, wantedDep.Kind =
     appsv1.SchemeGroupVersion.
          WithKind("Deployment").
          ToAPIVersionAndKind()
patch := client.Apply
patchData, err := patch.Data(&wantedDep)
patchedDep, err := clientset.
     AppsV1().Deployments("project1").Patch(
          ctx,
          "dep1",
          patch.Type(),
          patchData,
          metav1.PatchOptions{
          FieldManager: "my-program",
     },
     )
You can pass PatchOptions when calling the Patch operation. The following are the possible options:

  • DryRun – this indicates which operations on the API Server-side should be executed. The only available value is metav1.DryRunAll, indicating the execution of all the operations except persisting the resource to storage.

  • Force – this option indicates force Apply requests. It means the field manager for this request will acquire conflicting fields owned by other field managers.

  • FieldManager – this indicates the name of the field manager for this operation. This option is mandatory for Apply Patch requests.

  • FieldValidation – this indicates how the server should react when duplicate or unknown fields are present in the structure. The possible values are:

    • metav1.FieldValidationIgnore – to ignore all duplicate or unknown fields

    • metav1.FieldValidationWarn – to warn when duplicate or unknown fields are present

    • metav1.FieldValidationStrict – to fail when duplicate or unknown fields are present

The following are the possible errors specific to the ApplyPatch operation:

  • IsInvalid – this function indicates whether the data passed as a patch is invalid.

  • IsConflict – this function indicates whether some fields modified by the patch are in conflict because they are owned by another field manager. To resolve this conflict, you can use the Force option so that these fields will be acquired by the field manager of this operation.

Server-side Apply Using Apply Configurations

The previous section has shown how to execute a server-side Apply operation by using the Patch method. The disadvantage is that the data must be passed in JSON format, which can be error-prone.

Starting with Version 1.21, the Client-go Clientset provides an Apply method to execute the server-side Apply operation using typed structures. The following is the signature for the Apply method:
Apply(
     ctx context.Context,
     deployment *acappsv1.DeploymentApplyConfiguration,
     opts metav1.ApplyOptions,
) (result *v1.Deployment, err error)
The ApplyOptions structure defines the following options:

  • DryRun – this indicates which operations on the API Server-side should be executed. The only available value is metav1.DryRunAll, indicating execution of all the operations except persisting the resource to storage.

  • Force – this caller will reacquire the conflicting fields owned by other managers.

  • FieldManager – this is the name of the manager making the Apply operation. This value is required.

This signature of Apply is like the signatures of the Create or Update operations, except that a DeploymentApplyConfiguration object is expected, instead of a Deployment object.

As seen in Chapter 2’s “Applying Resources Server-side” section, the Apply operation permits several managers to work on the same resource, each manager owning a set of values in the resource specification.

For this reason, the data passed for the operation will not define all the fields, but only the fields the manager is responsible for. Some of the fields are required in the structures of resource definitions; it is not possible to use these structures for the Apply operation.

The Client-go Library introduces new structures, named Apply Configurations, in the applyconfigurations directory of the library, with all fields being optional, using pointers. This directory contains generated source code for all native resources of the Kubernetes API, with the same structure of it. For example, to access the structures needed to define the data for applying a Deployment from the apps/v1 group, you need to import the following package:
import (
     acappsv1 "k8s.io/client-go/applyconfigurations/apps/v1"
)

Note that, for the same reason as you want to define an alias for packages imported from the Kubernetes API Library (because most packages are named v1), you will want to use aliases when importing these packages. This book uses the same system as the API Library, prefixing the alias with ac to indicate it comes from the applyconfigurations directory.

Two possibilities are offered by the Client-go to build an ApplyConfiguration: from scratch or from an existing resource.

Building an ApplyConfiguration from Scratch

The first way to build an ApplyConfiguration is to build it from scratch. You first need to initialize the structure with the mandatory fields: kind, apiVersion, name, and namespace (if the resource is namespaced); this is done with a helper function that is provided by the related package. For example, for a Deployment resource:
deploy1Config := acappsv1.Deployment(
     "deploy1",
     "default",
)
The implementation of this function uses the following:
func Deployment(
     name string,
     namespace string,
) *DeploymentApplyConfiguration {
     b := &DeploymentApplyConfiguration{}
     b.WithName(name)
     b.WithNamespace(namespace)
     b.WithKind("Deployment")
     b.WithAPIVersion("apps/v1")
     return b
}
Then, you can specify the fields you want to manage. Helper functions in the form WithField() are provided to establish specific fields. For example, if your program is responsible only for setting the number of replicas for the deployment, the code will be:
deploy1Config.WithSpec(acappsv1.DeploymentSpec())
deploy1Config.Spec.WithReplicas(2)
Finally, you can call the Apply method. The complete code is the following:
import (
     acappsv1 "k8s.io/client-go/applyconfigurations/apps/v1"
)
deploy1Config := acappsv1.Deployment(
     "deploy1",
     "default",
)
deploy1Config.WithSpec(acappsv1.DeploymentSpec())
deploy1Config.Spec.WithReplicas(2)
result, err := clientset.AppsV1().
     Deployments("default").Apply(
          ctx,
          deploy1Config,
          metav1.ApplyOptions{
               FieldManager: "my-manager",
               Force:        true,
          },
     )

Building an ApplyConfiguration from an Existing Resource

The second way to build an ApplyConfiguration is to start from an existing resource in the cluster. Sometimes a program is not able to build the entire Apply Configuration in one place. For example, imagine your program is responsible for defining a container with a specific image for a Deployment in one place and also is responsible for setting the number of replicas in another place.

If the program defines the container and its image first, it will be marked as the owner of the container and its image. Then, if the program builds an ApplyConfiguration and sets only the number of replicas, without specifying the container and its image, the server-side Apply operation will try to delete the container. This is because the program designated the owner of this container, but it does not specify it anymore in the ApplyConfiguration.

A possibility would be to use diverse manager names for the various parts of the program. If you want to keep a single manager name, however, the packages in the applyconfigurations directory provide an ExtractResource() helper function to assist you in this case.

These functions will get as parameter a resource read from the cluster (with a Get or a List operation) and will build an ApplyConfiguration containing only the fields owned by the specified fieldManager. For example, the signature for the ExtractDeployment helper function is:
ExtractDeployment(
     deployment *apiappsv1.Deployment,
     fieldManager string,
) (*DeploymentApplyConfiguration, error)
The first steps are to read the deployment from the cluster, then extract the ApplyConfiguration from it. At this point, it will contain all the fields managed by the program (container and its image and the replicas). Then, you can specify the only fields you want to modify—that is, the replicas for in this example:
gotDeploy1, err := clientset.AppsV1().
     Deployments("default").Get(
          ctx,
          "deploy1",
     metav1.GetOptions{},
     )
if err != nil {
     return err
}
deploy1Config, err := acappsv1.ExtractDeployment(
     gotDeploy1,
     "my-manager",
)
if err != nil {
     return err
}
If deploy1Config.Spec == nil {
     deploy1Config.WithSpec(acappsv1.DeploymentSpec())
}
deploy1Config.Spec.WithReplicas(2)
result, err := clientset.AppsV1().
     Deployments("default").Apply(
          ctx,
          deploy1Config,
          metav1.ApplyOptions{
               FieldManager: "my-manager",
               Force:        true,
          },
     )

Watching Resources

Chapter 2’s “Watching Resources” section describes how the Kubernetes API can watch resources. Using the Client-go Library, you can use the Watch method for the resource you want to watch.
Watch(
     ctx context.Context,
     opts metav1.ListOptions,
) (watch.Interface, error)
This Watch method returns an object that implements the interface watch.Interface and providing the following methods:
import "k8s.io/apimachinery/pkg/watch"
type Interface interface {
     ResultChan() <-chan Event
     Stop()
}

The ResultChan method returns a Go channel (which you can only read) on which you will be able to receive all the events.

The Stop method will stop the Watch operation and close the channel that was received using ResultChan.

The watch.Event object, received using the channel, is defined as follows:
type Event struct {
     Type EventType
     Object runtime.Object
}

The Type field can get the values described earlier in Chapter 2’s Table 2-2, and you can find constants for these various values in the watch package: watch.Added, watch.Modified, watch.Deleted, watch.Bookmark, and watch.Error.

The Object field implements the runtime.Object interface, and its concrete type can be different depending on the value of the Type.

For a Type, other than Error, the concrete type of the Object will be the type of the resource you are watching (e.g., the Deployment type if you are watching for deployments).

For the Error type, the concrete type generally will be metav1.Status, but it could be any other type, depending on the resource you are watching. As an example, here is a code for watching Deployments:
import "k8s.io/apimachinery/pkg/watch"
watcher, err := clientset.AppsV1().
     Deployments("project1").
     Watch(
          ctx,
          metav1.ListOptions{},
     )
if err != nil {
     return err
}
for ev := range watcher.ResultChan() {
     switch v := ev.Object.(type) {
     case *appsv1.Deployment:
          fmt.Printf("%s %s ", ev.Type, v.GetName())
     case *metav1.Status:
          fmt.Printf("%s ", v.Status)
          watcher.Stop()
     }
}
The various options to be declared into the ListOptions structure, when watching resources, are the following:

  • LabelSelector, FieldSelector – this is used to filter the elements watched by label or by field. These options are detailed in the “Filtering the Result of a List” section.

  • Watch, AllowWatchBookmarks – the Watch option indicates that a Watch operation is running. This option is set automatically when executing the Watch method; you do not have to set it explicitly.

  • The AllowWatchBookmarks option asks the server to return Bookmarks regularly. The use of bookmarks is described in Chapter 2’s, “Allowing Bookmarks to Efficiently Restart a Watch Request” section.

  • ResourceVersion, ResourceVersionMatch – this indicates at which version of the List of resources you want to start the Watch operation.

  • Note that, when receiving a response of a List operation, a ResourceVersion value is indicated for the list element itself, as well as ResourceVersion values for each element of the list. The ResourceVersion to indicate in the Options refers to the ResourceVersion of the list.

  • The ResourceVersionMatch option is not used for Watch operations. For a Watch operations do the following:

    • When ResourceVersion is not set, the API will start the Watch operation at the most recent list of resources. The channel first receives ADDED events to declare the initial state of the resource, followed by other events when changes occur on the cluster.

    • When ResourceVersion is set to a specific version, the API will start the Watch operation at the specified version of the list of resources. The channel will not receive ADDED events to declare the initial state of the resource, but only events when changes occur on the cluster after this version (which can be events that occurred between the specified version and the time you run the Watch operation).

    • A use case is to watch for the deletion of a specific resource. For this, you can:

      1. List the resources, including the one you want to delete, and save the ResourceVersion of the received List.

      2. Execute a Delete operation on the resource (the deletion being asynchronous, the resource probably will not be deleted when the operation terminates).

      3. Start a Watch operation by specifying the ResourceVersion received in Step 1. Even if the deletion occurs between Steps 2 and 3, you are guaranteed to receive the DELETED event.

    • When ResourceVersion is set to “0,” the API will start the Watch operation at any list of resources. The channel first receives ADDED events to declare the initial state of the resource, followed by other events when changes occur on the cluster after this initial state.

    • You have to take special care when using this semantic because the Watch operation will generally start with the most recent version; however, starting with an older version is possible.

  • TimeoutSeconds – this limits the duration of the request to the indicated number of seconds.

  • Limit, Continue – this is used for paginating the result of a List operation. These options are not supported for a Watch operation.

Note that, if you specify a nonexisting namespace for a Watch operation, you will not receive a NotFound error.

Also note that, if you specify an expired ResourceVersion, you will not receive an error when calling the Watch method, but will get an ERROR event containing a metav1.Status object indicating a Reason with a value metav1.StatusReasonExpired.

The metav1.Status is the base object used to build the errors returned by calls using the Clientset. You will be able to learn more in the “Errors and Statuses” section.

Errors and Statuses

As Chapter 1 has shown, the Kubernetes API defines Kinds for exchanging data with the caller. For the moment, you should consider that Kinds are related to the resources, either the Kind having the singular name of the resource (e.g., Pod), or the Kind for a list of resources (e.g., PodList). When an API operation returns neither a resource nor a list or resources, it uses a common Kind, metav1.Status, to indicate the status of the operation.

Definition of the metav1.Status Structure

The metav1.Status structure is defined as follows:
type Status struct {
     Status      string
     Message     string
     Reason      StatusReason
     Details     *StatusDetails
     Code        int32
}

  • Status – this indicates the status of the operation and is either metav1.StatusSuccess or metav1.StatusFailure.

  • Message – this is a free form human-readable description of the status of the operation.

  • Code – this indicates the HTTP status code returned for the operation.

  • Reason – this indicates why the operation is in the Failure status. A Reason is related to a given HTTP status code. The defined Reasons are:

    • StatusReasonBadRequest (400) – this request itself is invalid. This is different from StatusReasonInvalid, which indicates that the API call could possibly succeed, but the data was invalid. A request replying StatusReasonBadRequest can never succeed, whatever the data.

    • StatusReasonUnauthorized (401) – the authorization credentials are missing, incomplete, or invalid.

    • StatusReasonForbidden (403) – the authorization credentials are valid, but the operation on the resource is forbidden for these credentials.

    • StatusReasonNotFound (404) – the requested resource or resources cannot be found.

    • StatusReasonMethodNotAllowed (405) – the operation requested in the resource is not allowed because it id not implemented. A request replying StatusReasonMethodNotAllowed can never succeed, whatever the data.

    • StatusReasonNotAcceptable (406) – none of the Accept types indicated in the Accept header by the client is possible. A request replying StatusReasonNotAcceptable can never succeed, whatever the data.

    • StatusReasonAlreadyExists (409) – the resource being created already exists.

    • StatusReasonConflict (409) – the request cannot be completed because of a conflict—for example, because the operation tries to update a resource with an older resource version, or because a precondition in a Delete operation is not respected.

    • StatusReasonGone (410) – an item is no longer available.

    • StatusReasonExpired (410) – the content has expired and is no longer available—for example, when executing a List or Watch operation with an expired resource version.

    • StatusReasonRequestEntityTooLarge (413) – the request entity is too large.

    • StatusReasonUnsupportedMediaType (415) – the content type indicated in the Content-Type header is not supported for this resource. A request replying StatusReasonUnsupportedMediaType can never succeed, whatever the data.

    • StatusReasonInvalid (422) – the data sent for a Create or Update operation is invalid. The Causes field enumerates the invalid fields of the data.

    • StatusReasonTooManyRequests (429) – the client should wait at least the number of seconds specified in the field RetryAfterSeconds of the Details field before performing an action again.

    • StatusReasonUnknown (500) – the server did not indicate any reason for the failure.

    • StatusReasonServerTimeout (500) – the server can be reached and understand the request, but cannot complete the action in a reasonable time. The client should retry the request after the number of seconds specified in the field RetryAfterSeconds of the Details field.

    • StatusReasonInternalError (500) – an internal error occurred; it is unexpected and the outcome of the call is unknown.

    • StatusReasonServiceUnavailable (503) – the request was valid, but the requested service is unavailable at this time. Retrying the request after some time might succeed.

    • StatusReasonTimeout (504) – the operation cannot be completed within the time specified by the timeout in the request. If the field RetryAfterSeconds of the Details field is specified, the client should wait this number of seconds before performing the action again.

  • Details – these can contain more details about the reason, depending on the Reason field.

  • The type StatusDetails of the Details field is defined as follows:

type StatusDetails struct {
     Name string
     Group string
     Kind string
     UID types.UID
     Causes []StatusCause
     RetryAfterSeconds int32
}

  • The Name, Group, Kind, and UID fields indicate, if specified, which resource is impacted by the failure.

  • The RetryAfterSeconds field, if specified, indicates how many seconds the client should wait before performing an operation again.

  • The Causes field enumerates the causes of the failure. When performing a Create or Update operation resulting in a failure with a StatusReasonInvalid reason, the Causes field enumerates the invalid fields and the type of error for each field.

  • The StatusCause type of the Causes field is defined as follows:

type StatusCause struct {
     Type       CauseType
     Message    string
     Field      string
}

Error Returned by Clientset Operations

This chapter earlier contained a description of the various operations provided by the Clientset that the operations generally return an error, and that you can use functions from the errors package to test the cause of the error—for example, with the function IsAlreadyExists.

The concrete type of these errors is errors.StatusError, defined as:
type StatusError struct {
     ErrStatus metav1.Status
}
It can be seen that this type includes only the metav1.Status structure that has been explored earlier in this section. Functions are provided for this StatusError type to access the underlying Status.

  • Is<ReasonValue>(err error) bool – one for each Reason value enumerated earlier in this section, indicating whether the error is of a particular status.

  • FromObject(obj runtime.Object) error – When you are receiving a metav1.Status during a Watch operation, you can build a StatusError object using this function.

  • (e *StatusError) Status() metav1.Status – returns the underlying Status.

  • ReasonForError(err error) metav1.StatusReason – returns the Reason of the underlying Status.

  • HasStatusCause(err error, name metav1.CauseType) bool – this indicates whether an error declares a specific cause with the given CauseType.

  • StatusCause(err error, name metav1.CseType) (metav1.StatusCause, bool) – returns the cause for the given causeType if it exists, or false otherwise.

  • SuggestsClientDelay(err error) (int, bool) – this indicates whether the error indicates a value in the RetryAfterSeconds field of the Status and the value itself.

RESTClient

Earlier in this chapter in the “Using the Clientset” section, you can get a REST client for each group/version of the Kubernetes API. For example, the following code returns the REST client for the Core/v1 group:
restClient := clientset.CoreV1().RESTClient()
The restClient object implements the interface rest.Interface, defined as:
type Interface interface {
     GetRateLimiter()           flowcontrol.RateLimiter
     Verb(verb string)          *Request
     Post()                     *Request
     Put()                      *Request
     Patch(pt types.PatchType)  *Request
     Get()                      *Request
     Delete()                   *Request
     APIVersion()                schema.GroupVersion
}

In this interface, you can see the generic method, Verb, and the helper methods Post, Put, Patch, Get, and Delete returning a Request object.

Building the Request

The Request structure contains only private fields, and it provides methods to personalize the Request. As shown in Chapter 1, the form of the path for a Kubernetes resource or subresource (some segments may be absent depending on the operation and resource) is the following:
/apis/<group>/<version>
    /namespaces/<namesapce_name>
        /<resource>
            /<resource_name>
                /<subresource>
The following methods can be used to build this path. Note that the <group> and <version> segments are fixed, as the REST client is specific to a group and version.

  • Namespace(namespace string) *Request; NamespaceIfScoped(namespace string, scoped bool) *Request – these indicate the namespace of the resource to query. NamespaceIfScoped will add the namespace part only if the request is marked as scoped.

  • Resource(resource string) *Request – this indicates the resource to query.

  • Name(resourceName string) *Request – this indicates the name of the resource to query.

  • SubResource(subresources ...string) *Request – this indicates the subresource of the resource to query.

  • Prefix(segments ...string) *Request; Suffix(segments ...string) *Request – add segments to the beginning or end of the request path. The prefix segments will be added before the “namespace” segment. The suffix segments will be added after the subresource segment. New calls to these methods will add prefixes and suffixes to the existing ones.

  • AbsPath(segments ...string) *Request – resets the prefix with the provided segments.

The following methods complete the request with query parameters, body, and headers:

  • Param(paramName, s string) *Request – adds a query parameter with the provided name and value.

  • VersionedParams(

    obj runtime.Object,

    codec runtime.ParameterCodec,

    ) *Request

  • Adds a series of parameters, extracted from the object obj. The concrete type of obj is generally one of the structures ListOptions, GetOptions, DeleteOptions, CreateOptions, PatchOptions, ApplyOptions, UpdateOptions, or TableOptions.

  • The codec is generally the parameter codec provided by the scheme package of the client-go library: scheme.ParameterCodec.

  • SpecificallyVersionedParams(

    obj runtime.Object,

    codec runtime.ParameterCodec,

    version schema.GroupVersion,

    ) *Request

  • With VersionedParams, the object will be encoded using the group and version of the REST Client. With SpecificallyVersionedParams, you can indicate a specific group and version.

  • SetHeader(key string, values ...string) *Request – sets values for the specified header for the request. If the header with this key is already defined, it will be overwritten.

  • Body(obj interface{}) *Request – sets the body content of the request, based on obj. The obj can be of different type:

    • string – the file with the given name will be read and its content used as body

    • []byte – the data will be used for the body

    • io.Reader – the data read from the reader will be used for the body

    • runtime.Object – the object will be marshaled and the result used for the body. The Content-Type header will be set to indicate in which type the object is marshaled (json, yaml, etc.).

Other methods can be used to configure the technical properties of the request:

  • BackOff(manager BackoffManager) *Request – sets a Backoff manager for the request. The default backoff manager is a rest.NoBackoff manager provided by the rest package, which won’t wait before to execute a new request after a failing request.

  • The rest package provides another backoff manager, rest.URLBackoff, which will wait before to retry a new request on a server which replied previously with a 5xx error.

  • You can build and use a rest.URLBackoff object with:
    request.BackOff(&rest.URLBackoff{
         Backoff: flowcontrol.NewBackOffWithJitter(
              1*time.Second,
              30*time.Second,
              0.1,
         ),
    })

  • If you get continuous 5xx errors calling the Kubernetes API, the RESTClient will add exponential delays between the requests, here 1 second, then 2 seconds, then 4 seconds, and so on, capping the delays to 30 seconds, and adding a jitter of maximum 10% to delays, until the server replies with a non-5xx status.

  • If you do not want to add jitter:
    request.BackOff(&rest.URLBackoff{
         Backoff: flowcontrol.NewBackOff(
              1*time.Second,
              30*time.Second,
         ),
    })

  • Note that, instead of using this code to declare an exponential backoff for each requests, you can declare the environment variables KUBE_CLIENT_BACKOFF_BASE=1 and KUBE_CLIENT_BACKOFF_DURATION=30 to have a similar behavior (without adding jitter) when running programs using the client-go library for all requests.

  • The parameter accepted by BackOff() being an interface, you can write your own BackOff manager, by implementing the rest.BackoffManager interface:
    type BackoffManager interface {
         UpdateBackoff(
              actualUrl *url.URL,
              err error,
              responseCode int,
         )
         CalculateBackoff(
              actualUrl *url.URL,
         ) time.Duration
         Sleep(d time.Duration)
    }
  • For example, to implement a linear backoff on 5xx errors (working when calling only one host):
    type MyLinearBackOff struct {
         next time.Duration
    }
    func (o *MyLinearBackOff) UpdateBackoff(
         actualUrl *url.URL,
         err error,
         responseCode int,
    ) {
         if responseCode > 499 {
              o.next += 1 * time.Second
              return
         }
         o.next = 0
    }
    func (o *MyLinearBackOff) CalculateBackoff(
         actualUrl *url.URL,
    ) time.Duration {
         return o.next
    }
    func (o *MyLinearBackOff) Sleep(
         d time.Duration,
    ) {
         time.Sleep(d)
    }

  • Throttle(limiter flowcontrol.RateLimiter) *Request – throttling will limit the number of requests per second the RESTClient can execute.

  • By default, a Token Bucket Rate Limiter is used, with a QPS of 5 queries/second and a Burst of 10.

  • This means that the RESTClient can make a maximum of 5 requests per second, plus a bonus (bucket) of 10 requests that it can use at any time. The bucket can be refilled at the rate of QPS, in this case the bucket is refilled with 5 tokens per second, the maximal size of the bucket remaining 10.

  • You can build and use a flowcontrol.tokenBucketRateLimiter object with:
    request.Throttle(
         flowcontrol.NewTokenBucketRateLimiter(5.0, 10),
    )

  • In this example, a Token Bucket Rate Limiter with a rate of 5.0 queries/second (QPS) and a burst of 10 will be used for the Request.

  • Note that you can obtain the same behavior for all requests by setting the QPS and Burst of the Config used to create the RESTClient.

  • The default Rate Limiter for a Request is inherited from the Config used to create the clientset. You can change the Rate Limiter for all the requests by setting the Rate Limiter in the Config.

  • MaxRetries(maxRetries int) *Request – this indicates the number of retries the RESTClient will perform when the Request receives a response with a Retry-After header and a 429 status code (Too Many Requests).

  • The default value is 10, meaning that the Request will be performed a maximum of 11 times before to return with an error.

  • Timeout(d time.Duration) *Request – this indicates the number of seconds the RESTClient will wait for a response to the Request before returning an error. The default Timeout for a Request is inherited from the HTTPClient used to build the clientset.

  • WarningHandler(handler WarningHandler) *Request – the API server can return Warnings for a Request using a specific header (the “Warning” header). By default, the warnings with be logged, and the rest package provides several built-in implementations of handlers:

    • WarningLogger{} logs warnings (the default)

    • NoWarning{} suppresses warnings

    • NewWarningWriter() writes warnings to the provided writer. Options can be specified:

      • Deduplicate – true to write a given warning only once

      • Color – true to write warning in Yellow color

  • You can write your own implementation of a WarningHandler by implementing this interface:
    type WarningHandler interface {
         HandleWarningHeader(
              code int,
              agent string,
              text string,
         )
    }

  • Note that you can set a default Warning Handler for all the requests from all clients by calling the following global function:

rest.SetDefaultWarningHandler(l WarningHandler)

Executing the Request

Once the Request is built, we can execute it. The following methods on a Request object can be used:

  • Do(ctx context.Context) Result – this executes the Request and return a Result object. We will see in the next section how to exploit this Result object.

  • Watch(ctx context.Context) (watch.Interface, error) – this executes a Watch operation on the requested location, and returns an object implementing the interface watch.Interface, used to receive events. You can see the section “Watching Resources” of this chapter to see how to use the returned object.

  • Stream(ctx context.Context) (io.ReadCloser, error) – this executes the Request and Stream the result body through a ReadCloser.

  • DoRaw(ctx context.Context) ([]byte, error) – this executes the Request and return the result as an array of bytes.

Exploiting the Result

When you execute the Do() method on a Request, the method returns a Result object.

The Result structure does not have any public field. The following methods can be used to get information about the result:

  • Into(obj runtime.Object) error – this decodes and stores the content of the result body into the object, if possible. The concrete type of the object passed as parameter must match the kind defined in the body. Also return the error executing the request.

  • Error() error – this returns the error executing the request. This method is useful when executing a request returning no body content.

  • Get() (runtime.Object, error) – this decodes and returns the content of the result body as an object. The concrete type of the returned object will match the kind defined in the body. Also return the error executing the request.

  • Raw() ([]byte, error) – this returns the body as an array of bytes, and the error executing the request.

  • StatusCode(statusCode *int) Result – this stores the status code into the passed parameter, and return the Result, so the method can be chained.

  • WasCreated(wasCreated *bool) Result – this stores a value indicating if the resource requested to be created has been created successfully, and return the Result, so the method can be chained.

  • Warnings() []net.WarningHeader – this returns the list of Warnings contained in the Result.

Getting Result as a Table

You have seen in Chapter 2’s “Getting Result as a Table” section that it is possible to get the result of a List request as a list of columns and rows so as to display the information in a tabular representation. For this, you have to make a List operation and specify a specific Accept header.

Here is the code to list the pods of the project1 namespace, in a tabular representation, using the RESTClient:
import (
     metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
restClient := clientset.CoreV1().RESTClient() ➊
req := restClient.Get().
     Namespace("project1").          ➋
     Resource("pods").               ➌
     SetHeader(                    ➍
     "Accept",
     fmt.Sprintf(
          "application/json;as=Table;v=%s;g=%s",
          metav1.SchemeGroupVersion.Version,
          metav1.GroupName
     ))
var result metav1.Table          ➎
err = req.Do(ctx).               ➏
     Into(&result)               ➐
if err != nil {
     return err
}
for _, colDef := range result.ColumnDefinitions {     ➑
     // display header
}
for _, row := range result.Rows {          ➒
     for _, cell := range row.Cells {          ➓
          // display cell
     }
}

  • ➊ Get the RESTClient for the core/v1 group

  • ➋ Indicate the namespace from which to list the resources (here project1)

  • ➌ Indicate the resources to list (here pods)

  • ➍ Set the required header to get the result as tabular information

  • ➎ Prepare a variable of type metav1.Table to store the result of the request

  • ➏ Execute the request

  • ➐ Store the result in the metav1.Table object

  • ➑ Range over the definitions of the columns returned to display the table header

  • ➒ Range over the rows of the table returned to display the row of data containing information about a specific pod

  • ➓ Range over the cells of the row to display them

Discovery Client

The Kubernetes API provides endpoints to discover the resources served by the API. kubectl is using these endpoints to display the result of the command kubectl api-resources (Figure 6-1).

A screenshot depicts the list of kubectl A P I resources. The columns are labeled name, short names, A P I version, namespaced, and kind.

Figure 6-1

kubectl api-resources

The client can be obtained either by calling the Discovery() method on a Clientset (see how to obtain a Clientset in Chapter 6’s “Getting a Clientset” section), or by using functions provided by the discovery package.
import "k8s.io/client-go/discovery"
All these function, expect a rest.Config, as a parameter. You can see in Chapter 6’s “Connecting to the Cluster” section how to get such a rest.Config object.

  • NewDiscoveryClientForConfig( 

    c *rest.Config,

    ) (*DiscoveryClient, error)

    – this returns a DiscoveryClient, using the provided rest.Config

  • NewDiscoveryClientForConfigOrDie(

    c *rest.Config,

    ) *DiscoveryClient

  • Similar to the previous one, but panics in case of error, instead of returning the error. This function can be used with a hard-coded config whose we want to assert the validity.

  • NewDiscoveryClientForConfigAndClient(

    c *rest.Config,

    httpClient *http.Client,

    ) (*DiscoveryClient, error)

    – this returns a DiscoveryClient, using the provided rest.Config, and the provided httpClient.

  • The previous function NewDiscoveryClientForConfig uses a default HTTP Client built with the function rest.HTTPClientFor. If you want to personalize the HTTP Client before building the DiscoveryClient, you can use this function instead.

RESTMapper

You have seen in Chapter 5’s “RESTMapper” section that the API Machinery provides a concept of RESTMapper, used to map between REST Resources and Kubernetes Kinds.

The API Machinery also provides a default implementation of the RESTMapper, the DefaultRESTMapper, for which the group/version/kinds must be added manually.

The Client-go Library provides several implementations of a RESTMapper, taking advantage of the Discovery client to provide the list of group/version/kinds and resources.

PriorityRESTMapper

The PriorityRESTMapper is gets all the groups served by the Kubernetes API, with the help of the Discovery client, and takes care about the multiple versions that could be part of given groups and the preferred version for each group, to return the preferred version.

A restmapper.PriorityRESTMapper object is obtained by calling the function restmapper.NewDiscoveryRESTMapper:
import "k8s.io/client-go/restmapper"
func NewDiscoveryRESTMapper(
     groupResources []*APIGroupResources,
) meta.RESTMapper
The groupResources parameter can be built with the function restmapper.GetAPIGroupResources:
func GetAPIGroupResources(
     cl discovery.DiscoveryInterface,
) ([]*APIGroupResources, error)
Here is the code necessary to build a PriorityRESTMapper:
import "k8s.io/client-go/restmapper"
discoveryClient := clientset.Discovery()
apiGroupResources, err :=
     restmapper.GetAPIGroupResources(
          discoveryClient,
     )
if err != nil {
     return err
}
restMapper := restmapper.NewDiscoveryRESTMapper(
     apiGroupResources,
)

You can now use the RESTMapper as defined in Chapter 5’s “RESTMapper” section.

DeferredDiscoveryRESTMapper

The DeferredDiscoveryRESTMapper uses a PriorityRESTMapper internally, but will wait for the first request to initialize the RESTMapper.
func NewDeferredDiscoveryRESTMapper(
     cl discovery.CachedDiscoveryInterface,
) *DeferredDiscoveryRESTMapper

The function NewDeferredDiscoveryRESTMapper is used to build such a RESTMapper, and it gets an object which implements the the discovery.CachedDiscoveryInterface to get a Cached Discovery Client.

The Client-go Library provides an implementation for this interface, which is returned by the function memory.NewMemCacheClient.
func NewMemCacheClient(
     delegate discovery.DiscoveryInterface,
) discovery.CachedDiscoveryInterface
Here is the code necessary to build a DeferredDiscoveryRESTMapper:
import "k8s.io/client-go/restmapper"
discoveryClient := clientset.Discovery()
defRestMapper :=
     restmapper.NewDeferredDiscoveryRESTMapper(
          memory.NewMemCacheClient(discoveryClient),
     )

You can now use the RESTMapper as defined in Chapter 5’s “RESTMapper” section.

Conclusion

In this chapter, you have seen how to connect to a cluster and how to obtain a Clientset. It is a set of clients, one for each Group-Version, with which you can execute operations on resources (get, list, create, etc.).

You also have covered the REST client, internally used by the Clientset, and that the developer can use to build more specific requests. Finally, the chapter covered the Discovery client, used to discover the resources served by the Kubernetes API in a dynamic way.

The next chapter covers how to test applications written with the Client-go Library, using the fake implementations of the clients provided by it.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.137.181.52