In the previous chapter, you have seen how to declare a new custom resource to be served by the Kubernetes API using CustomResourceDefinition resources, and how to create new instances of this custom resource using kubectl. But for the moment, you do not have any Go library that allows you to work with instances of custom resources.
Generating code for a dedicated Clientset for the custom resource.
Using the unstructured package of the API Machinery Library and the dynamic package of the Client-go Library.
Generating a Clientset
The repository https://github.com/kubernetes/code-generator contains the Go code generators. Chapter 3’s “Content of a Package” section contains a very quick overview of these generators, where the content of the API Library was explored.
To use these generators, you will need to first write the Go structures for the kinds defined by the custom resource. In this example, you will write structures for the MyResource and MyResourceList kinds.
To stick with the organization found in the API Library, you will write these types in a types.go file, placed in the directory pkg/apis/<group>/<version>/.
Likewise, to work correctly with the generators, the root directory of your project must be in a directory named after the Go package after the Go package of your project. For example, if the package of the project is github.com/myid/myproject (defined in the first line of the project’s go.mod file), the root directory of the project must be in the directory github.com/myid/myproject/.
deepcopy-gen – this will generate a DeepCopyObject() method for each Kind structure, which is needed for these types to implement the runtime.Object interface.
client-gen – this will generate the clientset for the group/version.
Using deepcopy-gen
Installing deepcopy-gen
You can use either the @latest tag to use the latest revision of the Kubernetes code or select a specific version.
Adding Annotations
The deepcopy-gen generator needs annotations to work. It first needs the // +k8s:deepcopy-gen=package annotation to be defined at the package level. This annotation asks deepcopy-gen to generate deepcopy methods for all structures of the package.
Running deepcopy-gen
The generator needs a file that contains the text (generally the license) added at the beginning of the generated files. For this, you can create an empty file (or with the content you prefer, do not forget that the text should be Go comments) named hack/boilerplate.go.txt.
Note the “../../..” as output-base. It will place the output base in the directory from which you created the directory for the project:
$ mkdir -p github.com/myid/myresource-crd
You will need to adapt this to the number of subdirectories you created if different from the three.
Using client-gen
Installing client-go
You can use either the @latest tag to use the latest revision of the Kubernetes code or select a specific version if you want to run the command in a reproducible way.
Adding Annotations
// +genclient (with no option) will ask client-gen to generate a Clientset for a namespaced resource.
// +genclient:nonNamespaced will generate a Clientset for a non-namespaced resource.
+genclient:onlyVerbs=create,get will generate these verbs only, instead of generating all verbs by default.
+genclient:skipVerbs=watch will generate all verbs except these ones, instead of all verbs by default.
+genclient:noStatus – if a Status field is present in the annotated structure, an updateStatus function will be generated. With this option, you can disable the generation of the updateStatus function (note that it is not necessary if the Status field does not exist).
Adding AddToScheme Function
The generated code relies on an AddToScheme function defined in the package of the resource. To stick with the convention found in the API Library, you will write this function in a register.go file, placed in the directory pkg/apis/<group>/<version>/.
Running client-go
The client-gen needs a file containing the text (generally the license) added at the beginning of generated files. You will use the same file as with deepcopy-gen: hack/boilerplate.go.txt.
Note the “../../..” as output-base. It will place the output base in the directory from which you created the directory for the project:
$ mkdir -p github.com/myid/myresource-crd
You will need to adapt this to the number of subdirectories you created if different from the three.
Note that you will need to run this command again when you update the definition of your custom resource. It is recommended to place this command in a Makefile to automatically run it every time the files defining the custom resource are modified.
Using the Generated Clientset
Using the Generated fake Clientset
The client-gen tool also generates a fake Clientset that you can use the same way you use the fake Clientset from the Client-go Library. For more information, see Chapter 7’s “Fake Clientset” section.
Using the Unstructured Package and Dynamic Client
These types can be used to represent any Kubernetes Kind, either a list or a nonlist.
The Unstructured Type
Using this type, it is possible to define any Kubernetes resource without having to use the typed structures (e.g., the Pod structure, found in the API Library).
Getters and Setters methods are defined for this type to access generic fields from the TypeMeta and ObjectMeta fields, common to all structures representing Kubernetes Kinds.
Getters and Setters to Access TypeMeta Fields
The APIVersion and Kind Getters/Setters can be used to directly get and set the apiVersion and Kind fields of the TypeMeta.
Getters and Setters to Access ObjectMeta Fields
Getters and Setters are defined for all fields of the ObjectMeta structure. The details of the structure can be found in Chapter 3’s “The ObjectMeta Fields” section.
As an example, the getter and setter to access the Name field are GetName() string and SetName(name string).
Methods for Creating and Converting
NewEmptyInstance() runtime.Unstructured – this returns a new instance with only the apiVersion and kind fields copied from the receiver.
MarshalJSON() ([]byte, error) – this returns the JSON representation of the receiver.
UnmarshalJSON(b []byte) error – this populates the receiver with the passed JSON representation.
UnstructuredContent() map[string]interface{} – this returns the value of the Object field of the receiver.
SetUnstructuredContent(
content map[string]interface{},
)
– this sets the value of the Object field of the receiver,].
IsList() bool – this returns true if the receiver describes a list, by checking if an items field exists, and is an array.
ToList() (*UnstructuredList, error) – this converts the receiver to an UnstructuredList.
Helpers to Access Non-meta Fields
The following helpers can be used to get and set the value of specific fields in the Object field of an Unstructured instance.
Note that these helpers are functions, and not methods on the Unstructured type.
A first parameter obj of type map[string]interface{} used to pass the Object field of the Unstructured instance,
A last parameter fields of type …string used to pass the keys to navigate into the object. Note that no array/slice syntax is supported.
The value of the requested field, if possible
A boolean indicating if the requested field has been found
An error if the field has been found but is not of the requested type
RemoveNestedField – this removes the requested field
NestedFieldCopy, NestedFieldNoCopy – this returns a copy or the original value of the requested field
NestedBool, NestedFloat64, NestedInt64, NestedString, SetNestedField - this gets and sets bool / float64 / int64 / string field
NestedMap, SetNestedMap - this gets and sets fields of type map[string]interface{}
NestedSlice, SetNestedSlice - this gets and sets fields of type []interface{}
NestedStringMap, SetNestedStringMap - this gets and sets fields of type map[string]string
NestedStringSlice, SetNestedStringSlice - this gets and sets fields of type []string
Example
The UnstructuredList Type
Getters and Setters to Access TypeMeta Fields
The APIVersion and Kind Getters/Setters can be used to directly get and set the apiVersion and kind fields of the TypeMeta.
Getters and Setters to Access ListMeta Fields
Methods for Creating and Converting
NewEmptyInstance() runtime.Unstructured – this creates a new instance of an Unstructured object, using the apiVersion and kind copied from the List receiver.
MarshalJSON() ([]byte, error) – this returns the JSON representation of the receiver.
UnmarshalJSON(b []byte) error – this populates the receiver with the passed JSON representation.
UnstructuredContent() map[string]interface{} SetUnstructuredContent(content map[string]interface{})– this gets the value of the Object field of the receiver.
EachListItem(fn func(runtime.Object) error) error – this executes the fn function for each item of the list.
Converting Between Typed and Unstructured Objects
❶ Get the converter
❷ Convert an unstructured object (defining a Pod) to a typed Pod
❸ Convert a typed Pod to an unstructured object
The Dynamic Client
As you have seen in Chapter 6, the Client-go provides clients with the ability to work with the Kubernetes API: the Clientset to access typed resources, the REST client to make low-level REST calls to the API, and the Discovery client to get information about the resources served by the API.
It provides another client, the dynamic client, to work with untyped resources, described with the Unstructured type.
Getting the dynamic Client
func NewForConfig(c *rest.Config) (Interface, error) – this function returns a dynamic client, using the provided rest.Config built with one of the methods seen in chapter 6, section “Connecting to the cluster”.
func NewForConfigOrDie(c *rest.Config) Interface – this function is like the previous one, but panics in case of error, instead of returning the error. This function can be used with a hard-coded configuration, for which you want to assert the validity.
NewForConfigAndClient(–
c *rest.Config,
httpClient *http.Client,
) (Interface, error)
- this function returns a dynamic client, using the provided rest.Config, and the provided httpClient.
The previous function NewForConfig uses a default HTTP Client built with the function rest.HTTPClientFor. If you want to personalize the HTTP Client before building the dynamic client, you can use this function instead.
Working with the dynamic Client
It is possible to chain the call to Resource(gvr) with a call to the method Namespace(ns), returning an object implementing the ResourceInterface. Or, if Resource(gvr) is not chained with the Namespace(ns) method, it also implements the ResourceInterface.
Thanks to this, after calling Resource(gvr), you can either chain with Namespace(ns), if the resource described by gvr is a namespaced resource, and you want to define on which namespace to work, or else you can omit this call for non-namespaced resources if you want to make a cluster-wide operation.
Typed objects (e.g., corev1.Pod) for Clientset are replaced by Unstructured for the dynamic client.
A subresource… parameter is present for dynamic methods, whereas Clientset provides specific methods for subresources.
You can refer to Chapter 6 for more about the behavior of the various operations.
Example
The fake dynamic Client
As you have seen in Chapter 7, the Client-go Library provides fake implementations for the Clientset, the discovery client, and the REST client. The library also provides a fake implementation for the dynamic client.
Like other fake implementations, you can register reactors using PrependReactor and similar methods on the dynamicClient.Fake object, and you can inspect Actions of the dynamicClient.Fake object after the test is executed.
The objects parameter indicates the resources to create in the fake cluster when creating the fake client. As an example, here is a test for a function using the dynamic client.
Conclusion
This chapter has explored various solutions to work with custom resources in Go.
A first solution is to generate Go code based on the definition of a custom resource, to generate a Clientset dedicated to this specific custom resource definition, so you can work with custom resource instances the same as you have been working with for native Kubernetes resources.
Another solution is to work with the dynamic client from the Client-go Library and to rely on the Unstructured type to define the custom resources.