14.1.1 Liveness and Readiness

A liveness check determines whether the container needs to be restarted. If a liveness check fails, Kubernetes restarts the pod.

A Kubernetes service selects pods with criteria. Only pods where the readiness check succeeds are included in the service.

A service is often used by different methods of load-balancing traffic to decide where to send the traffic. For a service used in this way, a failed readiness check means no traffic is sent to the relevant pod. Ideally, for web-based applications, a dedicated endpoint or several endpoints is configured to determine health and readiness.

While it is possible to configure liveness checks to wait for an initial boot-up time, this is not the best way to do so.

If an application has a slow start-up time, the best way to handle it is to start responding to liveness early, as the application does any pre-computing or initial configuration, and configure a readiness check to indicate the initial configuration is done, and the application is ready to receive traffic.

Slow start-up is sometimes caused by needing some initial data. This data might come from a slow data service or might need slow pre-processing before being in the proper shape to respond to queries. In the following example, for simplicity, we avoid a back-end service altogether and focus on an application that requires a heavy computation before it starts.

There are faster ways to compute Fibonacci numbers. This function is useful as a stand-in for a slow computation that is harder to optimize.

On a test machine, this took 10 to 20 seconds.

When configuring a liveness check, it is useful if it has a short timeout. After the application is started, something has gone seriously wrong if the liveness check takes more than a second.

However, configuring a liveness check with a 1-second timeout results in a pod that goes into a Kubernetes crash loop. This is a case where a better start-up, combined with separate readiness and liveness checks, can be useful.

The application in this example uses the klein web framework, which uses Twisted as its underlying networking framework.

Notice that this code fails if the cache is not initialized yet. Because of this, the pod should be configured with a readiness check.

Note that while / could be configured as the readiness endpoint since it fails if the result is missing, this is suboptimal. If it does not fail, it does some extra computation: creating a big string and sending it to the client.

In a more realistic example, this could be even worse. For example, this might call back to a back-end service or have some undesirable side effect.

In this case, failure is accomplished by raising an exception. In a more sophisticated readiness code, it is sometimes worthwhile to send a specific HTTP error code and give more details. This can help in troubleshooting.

Finally, since neither of these endpoints would serve as a useful liveness check, the application needs a dedicated endpoint. Again, in general, it is useful to give applications dedicated health checks.

This pattern of starting up immediately but having a readiness check that stops incoming traffic is powerful. In this case, we configured a pod directly. In more realistic cases, this configuration would be part of a deployment.

Having a fast start-up and careful readiness makes blue/green deployment patterns easier to accomplish. It allows configuring which colors are active based on how many nodes are ready. Unlike initialDelaySeconds configured in Kubernetes, optimizations to application start-up time are immediately reflected in better use of computing resources.

Even better, degradations in application start-up time cause degradation in the use of computing resources, not a catastrophic failure. While the degradation should be fixed, allowing the deployment to go forward based on human decisions, not Kubernetes configuration, is an operational advantage.

14.1.2 Configuration

When writing applications that are designed to run on Kubernetes, it is important to consider where their configuration comes from. In this context, configuration refers to things that need to be different between environments (development, staging, production, etc.). Any configuration that does not change between the environments can be added to the container images.

14.1.3 Python Sidecars

Until now, the pods in the examples were single-container pods. One of the main benefits of pods is that they can include more than one container.

All containers in a pod share a network namespace but not the filesystem. In practice, containers in a pod can be built using different stacks and different container images. They can communicate using 127.0.0.1 as an endpoint.

Containers designed to go as the other container in the pod, next to the main one, are called sidecars. It is important to note that from the perspective of Kubernetes, nothing makes a container. When managing pods, though, it is usually the case that one container does the real work while the rest are in support positions.

As a contrived example, imagine an application with a JSON diagnostic endpoint, say on /status. The HTTP status code is always 200. One of the fields in the JSON result is "database-connected": BOOLEAN, where the BOOLEAN is a JSON boolean: either true or false.

The application should not be considered ready if the database is not connected. An httpGet probe is not enough. It always returns successfully.

This is a better alternative to using exec probes, which rely on running commands inside a container. Sidecars can be used for more than readiness checks: exporting metrics, proxying, scheduled cleanup, and more are just a small sample of what can be done with sidecars.

Since sidecar container images are often built ad hoc for existing images, Python is a useful language for them even if the main container is not written in Python.

14.3.1 Permissions

Kubernetes operators need permissions to read the inputs and read/write the outputs. Since operators are regular Kubernetes API clients, they get permissions the same way all other clients get permissions: Resource-Based Access Control (RBAC).

The full details of Kubernetes RBAC are beyond the current scope. One important thing to note is that a common pattern is that operators are deployed as a Kubernetes deployment. Usually, this means the operator uses a deployment running the operators’ code in pods.

These pods can be assigned a Kubernetes service role with the appropriate permissions. In this case, the API client has the right access, as long as it uses the automatically mounted API configuration to access Kubernetes.

14.3.2 Custom Types

To avoid having the inputs having any other effect, many operators also define a custom resource type. Kubernetes allows adding a custom type. The operator can do this, but more commonly, it is done as part of whatever process sets up the operator.

Output types may or may not be custom types; for example, if the output type is a Deployment or a ReplicaSet, it is a standard type. However, for higher-level operators, the outputs might be inputs to a lower-level operator, so a custom type.

Definining a Kubernetes custom resource type is done by creating an object of kind CustomResourceDefinition.

The most important part of the definition is the spec. In this case, it notes that the object has two properties: name, a string, and universe, an integer.

Since this is a relatively simple object, the definition is pretty short. In this case, the Character object represents Dorothy from universe 42.

14.3.3 Retrieval

An operatorfirst has to get all the input objects. This is one advantage of having a custom input kind: all objects of that kind are relevant.

For non-custom types, the Kubernetes Python classes have autogenerated classes. However, for custom ones, it does not. One alternative is to regenerate the client. This is complicated and error-prone.

A quicker, though messier, an alternative is to write ad hoc classes.

This representation of the metadata, with frozenset for labels and an attr.frozen decorator makes Metadata immutable.

Immutable objects are hashable by value and can be added to sets and used as dictionary keys. Although the operator does not require it, this is a powerful invariant in more complicated operators.

14.3.4 Goal State

After retrieving the input objects, the reconciliation loop needs to calculate the goal state: what is the right state for the output objects given the input objects.

This is the business logic of the operator. The parts until now were the plumbing: representing and getting the data from Kubernetes.

Note that the name is automatically generated as a UUID. This is useful since it guarantees there are no collisions with little effort. Many Kubernetes loops, including built-in loops like Deployment and ReplicaSet, include a UUID-like name.

14.3.5 Comparison

A quest not marked with a character-name label is assumed to have been manually added. The operator leaves such quests alone. In a more realistic example, one of the labels added would have been to note the quest as created by the operator, and those not created by the operator would have been filtered out.

The comparison generates an Action.delete result for any quest that exists and should not and an Action.create result for any quest that does exist and should. Some operators instead choose to disable invalid goals and delete them later as part of a garbage collection process. This can be useful for troubleshooting operators.

14.3.6 Reconciliation

To reconcile the desired state with the existing state, the actions need to be performed by the Kubernetes API client. There are two options: using the CustomObjectsApi or using DynamicClient.

The DynamicClient requires a more upfront setup, though it leads to simpler code for creating objects. Since this is a simple operator, the overhead of an upfront setup is more than its worth.

Note that with the CustomObjectsApi the creation logic has to duplicate some parameters between the body and the call to create_namespaced_custom_object().

14.3.7 Combining the Pieces

It is possible to avoid polls by using the watch() API, which makes the code slightly more complicated. In practice, polling loops have little enough overhead. They are easier to write and troubleshoot, so it is usually a good idea to avoid the watch() API until it has proven necessary.