The Platform: A Three-Node Kubernetes Cluster

Technically, the target system is made up of a Kubernetes cluster that is once again running nothing more than a simple service.

Kubernetes provides the platform, with Kubernetes nodes providing the lower-level infrastructure resources that support and run containers and services in the cluster. In this case, the target system has a three-node cluster topography.

So, for the purposes of your chaos experimentation in this chapter, there is going to be a service running on one or more replicated containers across a cluster made up of three nodes. Now it’s time to describe and deploy that service to the cluster.

The Application: A Single Service, Replicated Three Times

Once again, you don’t need a supercomplicated application made up of hundreds of services to have a system that is sufficiently complicated for weaknesses to be found. The application deployed onto your target system is going to be made up of a single service, defined as follows:

import platform

import cherrypy


class Root:
    @cherrypy.expose
    def index(self) -> str:
        return "Hello world from {}".format(platform.node())


if __name__ == "__main__":
    cherrypy.config.update({
        "server.socket_host": "0.0.0.0",
        "server.socket_port": 8080
    })
    cherrypy.quickstart(Root())

This code is available in the chapter4/before directory in the sample code for this book that you grabbed earlier (see “Setting Up the Sample Target System”). Accompanying this service description is a description of the deployment itself:

{
    "apiVersion" : "apps/v1beta1",
    "kind" : "Deployment",
    "metadata" : {
      "name" : "my-service"
    },
    "spec" : {
      "replicas" : 3,
      "selector" : {
        "matchLabels" : {
          "service" : "my-service"
        }
      },
      "template" : {
        "metadata" : {
          "name" : "my-app",
          "labels" : {
            "name" : "my-app",
            "service" : "my-service",
            "biz-app-id" : "retail"
          }
        },
        "spec" : {
          "containers" : [ {
            "name" : "my-app",
            "ports" : [ {
              "name" : "http",
              "containerPort" : 8080,
              "protocol" : "TCP"
            } ],
            "imagePullPolicy" : "Always",
            "image" : "docker.io/chaosiq/sampleservice:0.1.0",
            "resources" : {
              "limits" : {
                "cpu" : 0.1,
                "memory" : "64Mi"
              },
              "requests" : {
                "cpu" : 0.1,
                "memory" : "64Mi"
              }
            }
          } ]
        }
      },
      "strategy" : {
        "type" : "RollingUpdate",
        "rollingUpdate" : {
          "maxUnavailable" : 1,
          "maxSurge" : 1
        }
      }
    }
  }

The main thing to notice is the replicas directive in the deployment.json file. This directive specifies that the team that developed this service expects it to be run with three instances to provide a minimal fallback strategy should one or more of the replicas have a problem.

When you deploy these specifications, using a command such as kubectl apply -f before/, Kubernetes will establish a cluster where my-service will be replicated across the available nodes.

All good so far, but often there is more than one group of people involved in a system such as this. The service and deployment specifications represent the intent of the team responsible for the application itself, but there’s typically another group responsible for the cluster itself, and that can lead to complications and turbulent conditions.

The People: Application Team and Cluster Administrators

In a system like this, there is often a dividing line between those who are responsible for managing a Kubernetes cluster, including all the real costs associated with these resources (let’s call them the Cluster Administrators), and those trying to deploy and manage applications upon the cluster (the Application Team). The Cluster Administrators will be interested in tasks such as:

• Managing nodes

• Managing persistent disks

• Keeping an eye on the costs if these resources are hosted on a cloud provider

• Ensuring the cluster is not the problem, so they can go home for the weekend

The Application Team will be more focused on:

• Ensuring their application and its constituent services and third-party dependencies are healthy

• Ensuring they have enough redundancy and capacity across the Kubernetes pods being used to run their containers

• Ensuring they claim the persistent storage they need for their application to maintain its state

• Ensuring they can go home at 5:30 p.m. (other times are available)

The Application Team is responsible for making sure the deployment.json and service.json specifications ask for what they need from the platform. The Cluster Administrators will have their own dashboards and tools to ensure the underlying resources of the cluster are able to meet the needs expressed by the Application Team (see Figure 6-1).

Figure 6-1. Application Team and Cluster Administrators working on a Kubernetes cluster

Naming Your Experiment

Create a new experiment.json file in your chapter6 directory and start your experiment definition by considering how it should be named (I name chaos experiments according to the high-level question being explored). Then complete your title and description section with some tags:

{
    "version": "1.0.0",
    "title": "My application is resilient to admin-instigated node drainage", 
    "description": "Can my application maintain its minimum resources?", 
    "tags": [
        "service",
        "kubernetes" 
    ],

This is the statement of belief. You believe the system will be OK under the turbulent conditions explored in this experiment.

The description gives you an opportunity to describe things in a little more detail, sometimes raising the doubt that’s causing the experiment in the first place.

Since we know we are targeting a Kubernetes-based system, it’s helpful to others who might read and use our experiment to tag it as being applicable to that platform.

Defining Your Steady-State Hypothesis

What does “normal” look like? How should the system look so that you know it’s operating within particular bounds? Everything may not be working, but the steady-state hypothesis isn’t concerned with the health of the individual parts of a system; its main concern is that the system is still able to operate within declared tolerances, regardless of the turbulent conditions being injected through your experiment’s method.

As we saw in Chapter 5, a steady-state hypothesis is made up of one or more probes and associated tolerances. Each probe will look for a property within your target system and judge whether that property’s value is within the tolerance specified.

If all the probes declared in your steady-state hypothesis are within tolerance, then your system can be recognized as not in a “deviated” state. Define your steady-state hypothesis as the next section in your experiment.json file, as shown here:

"steady-state-hypothesis": {
    "title": "Services are all available and healthy", 
    "probes": [
        {
            "type": "probe",
            "name": "application-must-respond-normally",
            "tolerance": 200, 
            "provider": {
                "type": "http", 
                "url": "http://http://35.189.85.252/", 
                "timeout": 3 
            }
        },
        {
            "type": "probe",
            "name": "pods_in_phase",
            "tolerance": true,
            "provider": {
                "type": "python", 
                "module": "chaosk8s.pod.probes", 
                "func": "pods_in_phase", 
                "arguments": { 
                    "label_selector": "biz-app-id=retail",
                    "phase": "Running",
                    "ns": "default"
                }
            }
        }
    ]
},

The title of your hypothesis should describe your belief in the normal condition of the system.

The tolerance in this case expects the probe to return with a 200 status code.

The first probe uses HTTP to assess the return value of an endpoint.

You will need to change this value to the endpoint of the service as it is exposed from your Kubernetes cluster.

A reasonable timeout is a good idea to ensure your experiment doesn’t wait too long to decide that the system is normal.

The second probe uses a call to a Python module.

This is the name of the Python module to be used by this probe.

This is the name of the Python function to be used by this probe.

This is a key/value list of the arguments to be supplied to the probe’s Python function.

This steady-state hypothesis shows you how a simple HTTP call can be used as a probe, or, if more complicated processing is necessary, how a Python module’s function can be used instead. A third option that is not shown here is to use a call to a local process as a probe.

In summary, your steady-state hypothesis for this experiment probes the target system to detect (a) that it is responding with an HTTP 200 status code within three seconds on the main entry point to the system and (b) that Kubernetes is reporting that all the pods within the system are in the healthy Running phase.

Now it’s time to disrupt that with some turbulent conditions, declared in your experiment’s method.

Injecting Turbulent Conditions in an Experiment’s Method

A chaos experiment’s method defines what conditions you want to cause in the effort to surface a weakness in the target system. In this case, your target system is a Kubernetes cluster. The challenge is that, out of the box, the Chaos Toolkit supports only a few limited and basic ways of causing chaos and probing a system’s properties.

(chaostk) $ pip install chaostoolkit-kubernetes

Using the Kubernetes Driver from Your Method

A Chaos Toolkit driver provides a collection of probes and actions. Probes allow you to inspect the properties of your target system, either in support of a condition in your steady-state hypothesis or simply as a way of harvesting some useful information while your experiment is running. Actions are the turbulent condition–inducing parts of the driver.

For your experiment, you are taking on the role of a business-as-usual Kubernetes Cluster Administrator who is blissfully attempting to drain a node for maintenance reasons, unaware of the impact of this action on the needs of the application the cluster is running. Specifically, you’re going to create an entry in your experiment’s method that uses the drain_nodes action from the Kubernetes driver.

First you create a method block under the steady-state-hypothesis declaration in your experiment:

"steady-state-hypothesis": {
    // Contents omitted...
},
"method": [] 

Your experiment’s method may contain many actions and probes, so it is initialized with a JavaScript array.

Now you can add your action. An action starts with a type that indicates whether this is an action or a probe. If it is a probe, the result will be added to the experiment’s journal; otherwise, as an action it is simply executed, and any return value is ignored:

"steady-state-hypothesis": {
    // Contents omitted...
},
"method": [ {
    "type": "action",
}]

You can now name your action. You should aim to make this name as meaningful as possible, as it will be captured in the experiment’s journal when it records that this action has taken place:

"steady-state-hypothesis": {
    // Contents omitted...
},
"method": [ {
    "type": "action",
    "name": "drain_node",
}]

Now it’s time for the interesting part. You need to tell the Chaos Toolkit to use the drain_nodes action from your newly installed Chaos Toolkit Kubernetes driver:

"steady-state-hypothesis": {
    // Contents omitted...
},
"method": [ {
    "type": "action",
    "name": "drain_node",
    "provider": { 
        "type": "python", 
        "module": "chaosk8s.node.actions", 
        "func": "drain_nodes", 
        "arguments": { 
            "name": "gke-disruption-demo-default-pool-9fa7a856-jrvm", 
            "delete_pods_with_local_storage": true 
        }
    }
}]

The provider block captures how the action is to be executed.

The provider in this case is a module written in Python.

This is the Python module that the Chaos Toolkit is supposed to use.

This is the name of the actual Python function to be called by this action.

The drain_nodes function expects some parameters, so you can specify those here.

You must specify the name of a current node in your cluster here. You can select a node from the list reported by kubectl get nodes.

Node drainage can be blocked if a node is attached to local storage; this flag tells the action to drain the node even if this is the case.

And that’s it—your experiment’s method is complete, as the only action we’re concerned with that may cause a weakness to surface is a Kubernetes Cluster Administrator doing their job by draining a node. At all times the conditions of the steady-state hypothesis should be within tolerance before and after this method has been executed if no weakness is present in the system.

Right about now you might be keen to run your experiment, but there is one more thing to consider. You will have caused turbulent conditions through your experiment’s method that you may want to consider reversing. That’s what an experiment’s rollbacks section is for.

Overcoming a Weakness: Applying a Disruption Budget

A Kubernetes Disruption Budget limits the number of pods of a replicated application that are down simultaneously from voluntary disruption. Your experiment relies on the voluntary eviction of pods from the nodes that it is draining, so the Application Team can safeguard their pods and stop the Cluster Administrators from draining too many nodes at once by enforcing a Disruption Budget resource in the Kubernetes cluster:

{
	"apiVersion": "policy/v1beta1",
	"kind": "PodDisruptionBudget",
	"metadata": {
		"name": "my-app-pdb"
	},
	"spec": {
		"minAvailable": 3, 
		"selector": {
			"matchLabels": {
				"name": "my-app" 
			}
		}
	}
}

Specifies the minimum number of pods that must be in the READY state at a given moment in time.

Selects the pods to be protected by this Disruption Budget.

Next, you can apply this Disruption Budget to your cluster by executing the following command:

$ kubectl apply -f ./after

Now you can use your very own chaos experiment as a test to ensure that the weakness detected before is no longer present.

Success (as a tester)! Your chaos-experiment-turned-test passes, as there is no deviation from the steady-state hypothesis, and thus the original weakness is not present.