In this example, we're stepping away from the Prometheus Kubernetes Operator we've been using in previous chapters so that we can focus on the Prometheus native service discovery integration for this container orchestration platform. The manifests for getting Prometheus up and running in our Kubernetes test environment can be found, relative to the code repository root path, at the following path:

cd ./chapter12/provision/kubernetes/

The following steps will ensure a new Kubernetes environment with all the required software provisioned so that we can then focus on the service discovery component.

Validate that no other environment is running:

minikube status
minikube delete

Start an empty Kubernetes environment:

minikube start 
  --cpus=2 
  --memory=3072 
  --kubernetes-version="v1.14.0" 
  --vm-driver=virtualbox 
  --extra-config=kubelet.authentication-token-webhook=true 
  --extra-config=kubelet.authorization-mode=Webhook

The extra configuration we're providing to minikube is needed so that Prometheus is able to interact with kubelets using service account tokens. When the previous command finishes, a new Kubernetes environment will be ready to be used. We can then proceed to deploy our configurations using the following instructions:

kubectl apply -f ./bootstrap/
kubectl rollout status deployment/prometheus-deployment -n monitoring

The previous command applies several manifests, which, among other things, create a namespace called monitoring, a ServiceAccount, and all the required RBAC configurations so that Prometheus can query the Kubernetes API. A ConfigMap with the Prometheus server configuration is also included, which can be found at bootstrap/03_prometheus-configmap.yaml. It defines several scrape jobs for Kubernetes components that are targeted through the use of service discovery, as we can see in the following snippet:

$ cat bootstrap/03_prometheus-configmap.yaml
...
data:
  prometheus.yml: |
    scrape_configs:
...
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: hey
        source_labels:
        - __meta_kubernetes_pod_label_app
...

We can open the Prometheus web interface by issuing the following command:

minikube service prometheus-service -n monitoring

By moving into the service discovery section available on the /service-discovery endpoint, we can see that, even though several pods were discovered, none of them matched the label value hey for the app label, and as such are being dropped:

It's now time to add some new pods with the correct label/value pair to trigger our service discovery configuration. We can proceed by running the following commands, which will be deploying the hey application, and then follow the status of the deployment:

kubectl apply -f ./services/
kubectl rollout status deployment/hey-deployment -n default

After a successful deployment, we can go, once again, to the Prometheus web interface at the /service-discovery endpoint, where we can see that there are now three active targets in the kubernetes-pods scrape job. The following screenshot depicts one of those targets and all the labels provided by the Kubernetes API:

When you're finished testing, you can delete this Kubernetes-based environment by issuing the following command:

minikube delete

This approach to service discovery allows us to keep track of several Kubernetes objects automatically, without forcing us to change the Prometheus configuration manually. This environment allows us to test all sorts of settings and provides the basis for tailoring the Kubernetes service discovery to our specific needs.