Node

Node service discovery is used to discover the nodes comprising the Kubernetes cluster, and you will use it to monitor the infrastructure around Kubernetes. The Kubelet is the name of the agent that runs on each node, and you should scrape it as part of monitoring the health of the Kubernetes cluster (Example 9-5).

scrape_configs:
- job_name: 'kubelet'
  kubernetes_sd_configs:
   - role: node
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true

Example 9-5 shows the configuration being used by Prometheus to scrape the Kubelet. I’m going to break down the scrape config:

job_name: 'kubelet'

A default job label is provided, and as there are no relabel_configs, kubelet will be the job label:³

kubernetes_sd_configs:
- role: node

A single Kubernetes service discovery is provided with the node role. The node role discovers one target for each of your Kubelets. As Prometheus is running inside the cluster, the defaults for the Kubernetes service discovery are already set up to authenticate with the Kubernetes API.

scheme: https
tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  insecure_skip_verify: true

The Kubelet serves its /metrics over HTTPS, so we must specify the scheme. Kubernetes clusters usually have their own certificate authority that are used to sign their TLS certs, and the ca_file provides that for the scrape. Unfortunately Minikube doesn’t get this quite right, so insecure_skip_verify is required to bypass security checks.

The returned target points at the Kubelet, and authentication/authorisation is turned off in this Minikube setup, so no further configuration is needed.

The metadata available includes node annotations and labels, as you can see in Figure 9-3. You could use this metadata with relabel_configs to add labels to distinguish interesting subsets of nodes, such as those with different hardware.

Figure 9-3. The Kubelet on the service discovery status page of Prometheus

The Kubelet’s own /metrics only contain metrics about the Kubelet itself, not container-level information. The Kubelet has an embedded cAdvisor on its /metrics/cadvisor endpoint. Scraping the embedded cAdvisor only requires adding a metrics_path to the scrape config used with the Kubelet, as shown in Example 9-6. The embedded cAdvisor includes labels for the Kubernetes namespace and pod_name.

scrape_configs:
- job_name: 'cadvisor'
  kubernetes_sd_configs:
   - role: node
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  metrics_path: /metrics/cadvisor

Node service discovery can be used for anything you want to monitor that runs on each machine in a Kubernetes cluster. If the Node exporter was running on your Minikube node you could scrape it by relabelling the port, for example.

Service

Node service discovery is useful for monitoring the infrastructure of and under Kubernetes, but not much use for monitoring your applications running on Kubernetes.

There are several ways that you can organise your applications on Kubernetes, and no single clear standard has emerged yet. But you are likely using services, which is how applications on Kubernetes find each other.

While there is a service role, it is not what you usually want. The service role returns a single target for each port⁴ of your services. Services are basically load balancers, and scraping targets through load balancers is not wise as Prometheus can scrape a different application instance each time. However, the service role can be useful for blackbox monitoring, to check if the service is responding at all.

Endpoints

Prometheus should be configured to have a target for each application instance, and the endpoints role provides just that. Services are backed by pods. Pods are a group of tightly coupled containers that share network and storage. For each Kubernetes service port, the endpoints service discovery role returns a target for each pod backing that service. Additionally, any other ports of the pods will be returned as targets.

That’s a bit of a mouthful, so let’s look at an example. One of the services that is running in your Minikube is the kubernetes service, which are the Kubernetes API servers. Example 9-7 is a scrape config that will discover and scrape the API servers.

scrape_configs:
- job_name: 'k8apiserver'
  kubernetes_sd_configs:
   - role: endpoints
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
   - source_labels:
      - __meta_kubernetes_namespace
      - __meta_kubernetes_service_name
      - __meta_kubernetes_endpoint_port_name
     action: keep
     regex: default;kubernetes;https

Breaking down this scrape config:

job_name: 'k8apiserver'

The job label is going to be k8apiserver, as there’s no target relabelling to change it:

kubernetes_sd_configs:
- role: endpoints

There is a single Kubernetes service discovery using the endpoints role, which will return a target for every port of every pod backing each of your services:

scheme: https
tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

As with the Kubelet, the API servers are served over HTTPS. In addition, authentication is required, which is provided by the bearer_token_file:

relabel_configs:
- source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
    - __meta_kubernetes_endpoint_port_name
  action: keep
  regex: default;kubernetes;https

This relabel configuration will only return targets that are in the default namespace, and are part of a service called kubernetes with a port called https.

You can see the resulting target in Figure 9-4. The API server is special, so there isn’t much metadata. All the other potential targets were dropped.

Figure 9-4. The API server on the service discovery status page of Prometheus

While you will want to scrape the API servers, most of the time you’ll be focused on your applications. Example 9-8 shows how you can automatically scrape the pods for all of your services.

scrape_configs:
 - job_name: 'k8services'
   kubernetes_sd_configs:
    - role: endpoints
   relabel_configs:
    - source_labels:
       - __meta_kubernetes_namespace
       - __meta_kubernetes_service_name
      regex: default;kubernetes
      action: drop
    - source_labels:
       - __meta_kubernetes_namespace
      regex: default
      action: keep
    - source_labels: [__meta_kubernetes_service_name]
      target_label: job

Once again I’ll break it down:

job_name: 'k8services'
kubernetes_sd_configs:
 - role: endpoints

As with the previous example, this is providing a job name and the Kubernetes endpoints role, but this does not end up as the job label due to later relabelling.

There are no HTTPS settings, as I know the targets are all plain HTTP. There is no bearer_token_file, as no authentication is required, and sending a bearer token to all of your services could allow them to impersonate you:⁵

relabel_configs:
- source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_service_name
  regex: default;kubernetes
  action: drop
- source_labels:
    - __meta_kubernetes_namespace
  regex: default
  action: keep

I excluded the API server, as there is already another scrape config handling it. I also only looked at the default namespace, which is where I am launching applications:⁶

- source_labels: [__meta_kubernetes_service_name]
  target_label: job

This relabel action takes the Kubernetes service name and uses it as the job label. The job_name I provided for the scrape config is only a default, and does not apply.

In this way you can have your Prometheus automatically pick up new services and start scraping them with a useful job label. In this case that’s just Prometheus itself, as shown in Figure 9-5.

Figure 9-5. Prometheus has automatically discovered itself using endpoint service discovery

You could go a step further and use relabelling to add additional labels from service or pod metadata, or even set __scheme__ or __metrics_path__ based on a Kubernetes annotation, as shown in Example 9-9. These would look for prometheus.io/scheme, prometheus.io/path, and prometheus.io/port service annotations,⁷ and use them if present.

relabel_configs:
 - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
   regex: (.+)
   target_label: __scheme__
 - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
   regex: (.+)
   target_label: __metrics_path__
 - source_labels:
    - __address__
    - __meta_kubernetes_service_annotation_prometheus_io_port
   regex: ([^:]+)(:d+)?;(d+)
   replacement: ${1}:${3}
   target_label: __address__

This is limited to monitoring only one port per service. You could have another scrape config using the prometheus.io/port2 annotation, and so on for however many ports you need.

Pod

Discovering endpoints is great for monitoring the primary processes backing your services, but it won’t discover pods that are not part of services.

The pod role discovers pods. It will return a target for each port of every one of your pods. As it works off pods, service metadata such as labels and annotations are not available as pods do not know which services they are members of. But you will have access to all pod metadata. How you use this boils down to a question of what conventions you want to use. The Kubernetes ecosystem is rapidly evolving, and there is no one standard yet.

You could create a convention that all pods must be part of a service, and then use the endpoint role in service discovery. You could have a convention that all pods have a label indicating the (single) Kubernetes service they are a part of, and use the pod role for service discovery. As all ports have names, you could base a convention off that and have ports named with a prefix of prom-http be scraped with HTTP, and prom-https be scraped with HTTPS.

One of the components that comes with Minikube is kube-dns, which provides DNS services. Its pod has multiple ports, including a port named metrics that serves Prometheus metrics. Example 9-10 shows how you could discover this port and use the name of the container as the job label, as Figure 9-6 shows.

scrape_configs:
- job_name: 'k8pods'
  kubernetes_sd_configs:
   - role: pod
  relabel_configs:
   - source_labels: [__meta_kubernetes_pod_container_port_name]
     regex: metrics
     action: keep
   - source_labels: [__meta_kubernetes_pod_container_name]
     target_label: job

Figure 9-6. Two targets discovered using pod service discovery

Ingress

An ingress is a way for a Kubernetes service to be exposed outside the cluster. As it is a layer on top of services, similar to the service role the ingress role is also basically a load balancer. If multiple pods backed the service and thus ingress, this would cause you problems when scraping with Prometheus. Accordingly, you should only use this role for blackbox monitoring.