We'll be building upon the Kubernetes test environment started previously. In this particular scenario, we'll deploy an instance of Pushgateway and we'll be adding it as a target in Prometheus. To validate the correctness of our setup, we'll create a Kubernetes CronJob to emulate a batch job style of workload, and push its metrics to the Pushgateway service to ensure Prometheus collects our data.

To begin the deployment, ensure you move into the correct repository path, relative to the code repository root:

cd ./chapter06/provision/kubernetes/

To deploy an instance of Pushgateway, you can use the following manifest. Keep in mind that this service does not support high availability or clustering:

apiVersion: apps/v1
kind: Deployment
metadata: {name: pushgateway, namespace: monitoring}
spec:
  selector:
    matchLabels: {p8s-app: pushgateway}
  replicas: 1
  template:
    metadata:
      labels: {p8s-app: pushgateway}
    spec:
      containers:
      - name: pushgateway
...

Apply the manifest by executing the following command:

kubectl apply -f ./pushgateway/pushgateway-deployment.yaml

And follow the deployment using the following:

kubectl rollout status deployment/pushgateway -n monitoring

After a successful deployment, it's time to provide a Service to our new instance, using the following manifest:

apiVersion: v1
kind: Service
metadata:
  name: pushgateway-service
  namespace: monitoring
  labels:
    p8s-app: pushgateway
spec:
  type: NodePort
  ports:
  - {name: push-port, port: 9091, targetPort: push-port, protocol: TCP}
  selector:
    p8s-app: pushgateway

The following instruction applies to the previous manifest:

kubectl apply -f ./pushgateway/pushgateway-service.yaml 

You may now validate the web interface for Pushgateway using the following command:

minikube service pushgateway-service -n monitoring

This will open a new browser tab pointing to the newly created Pushgateway instance web interface, which should look like the following figure:

Now, we need to instruct Prometheus to scrape Pushgateway. This can be accomplished via a new ServiceMonitor manifest as follows:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    p8s-app: pushgateway
  name: pushgateway
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: push-port
    honorLabels: true
  selector:
    matchLabels:
      p8s-app: pushgateway

To apply this ServiceMonitor, we just type the following command:

kubectl apply -f ./pushgateway/pushgateway-servicemonitor.yaml

Now that we have our monitoring infrastructure in place, we need to simulate a batch job to validate our setup.

We can rely on the following manifest, which pushes a dummy batchjob_example metric with several labels to the Pushgateway service endpoint using a handcrafted curl payload:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: batchjob
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: batchjob
            image: kintoandar/curl:7.61.1
            args:
            - -c
            - 'echo "batchjob_example $(date +%s)" | curl -s --data-binary @- http://pushgateway-service.monitoring.svc.cluster.local:9091/metrics/job/batchjob/app/example/squad/yellow'
          restartPolicy: OnFailure

To apply the previous manifest, use the following command:

kubectl apply -f ./pushgateway/batchjob-cronjob.yaml

After a minute, the web interface for Pushgateway will look similar to this screenshot:

We can now use the Prometheus expression browser to validate the metric is being scraped from Pushgateway:

As Pushgateway's job is to proxy metrics from other sources, it provides very little metrics of its own - just the standard Go runtime metrics, process metrics, HTTP handler metrics and build info. However, there is one application metric to note, which is push_time_seconds. This will tell you the last time a specific group (combination of labels used in the HTTP API when pushing) was seen. This can be used to detect missing or delayed jobs.