Prometheus is a backend time series data collection service that is configured by default in Linkerd. It only keeps data for 6 hours in order to limit the Linkerd telemetry footprint to an acceptable range without adversely affecting performance. Grafana is the frontend and contains Linkerd pre-built dashboards, as we will soon see.

We can access Prometheus from inside the VM or from outside if we define an ingress rule for it. Let's define the Ingress for Prometheus using prometheus.linkerd.io hostname, which resolves to the internal IP address of the VM through our simulated load balancer:

1. Let's define the Prometheus ingress so that we can direct external traffic to the Linkerd-Prometheus service:

# Script : 06-create-prometheus-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus
  annotations:
    nginx.org/websocket-services: "linkerd-prometheus"
    ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:9090;
      proxy_hide_header l5d-remote-ip;
      proxy_hide_header l5d-server-id;
spec:
  rules:
  - host: prometheus.linkerd.local
    http:
      paths:
      - backend:
          serviceName: linkerd-prometheus
          servicePort: 9090
        path: /

2. Next, we need to create a Prometheus ingress:

$ kubectl -n linkerd apply -f 06-create-prometheus-ingress.yaml
ingress.extensions/prometheus created

3. Now, create an entry in /etc/hosts for the prometheus.linkerd.local host:

$ export INGRESS_HOST=$(kubectl -n kube-system get service nginx-controller -o jsonpath='{.status.loadBalancer.ingress..ip}') ; echo $INGRESS_HOST
192.168.142.249

$ sudo sed -i '/prometheus.linkerd.local/d' /etc/hosts

$ echo "$INGRESS_HOST prometheus.linkerd.local" | sudo tee -a /etc/hosts

4. Run http://prometheus.linkerd.local from your browser. Select Status | Targets: 

Here, we can see the metrics for the endpoints for Grafana, Linkerd controllers, Linkerd proxies, Prometheus, the status of the state, and the last scrapped metrics. 

You can check the Prometheus configuration by clicking on Status | Configuration. The custom service discovery integration for Linkerd can be viewed from Status | Service Discovery.

5. Click on Prometheus and type or select process_cpu_seconds_total. from the drop-down menu.

6. Click Execute and select the Graph tab to see the time series graph:

Prometheus has some essential built-in graphing capabilities, but Grafana provides a better frontend GUI for the data that has been collected by the Prometheus pull model.

Prometheus is the data collector and contains basic GUI features, as we saw for the Linkerd data collection. Next, we will look at the data that's collected through the Grafana Web UI.