Prometheus, which runs as part of Linkerd's control plane, retains data for approximately 6 hours. If further data retention is required, the metrics can be exported to a data store such as the Elasticsearch, Logstash, and Kibana (ELK) stack or to a dedicated Prometheus instance.
The Prometheus federation API or ServiceMonitors can be used to copy data using the /federate path from Linkerd Prometheus to the dedicated Prometheus store.
Alternatively, you can call the federation API directly, and then it can be ingested by a Kafka stream, which can then dump it to an ELK stack, like so:
$ curl -G
--data-urlencode 'match[]={job="linkerd-proxy"}'
--data-urlencode 'match[]={job="linkerd-controller"}'
http://linkerd-prometheus.linkerd.svc.cluster.local:9090/federate
From the preceding curl command, you can use an external name through Ingress to route traffic to the linkerd-prometheus microservice if the /federate path needs to be scraped external to the cluster.
There can be multiple Linkerd proxies running within a data plane and a control plane. You can scrape metrics directly from a Linkerd proxy like so:
$ kubectl -n linkerd port-forward
(kubectl -n linkerd get pods
-l linkerd.io/control-plane-ns=linkerd
-o jsonpath='{.items[0].metadata.name}')
4191:4191
From your local machine, after you run the preceding kubectl command, you can run the following curl command and access the Prometheus federated metrics datastore:
$ curl localhost:4191/metrics
Next, let's dive into the Linkerd sidecar proxy and how to enable the debugging flag. This will provide detailed views into the microservice it's attached to.