Pilot generates dynamic configuration for the sidecar based upon the startup parameters and configuration information that's saved in the Kubernetes API server. Let's take a look:

1. We can get this configuration by using the following code for the productpage:

$ PRODUCTPAGE_POD=$(kubectl -n istio-lab get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}')

$ kubectl -n istio-lab exec -i $PRODUCTPAGE_POD -c istio-proxy -- cat /etc/istio/proxy/envoy-rev0.json

2. If you scroll through the preceding JSON output, you will notice that the listener port 15090 has a route called /stats/prometheus that Prometheus will scrape to gather data:

...
  "listeners":[
      {
        "address": {
          "socket_address": {
            "protocol": "TCP",
            "address": "0.0.0.0",
            "port_value": 15090
          }
        },
        "filter_chains": [
          {
            "filters": [
              {
                "name": "envoy.http_connection_manager",

...
                        "routes": [
                          {
                            "match": {
                              "prefix": "/stats/prometheus"
... 

3. Run the curl command to scrape the Prometheus metrics:

$ kubectl -n istio-lab exec -i $PRODUCTPAGE_POD -c istio-proxy -- curl http://localhost:15090/stats/prometheus

Each sidecar proxy has a management port of 15000, which is bound to the local loopback adapter of the pod and hence can only be accessed within the pod.

4. The sidecar proxy stats can be seen by running the following command:

$ kubectl -n istio-lab exec -i $PRODUCTPAGE_POD 
-c istio-proxy -- curl http://localhost:15000/stats

5. Let's look at an example for all the pods in istio-lab to check how many days are left until the certificates expire:

$ ALL_PODS=$(kubectl -n istio-lab get pods -o jsonpath='{.items..metadata.name}')

$ for pod in $ALL_PODS; do echo For pod $pod; kubectl -n istio-lab exec -i $pod -c istio-proxy -- curl -s http://localhost:15000/stats | grep server.days_until_first_cert_expiring; done

6. The output will look similar to the following:

...
For pod details-v1-6886b56dc8-ksmrh
server.days_until_first_cert_expiring: 82
...

7. The configuration for the proxy can be seen through proxy management port 15000 using the config_dump route:

$ kubectl -n istio-lab exec -i $PRODUCTPAGE_POD -c istio-proxy -- curl http://localhost:15000/config_dump

The preceding output can be used to confirm route propagation from Mixer to the sidecar proxy when Istio virtual services are created.

Now, we know how each Istio component publishes its Prometheus metrics through different endpoints. The Prometheus collector then scraps such metrics and puts in its own backend system.

Let's use the Prometheus UI to query some of the metrics that were collected. These values can be in the console or in the form of a graph. Prometheus is not a frontend web UI tool for visualizing such data, but it provides a basic UI. We will use Grafana as a UI frontend to show how data is collected from Istio's various components.

First, let's take a look at a basic Prometheus query through its UI. Later, we will switch to the Grafana UI.