appendix D. Troubleshooting Istio components
Throughout the book, we frequently query the Istio agent and Pilot to get information such as the configuration of a proxy, metrics exposed by it, and so on. These queries are shown on a use-case by use-case basis and are scattered throughout the book, making it difficult for the reader to recall what port 15000, 15020, or any of the other ports are. This appendix presents all the open ports and the endpoints where you can make requests to debug, troubleshoot, or get information from either the control plane or the service proxy to understand the workings of the mesh.
D.1 Information exposed by the Istio agent
The Istio sidecar provides a lot of functionality:
-
Health checking—Envoy as a proxy is ready as soon as it can process traffic. But from the perspective of the service mesh, that doesn’t suffice. There have to be more checks in place, such as whether the proxy receiving the configuration and being assigned an identity before it can serve traffic.
-
Metrics collection and exposure—Within a service, three components generate metrics: the application, the agent, and the Envoy proxy. The agent aggregates the metrics from the other components and exposes them.
-
DNS resolution, routing inbound and outbound traffic, and much more.
The services are exposed on numerous ports, and they may appear overwhelming when you list all of them:
$ kubectl -n istioinaction exec -it deploy/webapp
-c istio-proxy -- netstat -tnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:15021 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15021 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:15000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15001 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15001 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:15004 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15006 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15006 0.0.0.0:* LISTEN
tcp6 0 0 :::8080 :::* LISTEN
tcp6 0 0 :::15020 :::* LISTENFigure D.1 visualizes the ports the agent and proxy listen to and the functionality each exposes:
Figure D.1 Agent and Envoy proxy ports and their functionality
-
15020—Exposes a variety of functionalities, the main ones being as follows:
-
-
Aggregating and exposing the metrics of the Envoy proxy by querying metrics from port 15090, application metrics (if configured), and its own metrics.
-
Health-checking the Envoy and DNS proxies. The proxy can be configured to perform health checking of the application on this endpoint as well, but this is generally only used for non-Kubernetes workloads such as virtual machines.
-
Endpoints for debugging the pilot agent—useful for Istio development teams—that expose information such as memory information, CPU profiling, and so on.
-
-
15021—Pods with the sidecar injected are configured to check their readiness to receive traffic on this port. As explained previously, the Envoy proxy routes the health checks to the Pilot agent on port 15020, where the actual health-checking occurs.
-
15053—Local DNS proxy configured by
istiodto resolve edge cases where Kubernetes DNS resolution doesn’t suffice. -
15001—Outbound traffic from the application is redirected to this port by Iptable rules, from which point the proxy handles routing the traffic to the services.
-
15006—Inbound traffic to the application is redirected to this port by Iptable rules, where it is routed to the local application.
Ports that are useful for debugging and introspecting the agent:
-
15000—Envoy proxy administration interface (this is covered in chapter 10, specifically the section 10.3.1).
-
15090—Exposes Envoy proxy metrics such as xDS stats, connection stats, HTTP stats, outlier stats, health check stats, circuit-breaker stats, and so on.
-
15004—Exposes the Istio Pilot debug endpoints (more on that later in this appendix) through the agent. Useful to debug connection issues with the Pilot.
-
15020—Exposes endpoints for debugging the Pilot agent (as mentioned for the service-facing ports).
You may have noticed that port 15020 provides multiple functionalities. Let’s take a closer look at it.
D.1.1 Endpoints to introspect and troubleshoot the Istio agent
The agent exposes a set of endpoints in port 15020 that aid troubleshooting and introspecting the agent and the proxy. These endpoints are:
-
/healthz/ready—Performs a series of probes on the Envoy and DNS proxies to ensure that the workload is ready to process client requests.
-
/stats/prometheus—Merges the metrics of the Envoy proxy and the application with its own metrics and exposes them for scraping.
-
/app-health/—Executes the application health probes defined as the environment variable
ISTIO_KUBE_APP_PROBERSin the Istio proxy sidecar. When an application defines Kubernetes health probes, theistiodmutating webhook extracts the information and configures the health checks via this environment variable. (For more, see http://mng.bz/mxxP.) Thus the agent redirects queries on that path to the application. -
/debug/ndsz—Lists the hostnames for which DNS proxy is configured by
istiodusing the Name Discovery Service (NDS) API. -
/debug/pprof /*—Golang profiling endpoints to help debug performance issues, memory leaks, and so on (see https://golang.org/doc/diagnostics#profiling). You can see the entire list of debug endpoints by querying the base path localhost:15020/debug/pprof. The output is HTML and is best viewed in the browser (remember that you can port-forward the port to your localhost). The profiling endpoints are relevant for Istio developers and not a concern for Istio users.
The easiest way to access these endpoints is using kubectl exec to make an HTTP request in any workload that you are interested in. For example, to check the merged stats of the webapp workload we’d execute:
kubectl exec deploy/webapp -c istio-proxy --
curl localhost:15020/stats/prometheusIn the response, you’ll see metrics prefixed with istio_agent (originating in the agent) and envoy (originating in the proxy), which shows that those are merged. Before going into the next section take some time and investigate the other endpoints listed earlier.
D.1.2 Querying Istio Pilot debug endpoints through the Istio agent
The agent exposes a few istiod debug endpoints—you’ll learn more about those endpoints later in the appendix—by default on port 15004. Requests on those endpoints are forwarded securely to istiod as xDS events, which is a good way to verify connectivity to the control plane from the agent.
For example, one of the endpoints that’s exposed allows us to query the synchronization status of workloads. To view that, get a shell connection in one of the proxies, and make a request on the endpoint /debug/syncz on port 15004 of the pilot agent:
curl -v localhost:15004/debug/syncz
[
# other items are collapsed
{
"@type": "type.googleapis.com/
➥envoy.service.status.v3.ClientConfig",
"node": {
"id": "catalog-68666d4988-zjsmn.istioinaction" ❶
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/
➥envoy.config.listener.v3.Listener",
"configStatus": "SYNCED" ❷
},
{
"typeUrl": "type.googleapis.com/
➥envoy.config.route.v3.RouteConfiguration",
"configStatus": "SYNCED" ❷
},
{
"typeUrl": "type.googleapis.com/
➥envoy.config.endpoint.v3.ClusterLoadAssignment",
"configStatus": "SYNCED" ❷
},
{
"typeUrl": "type.googleapis.com/
➥envoy.config.cluster.v3.Cluster",
"configStatus": "SYNCED" ❷
}
]
}]❷ The xDS APIs are synchronized to the latest state.
The exposed information is a subset of the information exposed by the Istio Pilot debug endpoints. The same endpoints are exposed by the istioctlxinternal-debug command, which is a new addition to istioctl.
Knowledge of these ports and the services they expose will make troubleshooting easier. You can query the latest Envoy configuration, manually test DNS resolution, query metrics to learn about the workings of components, and so on. Next, let’s look at what the Istio Pilot exposes.
D.2 Information exposed by the Istio Pilot
The Pilot also exposes information to introspect and debug the service mesh. This information is useful for external services and as well for service mesh operators.
We can list the ports opened by the Istio Pilot as follows:
$ kubectl -n istio-system exec -it deploy/istiod -- netstat -tnl Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.1:9876 0.0.0.0:* LISTEN tcp6 0 0 :::15017 :::* LISTEN tcp6 0 0 :::8080 :::* LISTEN tcp6 0 0 :::15010 :::* LISTEN tcp6 0 0 :::15012 :::* LISTEN tcp6 0 0 :::15014 :::* LISTEN
In addition to the ports exposed for workloads to get their configuration and certificates, there are quite a few ports that are useful for you to introspect and debug the control plane. Figure D.2 visualizes the ports and the functionality they expose:
-
-
15010—Exposes the xDS APIs and the issuance of certificates in plain text. Using this port is not recommended because the traffic can be sniffed.
-
15012—Exposes the same information as port 15010 but makes it secure. This port uses TLS for issuing the identity, and subsequent requests are mutually authenticated.
-
15014—Exposes control-plane metrics such as those covered in chapter 11.
-
15017—Exposes the webhook server that the Kubernetes API server calls to inject the sidecar into newly created pods and validate Istio resources such as
Gateways,VirtualServices, and so on.
-
Figure D.2 The exposed Istio Pilot ports and their functionality
D.2.1 The Istio Pilot debug endpoints
The Istio Pilot debug endpoints expose the configuration and state of the entire service mesh—as it is known to the Pilot. The endpoints answer questions such as: Are the proxies synchronized? When was the last push to a proxy performed? What’s the state of the xDS APIs? All of these are important for resolving tricky cases and understanding how the proxy is configured.
To access the debug endpoints, port-forward one of istiod instances to your local environment:
$ kubectl -n istio-system port-forward deploy/istiod 8080 Forwarding from 127.0.0.1:8080 -> 8080 Forwarding from [::1]:8080 -> 8080
Then navigate to http://localhost:8080/debug to see a list of all the debug endpoints, as shown in figure D.3.
Figure D.3 The Istio Pilot debug endpoints
NOTE The debug endpoints contain sensitive information that could be misused if exposed. We recommend disabling the debug endpoints in production by setting the environment variable ENABLE_DEBUG_ON_HTTP to false during Istio installation. Doing so will break the functionality of tools dependent on those endpoints; however, in future releases, these endpoints will be exposed securely over xDS.
These endpoints can be logically grouped as follows:
-
Endpoints that represent the service mesh state as known to the Pilot:
-
Endpoints that represent the data-plane configuration as known to the Pilot:
-
-
/debug/config_distribution—Version status of all Envoys connected to this Pilot instance.
-
/debug/config_dump?proxyID=<pod>.<namespace>—Generates the Envoy configuration according to the current known state of Istio Pilot.
-
/debug/syncz—Displays the proxies managed by this Pilot. Additionally, it shows the latest nonce sent to the proxy and the latest nonce acknowledged. When those are the same, the proxy has the latest configuration.
-
As a service mesh operator, you will usually use the endpoints indirectly through other tools such as Kiali, istioctl, and so on. For example, the istioctl proxy-status command uses the /debug/syncz endpoint to check whether the proxies are synchronized. However, when the information provided by these tools is not enough, you can dig deeper using the debug endpoints on your own.
D.2.2 The ControlZ interface
The Istio Pilot comes bundled with an administrative user interface that enables inspecting the current state of the Pilot process and some minor configuration possibilities. This interface provides a quick lookup of information related to the Istio Pilot instance, as covered in table D.1.
Table D.1 Content in the ControlZ interface
To access the dashboard, port-forward it to your local environment using istioctl, and open it in your browser:
$ istioctl dashboard controlz deploy/istiod.istio-system http://localhost:9876
Besides looking up information related to the Istio Pilot in a simple web interface, the most common usage of the ControlZ dashboard is to change the logging scopes when you have to debug the Istio Pilot.