This recipe will take you through how to access Kubernetes logs and debug services locally.
Let's perform the following steps to view logs by using the various options that are available in Kubernetes:
- Get the list of pods running in the kube-system namespace. The pods running in this namespace, especially kube-apiserver, kube-controller-manager, kube-dns, and kube-scheduler, play a critical role in the Kubernetes control plane:
$ kubectl get pods -n kube-system
NAME READY STATUS RST AGE
dns-controller-6577fb57f7-hx9wz 1/1 Running 0 16d
etcd-manager-events-ip-172-20-8-2.ec2.internal 1/1 Running 0 16d
etcd-manager-main-ip-172-20-8-2.ec2.internal 1/1 Running 0 16d
kube-apiserver-ip-172-20-8-2.ec2.internal 1/1 Running 2 16d
kube-controller-manager-ip-172-20-8-2.ec2.int... 1/1 Running 0 16d
kube-dns-66d58c65d5-mw6n5 3/3 Running 0 16d
kube-dns-66d58c65d5-rntmj 3/3 Running 0 16d
kube-dns-autoscaler-6567f59ccb-c9rmv 1/1 Running 0 16d
kube-proxy-ip-172-20-32-123.ec2.internal 1/1 Running 0 16d
kube-proxy-ip-172-20-38-218.ec2.internal 1/1 Running 1 16d
kube-proxy-ip-172-20-45-93.ec2.internal 1/1 Running 0 16d
kube-scheduler-ip-172-20-58-244.ec2.internal 1/1 Running 0 3d6h
- View the logs from a pod with a single container in the kube-system namespace. In this example, this pod is kube-apiserver. Replace the pod's name and repeat this for the other pods as needed:
$ kubectl logs kube-apiserver-ip-172-20-58-244.ec2.internal -n kube-system
...
E1112 08:11:05.662027 1 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, Token has been invalidated]
I1112 09:09:39.448428 1 log.go:172] http: TLS handshake error from 124.84.242.10:49016: tls: first record does not look like a TLS handshake
I1112 09:30:00.726871 1 trace.go:81] Trace[76921086]: "GuaranteedUpdate etcd3: *coordination.Lease" (started: 2019-11-12 09:30:00.177607414 +0000 UTC m=+1250671.527180434) (total time: 549.223921ms):
As shown in the preceding output, you can find the time, source, and a short explanation of the event in the logs.
The output of the logs can become long, though most of the time all you need is the last few events in the logs. If you don't want to get all the logs since you only need the last few events in the log, you can add -tail to the end of the command, along with the number of lines you want to look at. For example, kubectl logs <podname> -n <namespace> -tail 10 would return the last 10 lines. Change the number as needed to limit the output.
- Pods can contain multiple containers. When you list the pods, the numbers under the Ready column show the number of containers inside the pod. Let's view a specific container log from a pod with multiple containers in the kube-system namespace. Here, the pod we're looking at is called kube-dns. Replace the pod's name and repeat this for any other pods with multiple containers:
$ kubectl -n kube-system logs kube-dns-66d58c65d5-mw6n5
Error from server (BadRequest): a container name must be specified for pod kube-dns-66d58c65d5-mw6n5, choose one of: [kubedns dnsmasq sidecar]
$ kubectl -n kube-system logs kube-dns-66d58c65d5-mw6n5 kubedns
- To view the logs after a specific time, use the --since-time parameter with a date, similar to what can be seen in the following code. You can either use an absolute time or request a duration. Only the logs after the specified time or within the duration will be displayed:
$ kubectl -n kube-system logs kube-dns-66d58c65d5-mw6n5 kubedns --since-time="2019-11-14T04:59:40.417Z"
...
I1114 05:09:13.309614 1 dns.go:601] Could not find endpoints for service "minio" in namespace "default". DNS records will be created once endpoints show up.
- Instead of pod names, you can also view logs by label. Here, we're listing pods using the k8s-app=kube-dns label. Since the pod contains multiple containers, we can use the -c kubedns parameter to set the target container:
$ kubectl -n kube-system logs -l k8s-app=kube-dns -c kubedns
- If the container has crashed or restarted, we can use the -p flag to retrieve logs from a previous instantiation of a container, as follows:
$ kubectl -n kube-system logs -l k8s-app=kube-dns -c kubedns -p
Now you know how to access pod logs through Kubernetes.