The preceding EFK stack collects Pods' logs only, because Fluentd is monitoring /var/log/containers/* in the Kubernetes node host. It is good enough to monitor an application's behavior, but, as a Kubernetes administrator, you also need some Kubernetes system logs such as master and node logs.

There is an easy way to achieve Kubernetes system log management that integrates with the EFK stack; add a Kubernetes Event Exporter, which keeps monitoring a Kubernetes event. When the new event has occurred, send a log to Elasticsearch. So, you can monitor a Kubernetes event with Kibana as well.

We have prepared an Eventer (Event Exporter) add-on (https://raw.githubusercontent.com/kubernetes-cookbook/second-edition/master/chapter9/9-1/eventer.yml). It is Heapster (https://github.com/kubernetes/heapster), based and expected to run on top of EFK add-ons. We can use this Eventer to monitor Kubernetes events through EFK:

1. Add eventer to your existing Kubernetes cluster:

$ kubectl create -f https://raw.githubusercontent.com/kubernetes-cookbook/second-edition/master/chapter9/9-1/eventer.yml
deployment "eventer-v1.5.2" created
serviceaccount "heapster" created
clusterrolebinding "heapster" created

2. Make sure Eventer Pod's STATUS is Running:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                        READY     STATUS    RESTARTS   AGE
kube-system   elasticsearch-logging-v1-776b8b856c-9vvfl   1/1       Running   0          9m
kube-system   elasticsearch-logging-v1-776b8b856c-gg5gx   1/1       Running   0          9m
kube-system   eventer-v1.5.2-857bcc76d9-9gwn8             1/1       Running   0          29s
kube-system   fluentd-es-v1.22-8prkn                      1/1       Running   0          9m
...

3. Use kubectl logs to keep observing Heapster and whether it can capture the event:

$ kubectl logs -f eventer-v1.5.2-857bcc76d9-9gwn8 --namespace=kube-system
I0327 03:49:53.988961       1 eventer.go:68] /eventer --source=kubernetes:'' --sink=elasticsearch:http://elasticsearch-logging:9200?sniff=false
I0327 03:49:53.989025       1 eventer.go:69] Eventer version v1.5.2
I0327 03:49:54.087982       1 eventer.go:95] Starting with ElasticSearch Sink sink
I0327 03:49:54.088040       1 eventer.go:109] Starting eventer
I0327 03:49:54.088048       1 eventer.go:117] Starting eventer http service
I0327 03:50:00.000199       1 manager.go:100] Exporting 0 events

4. For testing purposes, open another terminal, and then create a nginx Pod:

$ kubectl run my-nginx --image=nginx
deployment "my-nginx" created

5. Observe Heapster's log; some new events have been captured:

I0327 03:52:00.000235       1 manager.go:100] Exporting 0 events
I0327 03:52:30.000166       1 manager.go:100] Exporting 8 events
I0327 03:53:00.000241       1 manager.go:100] Exporting 0 events

6. Open Kibana and navigate to Settings | Indices| Add New. This will add a new index. 

7. Put the Index name as heapster-*, set the time-field name as Metadata.creationTimestamp, and then click Create:

8. Go back to the Discover page, and then choose the heapster-* index from the left-hand panel.

9. Select (click the Add button) Message, Source.component, and Source.host:

10. Now you can see the Kubernetes system log, which shows the nginx Pod creation event as follows:

Now you can monitor not only the application log, but also the Kubernetes system log in the EFK stack. Through switching indexes between either logstash-* (application log) or heapster-* (system log), you have a flexible log management environment.