In this chapter, you will look at an open source project called cAdvisor, which stands for Container Advisor. The complete source code can be found at https://github.com/google/cadvisor. This chapter uses version v0.39.3 of the project. The project is used to collect resource usage and performance data on running containers. cAdvisor supports Docker containers, and this is specifically what you are going to look at in this chapter.
Using system calls to monitor filesystem
Using cgroups
Collecting machine information using /proc and /sys
Source Code
The source code for this chapter is available from the https://github.com/Apress/Software-Development-Go repository.
Running cAdvisor
When cAdvisor runs, it collects different information related to the machine and containers, which can only be done if it has root access.
In the next section, you will explore further the cAdvisor UI and concepts that are related to the project.
Web User Interface
Make sure you have your cAdvisor running locally and open your browser to access it via URL http://localhost:8080. Let’s understand some of the data that is presented on the webpage.
The information you see on your local machine might be different from what you read in this book. It depends on the operating system or Linux distribution you are using.
In the next section, you will dive into the internals of cAdvisor and learn how it is able to do all these things in the code.
Architecture
Components
Events Channel | Channel used to report creation or deletion of containers |
InMemoryCache | Cache used to store metric information relevant to all containers being monitored |
Container Watcher | Watcher that monitors container activities |
Containers | Different containers monitored by cAdvisor |
Machine Info | Information related to the local machine that cAdvisor is running on |
Plugins | The different container that cAdvisor supports: Docker, Mesos, CRIO, Systemd, and ContainerD |
Handlers | HTTP handlers that take care of requests for metrics and other relevant APIs exposed by cAdvisor |
In the next few sections, you will look at different parts of cAdvisor and how they work.
Initialization
Setting up cache for storing a container and its metrics
Setting up Manager, which performs all the major processing to monitor containers
Setting up HTTP handlers to allow the web user interface to get metric data for different containers
Start collecting containers and metrics by starting up the Manager
Manager
The interfaces and implementation are found inside manager.go.
The main job of Manager is to monitor containers, but before it is able to do that, it needs to find out what containers are available and how to monitor them. Let’s take a look at the first step, which is finding out what containers will be monitored.
Creating a containerData struct that is populated with container-related information. In this case, it’s populated with information regarding the /sys/fs/cgroup directory.
Creating a ContainerHandler and CollectManager that will handle everything related to this particular container (in this case /sys/fs/cgroup) and collecting all the necessary metric information.
Once all structs have been initialized successfully, it will call Start() of the containerData struct to start monitoring.
From the steps above, it is clear that cAdvisor is monitoring activities that are happening inside the /sys/fs/cgroup directory. As you learned in Chapter 4, this directory refers to cgroups, which is the cornerstone of Docker containers.
After all containers have been set up to be watched, cAdvisor will be informed about any changes to them. This job is done by the goroutine shown in the above code snippets. In the next section, you will look at how cAdvisor uses inotify, which is provided by the Linux operating system to let applications to be notified if any activities are detected for the directories that are watched.
Monitoring Filesystem
In the previous section, you learned that cAdvisor monitors and listens for events for /sys/fs/cgroup and its subdirectories. This is how cAdvisor knows what Docker containers are created or deleted from memory. Let’s take a look at how it uses inotify for this purpose.
The function converts the events received into internal events that the code understands: watcher.ContainerAdd and watcher.ContainerDelete. These events are broadcast internally for other parts of the code to process.
Information from /sys and /proc
In Chapters 2 and 3, you learned about the /sys and /proc filesystems and what kind of system-related information can be found. cAdvisor uses the same way to collect machine information that is reported as part of the metric information.
Client Library
In the repository inside the chapter18 folder, there are examples of how to use the cAdvisor client library to communicate with cAdvisor. The examples show how to use the client library to get container information, event streaming from cAdvisor, and so on.
Summary
In this chapter, you learned about installing and running cAdvisor to monitor metrics of your local machine and Docker containers. The tool provides a lot of information that shows the performance of the different containers that are running on a machine. This chapter discussed how cAdvisor collects metric information for containers and local machines using the knowledge you learned in previous chapters.
cAdvisor provides much more functionality than what was discussed in this chapter. For example, it has built-in support for exporting metrics to a Prometheus, it provides an API that can be used to integrated with other third-party or in-house tools to monitor container performance, and more.