In this chapter, we are going to look at commands which will come in useful when troubleshooting your containers, all the commands we will look at are part of the core Docker Engine, we will also look at a way by which you can debug your Dockerfiles.
Once we have finished with the Troubleshooting commands, we will look at how we can monitor our containers using cAdvisor with a Prometheus backend fronted by a Grafana dashboard – don't worry, it is not as complicated as it sounds.
Computer programs (software) sometimes fail to behave as expected. This is due to faulty code or due to the environmental changes between the development, testing, and deployment systems. Docker container technology eliminates the environmental issues between development, testing, and deployment as much as possible by containerizing all the application dependencies. Nonetheless, there could still be anomalies due to faulty code or variations in the kernel behavior, which needs debugging. Debugging is one of the most complex processes in the software engineering world and it becomes much more complex in the container paradigm because of the isolation techniques. In this section, we are going to learn a few tips and tricks to debug a containerized application using the tools native to Docker, as well as the tools provided by external sources.
Initially, many people in the Docker community individually developed their own debugging tools, but later Docker started supporting native tools, such as exec
, top
, logs
, events
, and many more. In this section, we will dive deep into the following Docker tools:
exec
ps
top
stats
events
logs
attach
We shall also consider debugging a Dockerfile.
The docker container exec
command provided the much-needed help to users, who are deploying their own web servers or other applications running in the background.
Now, it is not necessary to log in to run the SSH daemon in the container.
First, launch a container:
docker container run -d --name trainingapp training/webapp:latest
Second, run the docker container ps
command to get the container ID. Now you have the container ID you can run the docker container exec
command to log in to the container using either the container ID or as we have named it trainingapp
you can use that:
docker container exec -it 32005e837724 bash
It is important to note that the docker container exec
command can only access the running containers, so if the container stops functioning then you need to restart the stopped container to proceed. The docker container exec
command spawns a new process in the target containers namespace using the Docker API and CLI.
A containers name space is what separates the containers from each other, for example you can have several containers all running the same process, but because the processes have been launched within each of the containers namespace they are isolated from one another. A good example of this is are MySQL processes, on a traditional server trying to run more than one MySQL server process will mean that you need to start the process on different ports, use different lock, PID and log files as well as different init scripts.
As Docker is isolating each MySQL server process all you need to worry about is that if you are exposing the MySQL port on the host machine is that you don't assign it on the same port as another container.
So, if you run the ps -aef
command inside the target container, it looks like this:
Here, python app.y
is the application that is already running in the target container, and the docker container exec
command has added the bash
process inside the container. If you run kill -9 59
(replacing the 59
with the PID of your own bash
process), you will be automatically logged out of the container.
It is recommended that you use the docker container exec
command only for monitoring and diagnostic purposes, and I personally believe in the concept of one process per container, which is one of the best practices widely accentuated.
The ps
command, which is available inside the container, is used to see the status of the process. This is like the standard ps
command in the Linux environment and is not a dockercontainerps
command that we run on the Docker host machine.
This command runs inside the Docker container:
Use ps --help <simple|list|output|threads|misc|all>
or ps --help <s|l|o|t|m|a>
for additional help text.
You can run the top command from the Docker host machine using the following command:
docker container top CONTAINER [ps OPTIONS]
This gives a list of the running processes of a container without logging into the container, as follows:
The within the container the top command provides information about the CPU, memory, and swap usage just like any normal Linux host:
In case you get the error as error - TERM environment variable not set
while running the top
command inside the container, perform the following steps to resolve it.
Run echo$TERM
and if you get the result dumb
, then, run the following command:
export TERM=dumb
This will resolve your error and you can run the top
command.
The docker container stats
command provides you with the capability to view the memory, CPU, and the network usage of a container from a Docker host machine, as illustrated here. Running the following command:
docker container stats 32005e837724
Gives you the following:
You can run the stats
command to also view the usage for multiple containers:
docker container stats 32005e837724 5e33f02f5fd2 7c9cf27ff46a
Since Docker 1.5, you have been able to access to container statistics read only parameters. This will streamline the CPU, memory, network IO, and block IO of your containers.
This helps you choose the resource limits and in profiling. The Docker stats utility provides you with these resource usage details only for running containers.
You can get detailed information using the endpoint APIs at the following URL https://docs.docker.com/engine/api/v1.26/.
Docker containers will report the following real-time events: create
, destroy
, die
, export
, kill
, omm
, pause
, restart
, start
, stop
, and unpause
. Let's pause
and unpause
our container:
If you specify an image it will also report the untag
and delete
events.
Using multiple filters will be handled as an AND
operation, for example:
docker events --filter container=32005e837724 --filter event=pause --filter event=unpause --since 12h
Preceding will display all pause
and unpause
events for the container a245253db38b
for the last 12 hours:
Currently, the supported filters are container
, event
, and image
.
This command fetches the log of a container without logging into the container. It batch-retrieves logs present at the time of execution. These logs are the output of STDOUT and STDERR. The general usage is shown in:
docker container logs [OPTIONS] CONTAINERID
The --follow
option will continue to provide the output till the Docker logs command is terminated printing any new log entries to the screen in real time,-t
will provide the timestamp, and --tail=<number of lines>
will show the number of lines of the log messages of your container:
docker container logs 32005e837724
docker container logs -t 32005e837724
We also used the docker container logs
command in previous chapters to view the logs of our database containers.
This command attaches the running container and it is very helpful when you want to see what is written in stdout
in real time, let's launch new test container which outputs something to stdout
:
docker container run -d --name=newtest alpine /bin/sh -c "while true; do sleep 2; df -h; done"
Now we can attach to the container using the following command to see the output;
docker container attach newtest
By default, this command attaches stdin
and proxies signals to the remote process. Options are available to control both behaviors. To detach from the process, use the default Ctrl + Q sequence.
18.118.217.168