Interactive debugging using a source level debugger, as described in the previous chapter, can give you an insight into the way a program works, but it constrains your view to a small body of code. In this chapter, I will look at the larger picture to see if the system is performing as intended.
Programmers and system designers are notoriously bad at guessing where bottlenecks are. So, if your system has performance issues, it is wise to start by looking at the full system and then work down, using more sophisticated tools. In this chapter I begin with the well-known command, top, as a means of getting an overview. Often the problem can be localized to a single program, which you can analyze using the Linux profiler, perf. If the problem is not so localized and you want to get a broader picture, perf can do that as well. To diagnose problems associated with the kernel, I will describe the trace tools, Ftrace and LTTng, as a means of gathering detailed information.
I will also cover Valgrind which, because of its sandboxed execution environment, can monitor a program and report on code as it runs. I will complete the chapter with a description of a simple trace tool, strace, which reveals the execution of a program by tracing the system calls it makes.
Before diving into the tools, let's talk about what the tools will show you. As is the case in many fields, measuring a certain property affects the observation itself. Measuring the electric current in a line requires measuring the voltage drop over a small resistor. However, the resistor itself affects the current. The same is true for profiling: every system observation has a cost in CPU cycles and that resource is no longer spent on the application. Measurement tools also mess up caching behavior, eat memory space, and write to disk, which all make it worse. There is no measurement without overhead.
I've often heard engineers say that the results of a profiling job were totally misleading. That is usually because they were performing the measurements on something approaching a real situation. Always try to measure on the target, using release builds of the software, with a valid data set, using as few extra services as possible.
We will hit a problem immediately. While it is important to observe the system in its natural state, the tools often need additional information to make sense of the events.
Some tools require special kernel options, specifically from those listed in the introduction, perf, Ftrace, and LTTng. Therefore, you will probably have to build and deploy a new kernel for these tests.
Debug symbols are very helpful in translating raw program addresses into function names and lines of code. Deploying executables with debug symbols does not change the execution of the code but it does require that you have copies of the binaries and the kernel compiled with debug, at least for the components you want to profile. Some tools work best if you have these installed on the target system, perf, for example. The techniques are the same as for general debugging, as I discussed in Chapter 12, Debugging with GDB.
If you want a tool to generate call graphs, you may have to compile with stack frames enabled. If you want the tool to attribute addresses with lines of code accurately, you may need to compile with lower levels of optimization.
Finally, some tools require instrumentation to be inserted into the program to capture samples, so you will have to recompile those components. This applies to gprof for applications, and Ftrace and LTTng for the kernel.
Be aware that, the more you change the system you are observing, the harder it is to relate the measurements you make to the production system.