While there are many tools to view or analyze the current server performance and behavior, how do we examine historical activity? Most Linux systems rotate logfiles in /var/log for varying periods of time. Unfortunately, these are programs and system logs, not performance measurements.
When we installed the sysstat package in a previous recipe, we gained the use of the sar utility. Some argue that sar is the Swiss Army knife of metric collection. A simple invocation can display past data regarding memory, CPUs, IRQs, disk devices, networks, or even TTYs.
When administering a highly available server, there are a few things as helpful as performance trends. Let's examine them.
As sar and iostat are both part of the sysstat package, we recommend that you review the Evaluating the current disk performance with iostat recipe before continuing.
Collect some sample sar data by following these steps:
- Display the default
saroutput with the following command:sar - Show the disk device status every 5 seconds with this command:
sar -d 5 - View memory usage between 4:00 A.M. and 6:00 A.M. today with this command:
sar -r -s 04:00:00 -e 06:00:00
Examine the I/O statistics for any existing past dates by following these steps:
- Find the appropriate
sysstatlog directory:- Red Hat, Fedora, CentOS, and Scientific Linux should use the
/var/log/sadirectory - Debian, Mint, and Ubuntu users should use the
/var/log/sysstatdirectory
- Red Hat, Fedora, CentOS, and Scientific Linux should use the
- List the contents of that directory and choose a file. Files are simply binary formats containing
sardata for each retained date. Files are prefixed withsa. Thus,sa23is thesardata for the 23rd of the month. - Execute the following command to view past I/O statistics for the 3rd of the month:
sar -f /var/log/sysstat/sa03 -b
By default, sar operates in CPU mode. Simply using the command as named, we will receive CPU activity samples for every 10 minutes of the current day. Once sar produces this output, it exits. If we want the current data, we must invoke it much like we did with iostat.
In our second example, we've chosen to emulate the iostat output by providing a summary of disk activity every 5 seconds until we cancel the command. The -d argument tells sar to display the disk statistics. Just like iostat, sar accepts two optional parameters for interval and count. As we didn't specify a count, sar will print disk performance every 5 seconds.
The third example is where we finally begin leveraging the real power of sar. If we had examined our PostgreSQL log and noticed a large amount of idle queries between 4:00 A.M. and 6:00 A.M., we would need a method to obtain data for that time period. Well, sar has one argument (-s) to specify the start time of a data extract and another (-e) to set the end time. These parameters must be written in HH:MM:SS format, or sar will ignore them with an error. We also elected to use the -r argument to display memory usage data, just to illustrate another metric that sar can expose.
Our final example depends entirely on what Linux distribution we're using. Unfortunately, each stores its collected sar data in different areas within /var/log. With that said, the directory assigned to sysstat for data storage keeps a default of 7 days worth of historical information for analysis.
Everyday, this data is collected in a file prefixed with sa and suffixed with the current month's day. On weeks that span 2 months, the count simply restarts with 01. Once again, we use a different output mode for sar and display the I/O activity.
Seven days may not be enough for some administrators. To increase this amount, modify /etc/sysconfig/sysstat or /etc/sysstat/sysstat and change the HISTORY setting to the desired amount of days to retain data. For example, to keep 3 weeks of records, we could use this:
HISTORY=30