CHAPTER 10. Troubleshooting and Maintaining Linux
Exam objectives in this chapter
■ Monitoring Tools
■ Analyzing Logs
■ Backing up and Restoring
Introduction
This chapter covers common tools for managing a system, including monitoring performance and logs and backing up data. If your system seems to be running slowly or having some other issues, it's very helpful to have some historical data to compare, so it is advisable to run a few of them now and then under normal load, so you have a good idea what “normal" looks like.
Monitoring Tools
Linux provides a number of handy tools for reviewing system status and other statistics. Not all of them may be included in a default installation, but they are all supported and can be made to work with a little effort.
Commands
The usage of the most popular utilities is described below.
SAR
sar can be used to gather both current and historic system statistics for the system processor. By default, sar displays information periodically gathered and stored in files at /var/log/sysstat/sa*, with the last two numbers of the sa* files corresponding to the calendar day the information was collected. Some other useful options include sar -A to show all statistics and sar -n DEV to show network statistics. An example showing network statistics is shown in Figure 10.1.
Fast Facts
You can control the collection of current information by appending two numbers at the end of the command.
■ The first will set the collection interval in seconds.
■ The second tells how many times it should gather the information.
■ sar -n DEV 5 4 shows network statistics for 5-s time periods a total of four times.
IOSTAT
The iostat command shows both CPU and disk utilization statistics and can also show remote Network File System (NFS) drive systems. It was covered in Chapter 5, “Using BASH."
VMSTAT
The vmstat command uses information in /proc/meminfo, /proc/stat, and /proc/*/stat to show virtual memory and other system usage statistics, including disk and processor usage. vmstat can be followed with two numbers indicating how long to wait and how many times to run. Memory statistics using units of blocks equate to 1024 bytes of memory per block in current Linux kernels. Using vmstat without options gives a brief overview of memory and CPU statistics. The -d option shows more detailed statistics related to the disk drives in the system.
UPTIME
The uptime command gives a quick one-line display showing the current system time, how long the system has been up, the current number of users, and the load average over the last 1, 5, and 15 min. The load average is a count of processes either currently being handled by the processor or waiting to be run.
TOP
The top command shows a current running tally of system usage and an ongoing list of processes. This is covered in Chapter 5, “Using BASH."
Load Average
Many of the commands that show system utilization display three numbers that represent the system load average. You can see it in the upper section of the top command and on the end of the uptime command, and it also shows up in the w command.
Exam Warning
You may have noticed in the exam objectives that the focus is on commands; however, there are several key statistics that are objectives as well, load average being one. You should be familiar with the commands to know which ones will display load average and other utilization statistics.
The load average is a three part statistic that indicates, on average, how many processes are currently running or actively waiting to be run by the processor and is essentially a count of how many programs are waiting to be run at a given moment, with a damping factor to give a more accurate time-weighted average. The 1-min average has a smaller damping factor and is allowed to swing more quickly, whereas the 15-min average has a higher damping factor and gives a better long-term overview of the system load.
Analyzing Logs
A record of events is kept in a variety of log files, and these files can be crucial for troubleshooting problems, tracking down failures, and finding security issues. Different distributions may have slightly different log files and directory structures. The syslog configuration is defined in the /etc/syslog.conf file and is used to determine which system processes record their events to what files and what level of logging to use.
System events are categorized into eight levels, listed in Table 10.1 and can be produced by one of 21 predefined facilities or system programs.
Crunch Time
The following log files and their usage are very important for the exam.
■ /var/log/messages: It is the standard location for most system events. All info, notice, and warning messages are sent here by default, along with some authorization and cron events.
■ /var/log/syslog: It collects information about authorization events, time changes, and system statistics events if you installed the sysstat package.
■ /var/log/maillog: It is the events related to the mail service.
■ /var/log/secure: It is used by Red Hat-based distributions to record authorization messages for sshd, sudo, and other authorization events.
■ /var/log/lastlog: It is the information about user login times and is a binary file. lastlog uses this file to show the last time a user has logged in to the system.
Rotating Logs
Over time, log files keep growing and would eventually fill whatever hard drive they are written to, so it is important to implement a log management scheme. The standard program for rotating log files is logrotate, and by default, it is run daily by the cron schedule. It can be set to rotate, compress, remove, and/or mail log files based on specified times. The /etc/logrotate.conf file contains the options used by logrotate.
Searching and Interpreting Log Files
Most log files are simple text with valuable data that can be read with your favorite paging application, such as less. If you are searching for a specific type of event or troubleshooting a particular issue, all the other events will simply be cluttered and get in the way.
Fast Facts
There are a number of useful commands to search log files.
■ The output of a command can be piped into grep and used to filter what shows up on the screen, or you can feed files directly into grep. The syntax is grep [options] PATTERN [file]. When used, grep will search the named file (or standard input) for lines that match the PATTERN and prints any matching lines. The number of options available is large, and regular expressions can be used.
■ tail shows the last 10 lines of text in a file, or with the -n NUM option, it shows the last NUM lines. The -f option f, for follow, refreshes the screen with new information as the log file gets updated. It may be convenient to open a separate terminal window just for running tail -f.
■ The awk utility is a programming language specialized for text and string functions. The general syntax is awk [options] -f script-file filename. If the filename is left off, awk will use STDIN. awk is much more flexible than grep.
■ sed is a stream editor, intended to edit files using scripts, and is useful for doing bulk search and replace functions within multiple files and acts on files one line at a time. The syntax is sed [options] -f script_file filename.
Backing Up and Restoring
Backups can be done at different levels. File level backups run at the operating system level and are convenient for dealing with individual files, but they require a working operating system to restore to. Some software offers online and offline backups. An online backup uses the database or e-mail system to read each record or mailbox and make a copy. An offline backup requires the software that manages the database or the e-mail system be shut down. It is also possible to do an offline backup of entire drive systems and computers themselves by booting them from an alternate device.
Fast Facts
There are two general kinds of backups – complete backups and partial backups.
■ Complete backups back up all the data you have selected.
■ Partial backups back up only the data that has changed since the most recent complete backup.
■ Partial backups are divided into differential and incremental.
■ Differential backups contain everything that has changed since the last complete backup.
■ Incremental backups copy everything that has changed since the last partial backup.
■ To restore data from incremental backups, you need the last complete backup and all the incremental backups.
Copying Data
File-level copies of data over a network can be done using rsync and ftp. By copying over a network, it is possible to easily send your backups to off-site locations, although bandwidth may become an issue.
RSYNC
rsync doesn't just copy data, but it synchronizes the data, so if you use rsync on a directory, the other end will be made to be an exact duplicate of the original. It can read data in blocks and only copy the portion of files that have changed, called the delta-transfer, which means it won't copy an entire file if only a small part of it has been modified. rsync can either be run using a remote shell or be configured to run as a service, using its own Transmission Control Protocol (TCP) socket and transport system.
The basic syntax for moving files using rsync is shown in Table 10.2.
FTP
Another common way to transfer files is by using ftp and can be used for backups. Using ftp requires both a client and a server that the client connects to and was discussed in more depth in Chapter 8, “Installing, Configuring as a Server."
Archiving and Restoring Commands
The following commands are the standard utilities for backing up data; each having their own strengths and weaknesses.
CPIO
The cpio program uses binary archive files. It has three basic modes:
■ copy-out mode creates an archive.
■ copy-in mode reads from an archive.
■ copy-pass mode transfers files from on place to another without creating the actually archive as a middle step.
In copy-out mode, cpio accepts a list of files that you want to put into your archive from STDIN and sends the output to STDOUT, so putting cpio to use requires some command-line redirection. The output can be a regular file, device file, or network location.
In copy-in mode, cpio reads an archive file from STDIN and spits out the files into the current directory. The specific files to be extracted from the archive can be selected using a pattern, where pattern is standard filename wildcard. The following commands demonstrate how this is done using cat and cpio:
cat archive_file | cpio -i “pattern"
TAR
tar is used to store files in and extract files from an archive file, called a tarfile. This tarfile may be created and stored on any rewriteable medium, such as a tape drive or hard disk. The following is the syntax for creating a tarfile with tar:
tar c [ bBeEfFhiklnopPqvwX [ 0-7 ] ] [ block ] [ tarfile ] [ exclude-file ] {-I include-file | -C directory | file | file }
The most commonly used options for tar are listed in Table 10.3.
Did You Know?
The tar command has a long history and is very versatile, being used for creating archives that are used for software source packages and being used as a way to back up and extract files.
■ tar was originally designed to make tape archives.
■ tar can be used to archive files to tapes or disks.
■ tarfiles can be compressed.
DUMP
dump can archive entire filesystems and works by examining files on a target filesystem and determining the files that need to be backed up. These files are then copied to the backup medium, usually hard disk or tape, and if the dump is larger than the backup medium, the dump is split and copied to multiple volumes. On most media, the size is determined by writing until an end-of-media indication is returned.
The following is the basic syntax for dump:
dump -0 -A [archive file] -f [destination file or device] [mountpoint of a filesystem, or a list of files and directories to be backed up]
In the example above, the -0 option tells dump to perform a full backup. The -A option is used to designate the archive file, which is read by the restore command (described below) when restoring the backed up files. -f is used to identify the target file or device that will host the “dumped" files.
Restore
restore performs a restore of files that have been backed up using dump and can be used to restore a full backup of a filesystem and apply any subsequent incremental backups. restore can be used to restore individual files or directories from a full or partial backup file archive and can operate across a network to restore filesystems or files and directories on remote systems.
The following code is the basic syntax for restore that will restore all files from the identified archive (mounted at /dev/nst0) in the current directory.
restore rf /dev/nst0
In this example, the r option retrieves all files from the archive and the f option is used to designate the archive (/dev/nst0).
DD
dd is useful for making and copying disk images, backing up and moving disks, and duplicating filesystems, as it makes an exact clone of a hard disk, including all blank space. To use it, the source disk must be unmounted and the output destination must be at least as large as the source.
To illustrate, the following syntax will create an image file in my home directory entitled, cdbackup.iso, from the CD in my CD-ROM drive:
dd if=/dev/cdrom of=/home/brian/cdbackup.iso
The option if designates the input file (the CD that is mounted in my CD-ROM drive), and of designates the output file, cdbackup.iso in my home directory.
Writing to Removable Media (CD-RW, DVD-RW)
Creating CDs and DVDs is a relatively commonplace activity that people do for a variety of reasons: backing up files on a computer, creating a photo CD, and transferring music and videos, among others. At a high level, there are two steps in the process of burning a CD or DVD:
1. Create an image file for the CD or DVD, which involves creating filesystem on the medium and adding data to an image file.
2. Apply the image to the CD or DVD.
Regardless of the media, nothing can be stored until a filesystem has been created on it. Unlike a hard disk, a CD-R or DVD-R is writeable only once, which means that if you create the empty filesystem as a step by itself, it will remain empty forever. Burning a CD or DVD involves creating a filesystem while transferring the files to the medium. The command that accomplishes the first part of the process – creating an image file (an .iso file) that includes both the filesystem and the actually files that will be transferred to the medium – is mkisofs. You can use the following syntax for creating a typical CD or DVD image:
mkisofs -r -o [filename of CD or DVD image.iso] [directory where files to be copied are located]
The option -r sets the permissions of all files on the CD or DVD to be public readable and enables RockRidge-extensions. If you do not use the -r option, the files will be copied with the permissions that were assigned to them in their original locations. The second step is to apply the image to the CD or DVD using a separate program, cdrecord. The following syntax will mount your image file at the mount point, /cdrom:
mount -t iso9660 -o ro, loop=/dev/loop0 [filename of CD or DVD image] /cdrom
The -t option, iso9660, identifies the filesystem of the image file as that of a CD or DVD. Once the CD or DVD image file is mounted, you can navigate to the /cdrom directory (using cd /cdrom) and verify that you have included all the files you wanted and that the directory structure is sound. Then, insert the appropriate blank media in your CD or DVD burner. You will need to figure out the SCSI device address of your CD or DVD burner. As root, issue the command cdrecord –scanbus, and then you can issue the command (substituting the correct device) address if necessary:
cdrecord dev=0,0,0 [filename of CD or DVD image.iso]
Summary of Exam Objectives
There are so many monitoring and measuring utilities available on Linux that there is no excuse not to monitor your system. The sar and top programs will produce detailed reports on how different components of your system are performing at specified intervals and in real time, respectively. You will want to pay special attention to the programs that calculate and display the load average: top, w, and uptime, among others.
Although monitoring will help you check up on how things are going with your system, you will need to know where to go to find information when the inevitable happens and things go badly. There are more logs; many of which reside under the /var/log directory structure, and knowing the various logs under /var/log is important. With the amount of data that is captured in each log, you need good searching and sifting tools such as grep, sed, awk, and tail.
Restoring data that has been lost due to a hard disk crash or user error is inevitable. Creating backup sets with tar and dump, restoring files with tar and restore, and using the dd command to create disk images and apply them to other media were covered.
1. You have been asked to back up users' data on a particular server before a core application is upgraded. Because of the amount of data, you need ensure that these files will fit on a remote hard disk. What command would you use to ensure that the smallest possible size of the backup file?
A. tar -cvf userdata.tar /home/*
B. tar -xjvf userdata.tar /home/*
C. tar -cjvf userdata.tar /home/*
D. tar -xvf userdata.tar /home/*
2. You are replacing Michael's computer and have backed up the hard disk to an external hard disk (/mount/usbhdd) using the following syntax: dump -0uf -A michaelhdd.archive -f /mount/usbhdd/michaelhdd.backup /. You want to restore the backup on another hard disk in the new computer. After booting the new computer and mounting the external hard disk, what command do you use?
A. restore -rf /mount/usbhdd/michaelhdd.backup
B. dump -xf /mount/usbhdd/michaelhdd.backup
C. tar -xvf /mount/usbhdd/michaelhdd.backup
D. restore -rf /mount/usbhdd/michaelhdd.archive
3. Lately you have been hearing reports that your Linux server is slow to respond and you have a suspicion that there are applications that are consuming more than their fair share of the server's memory. What key combination would you press while top is running so that the running programs are sorted by their respective percentage of memory utilization?
A. F, then M
B. F, then n
C. F, then k
D. F, then l
4. Users are reporting that a particular corporate server responds slowly for around 30 min between 10:30 a.m. and 11:00 a.m. You decide to run sar at regular intervals during this time to capture statistics on the server's network performance. What syntax would you use to capture six sets of these metrics every 10 min?
A. sar -A DEV 600 6
B. sar -n DEV 600 6
C. sar -n DEV 6 600
D. sar -A DEV 6 600
5. You are in the process of setting up an active mode FTP server. Whenever you try to connect, you can connect to the server, but you cannot enter a username and password. You made sure that TCP ports 21 and 20 are open on the server. What is the most probable cause of the problem?
A. TCP ports 1022 and below are open on the server
B. TCP ports 1023 and above are open on the server
C. TCP ports 1022 and below are closed on the server
D. TCP ports 1023 and above are closed on the server
Answers
1. Correct answer and explanation: C. Answer C is correct because the command with the -c and -j switches creates the archive and compresses the files in the archive with bzip, respectively.
Incorrect answers and explanations: A, B, and D. Answer A is incorrect because while the -c switch creates the tarfile, there is nothing to instruct tar to compress the files in the archive. Answer B is incorrect because the -x switch is used to extract the files from the archive, which in this case contains compressed files. Answer D is incorrect because the -x switch is used to extract the files from the archive.
2. Correct answer and explanation: D. Answer D is correct because the restore command with the -rf switches is required to extract all files from the archive and specify the archive file.
Incorrect answers and explanations: A, B, and C. Answer A is incorrect because the output file is specified and the -f switch is used to specify the archive file. Answer B is incorrect because dump is used to back up the files and has no restore functionality. Answer C is incorrect because only restore can be used to extract files from a backup set created with dump.
3. Correct answer and explanation: B. Answer B is correct because n will display the percentage of memory usage.
Incorrect answers and explanations: A, C, and D. Answer A is incorrect because M will display the CPU time in hundredths of a second. Answer C is incorrect because k will sort by percentage of CPU utilization. Answer D is incorrect because l displays utilization based on CPU time.
4. Correct answer and explanation: B. Answer B is correct because the switch and parameters needed to collect the desired statistics -n DEV 600 6. The –n switch tells sar to capture network statistics, the next parameter, 600, specifies the interval in seconds, and the last interval, 6, specifies that six sets of statistics should be captured.
Incorrect answers and explanations: A, C, and D. Answer A is incorrect because -A tells sar to capture all available system statistics. Answer C is incorrect because the last two parameters are inverted, which means that 600 sets of statistics will be captured at 6-s intervals. Answer D is incorrect because the -A switch is used, and the parameters are inverted.
5. Correct answer and explanation: D. Answer D is correct because TCP ports 1023 and above need to be open to ensure that all the connections required for active mode can be established.
Incorrect answers and explanations: A, B, and C. Answers A and C are incorrect because active mode FTP uses TCP ports 1023 and above, not below. Answer B is incorrect because the connection could be established if the ports were open.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.