4.2. System logs
System logs are critical components of an operating system that allow us to track and debug errors. They also provide more in-depth information about system performance and behavior.
4.2.1. Linux syslog
The system logging utility that we use in Linux is called “syslog”. The syslog bundled inside Linux is derived from BSD sources and is bundled natively into the operating system. It is turned on by default on Linux. This tool allows you to trap kernel messages and capture system messages, with capabilities to customize the level of criticality that you choose to capture.
Syslog provides two daemons as part of the package, syslogd and klogd. The syslogd daemon provides the system logging facilities, and klogd provides the kernel logging facility.
Daemons are processes that runs continuously in the operating system and are started during the runlevel processing at boot time (for example, /etc/rc.d/rc<N>.d/S06syslog).
The core configuration file in the syslog configuration is located in /etc/syslog.conf. It lists the types of errors, levels, and where in the file the errors be directed to.
The /etc/syslog.conf file contains logging parameters in the form of application.levelofseverity. The syslog.conf allows multiple entries of the same application error in different lines or separated by semicolons, as shown in the sample syslog configuration file in Example 4-6.
Example 4-6. Sample of /etc/syslog configuration file
# /etc/syslog.conf kern.warn;*.err;authpriv.none /dev/tty10 kern.warn;*.err;authpriv.none|/dev/xconsole *.emerg * *.* /dev/tty12 mail.* -/var/log/mail news.crit -/var/log/news/news.crit news.err -/var/log/news/news.err news.notice -/var/log/news/news.notice *.=warn;*.=err;*.crit-/var/log/warn *.*;mail.none;news.none-/var/log/messages |
The incoming logs are basically logged to files accordingly to their classification. Some of the common application classifications are listed in Table 4-1.
| auth | Authentication programs like login, telnet, ssh |
| authpriv | Authentication privileges |
| cron | cron daemons |
| daemon | Any other daemons which did not fall into the standard list |
| ftp | File transfer protocol service |
| kern | Linux kernel itself |
| lpr | Line printing service |
| auth | Authentication programs like login, telnet, ssh |
| Mail service daemon | |
| news | News service daemon |
| security | Miscellaneous security application |
| syslog | Syslog daemon |
| user | Generic User Level messages |
| uucp | UNIX-to-UNIX copy service application |
| local0-7' | Use by any application or daemons to write to local console |
Based on the application classification, you can filter the messages based on their severity levels. Table 4-2 lists some of the severity levels that you can define inside your /etc/syslog.conf. They are listed in increasing severity.
| none | Do not log message |
| debug | Debugging messages |
| info | Informational messages |
| notice | Notice which denote something is not amiss |
| warning | Warning condition |
| err | Error condition |
| crit | Critical errors that should be checked immediately |
| alert | Severe error |
| emerg | A unrecoverable error has occurred. If this occurred at the kernel space, it is often followed by kernel panic or your system could risk corruption. |
System logs in a centralized environment
In an environment with more than two servers, syslog can be configured to allow servers to forward syslogs to a central server where all the system logs can be stored. Syslog allows this operation by forwarding the logs from the client to the server through the port 514/UDP.
First, you need to configure the syslog server to accept incoming logs. By default, this is turned off.
Server side
Update /etc/sysconfig/syslog file under the SYSLOGD_PARAMS option.
SYSLOGD_PARAMS="-r"
Restart the syslog service by using the command /etc/init.d/syslog restart or you can use the command kill -HUP 'cat /var/run/syslogd.pid'
Now you need to configure the client to forward the logs to the server. You can still have logs stored in the client machines as if it is a standalone server.
Client side
In the client side:
Update /etc/syslog.conf with the respective configuration.
In Example 4-7, all the logs will be forwarded to the p630sles /var/log/messages log file. You can customize it further to forward only necessary facility and criticality.
Example 4-7. Client syslog.conf
# /etc/syslog.conf kern.warn;*.err;authpriv.none /dev/tty10 kern.warn;*.err;authpriv.none|/dev/xconsole *.emerg * *.* /dev/tty12 mail.* -/var/log/mail news.crit -/var/log/news/news.crit news.err -/var/log/news/news.err news.notice -/var/log/news/news.notice *.=warn;*.=err;*.crit-/var/log/warn *.*;mail.none;news.none-/var/log/messages *.*;mail.none;news.none-p630sles@/var/log/messages
Restart syslog service using the command /etc/init.d/syslog restart
Test logging using the logger command:
logger -p local0.crit -t TEST testing
On the server side, check the /var/log/messages file. You should see that the “testing” message is logged as shown in Example 4-8 on page 178.
Example 4-8. Example of logger test with output in the /var/log/messages
Oct 21 13:18:52 p630sles kernel: Kernel logging (proc) stopped.
Oct 21 13:18:52 p630sles kernel: Kernel log daemon terminating.
Oct 21 13:18:53 p630sles exiting on signal 15
Oct 21 13:18:54 p630sles syslogd 1.4.1: restart (remote reception).
Oct 21 13:18:59 p630sles kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Oct 21 13:18:59 p630sles kernel: Inspecting
/boot/System.map-2.4.21-83-pseries64
Oct 21 13:18:59 p630sles kernel: Loaded 31486 symbols from
/boot/System.map-2.4.21-83-pseries64.
Oct 21 13:18:59 p630sles kernel: Symbols match kernel version 2.4.21.
Oct 21 13:18:59 p630sles kernel: Loaded 110 symbols from 8 modules.
Oct 21 13:19:44 lpar8 syslogd 1.4.1: restart.
Oct 21 13:19:47 lpar8 TEST: testing
Oct 21 13:19:49 lpar8 kernel: klogd 1.4.1, log source = /proc/kmsg started.
|
Besides configuring syslog by hand, SLES8 bundles a graphical YaST2 tool to help you with the configuration. Figure 4-6 on page 179 shows the YaST2 utility for configuring system logging.
Figure 4-6. SuSE YaST2 syslog configuration
Tip
If the error logs appear in IP addresses instead of hostname, add the respective server’s hostname and IP address into the /etc/hosts of the syslog server and restart syslog.