The syslog Protocol

In the early days of Unix, myriad logging methods were used to track system and application events. Applications used different logging methods, making it difficult for system administrators to troubleshoot issues.

In the mid-1980s, Eric Allman defined a protocol for logging events from his Sendmail mail application, called syslog. The syslog protocol quickly became a standard for logging both system and application events in Unix and made its way to the Linux world.

What made the syslog protocol so popular is that it defines a standard message format that specifies the timestamp, type, severity, and details of an event. That standard can be used by the operating system, applications, and even devices that generate errors.

The type of event is defined as a facility value. The facility defines what is generating the event message, such as a system resource or an application. Table 17.1 lists the facility values defined in the syslog protocol.

As you can tell from Table 17.1, the syslog protocol covers many different types of events that can appear on a Linux system.

Each event is also marked with a severity. The severity value defines how important the message is to the health of the system. Table 17.2 shows the severity values as defined in the syslog protocol.

Combining the facility and severity codes with a short informational message provides enough logging information to troubleshoot almost any problem in Linux.

The History of Linux Logging

Over the years there have been many open source logging projects for Linux systems. The following have been the most prominent:

• Sysklogd: The original syslog application, this program includes two programs, the syslogd program to monitor the system and applications for events and the klogd program to monitor the Linux kernel for events.
• Syslogd-ng: This program added advanced features, such as message filtering and the ability to send messages to remote hosts.
• Rsyslog: The project claims the r stands for rocket fast. Speed is the focus of the rsyslog project, and the rsyslog application had quickly become the standard logging package for many Linux distributions.
• Systemd-journald: This is part of the Systemd application for system startup and initialization (see Chapter 6). Many Linux distributions are now using this for logging. It does not follow the syslog protocol but uses a completely different way of reporting and storing system and application events.

The following sections dive into the details of the two most popular logging applications, rsyslog and systemd-journald.

Configuration

The rsyslog package uses the rsyslogd program to monitor events and log them as directed, using the /etc/rsyslog.conf configuration file to define what events to listen for and how to handle them. Many Linux distributions also use a /etc/rsyslog.d directory to store individual configuration files that are included as part of the rsyslog.conf configuration. This allows separate applications to define their own log settings.

The configuration file contains rules that define how the program handles syslog events received from the system, kernel, or applications. The format of an rsyslogd rule is as follows:

facility.priority action

The facility entry uses one of the standard syslog protocol facility keywords. The priority entry uses the severity keyword as defined in the syslog protocol, but with a twist. When you define a severity, syslogd will log all events with that severity or higher (lower severity code). Thus, the entry

kern.crit

logs all kernel event messages with a severity of critical, alert, or emergency. To log only messages with a specific severity, use an equal sign before the priority keyword:

kern.=crit

You can also use wildcard characters for either the facility or priority. The entry

*.emerg

logs all events with an emergency severity level.

The action entry defines what rsyslogd should do with the received syslog message. There are six action options you have available:

• Forward to a regular file
• Pipe the message to an application
• Display the message on a terminal or the system console
• Send the message to a remote host
• Send the message to a list of users
• Send the message to all logged-in users

Listing 17.1 shows the entries in the configuration file for an Ubuntu 18.04 system.

Listing 17.1: The rsyslogd.conf configuration entries for Ubuntu 18.04

auth,authpriv.*            /var/log/auth.log
*.*;auth,authpriv.none     -/var/log/syslog
kern.*                     -/var/log/kern.log
mail.*                     -/var/log/mail.log
mail.err                   /var/log/mail.err
*.emerg                    :omusrmsg:*

The first entry shown in Listing 17.1 defines a rule to handle all auth and authpriv facility type messages. This shows that you can specify multiple facility types by separating them with commas. The rule also uses a wildcard character for the priority, so all severity levels will be logged. This rule indicates that all security event messages will be logged to the /var/log/auth.log file.

The second entry defines a rule to handle all events (*.*) except security events (the .none priority). The event messages are sent to the /var/log/syslog file. The minus sign in front of the filename tells rsyslogd not to sync the file after each write, increasing the performance. The downside to this is if the system crashes before the next normal system sync, you may lose the event message.

The kern.* entry defines a rule to store all kernel event messages in a separate log file, located in the /var/log/kern.log file. This has become somewhat of a standard in Linux distributions.

The *.emerg entry defines a rule to handle all emergency events. The omusrmsg command indicates to send the event message to a user account on the system. By specifying the wildcard character, this rule sends all emergency event messages to all users on the system.

For comparison, Listing 17.2 shows the entries in the rsyslogd configuration file for a CentOS 7 system.

Listing 17.2: The rsyslog.conf configuration file for CentOS 7

*.info;mail.none;authpriv.none;cron.none   /var/log/messages
authpriv.*                                 /var/log/secure
mail.*                                     -/var/log/maillog
cron.*                                     /var/log/cron
*.emerg                                    :omusrmsg:*
uucp,news.crit                             /var/log/spooler
local7.*                                   /var/log/boot.log

Notice that Red Hat–based systems use the /var/log/messages file for informational messages and the /var/log/secure file for security messages.

Making Log Entries

If you create and run scripts on your Linux system (see Chapter 25), you may want to log your own application events. You can do that with the logger command-line tool:

logger [-isd] [-f file] [-p priority] [-t tag] [-u socket] [message]

The -i option specifies the process ID (PID) of the program that created the log entry as part of the event message. The -p option allows you to specify the event priority. The -t option allows you to specify a tag to add to the event message to help make finding the message in the log file easier. You can either specify the message as text in the command line or it as a file using the -f option. The -d and -u options are advanced options for sending the event message to the network. The -s option sends the event message to the standard error output.

An example of using logger in a script would look like this:

$ logger This is a test message from rich

On an Ubuntu system, you can look at the end of the /var/log/syslog file to see the log entry:

$ tail /var/log/syslog
...
Feb  8 20:21:02 myhost rich: This is a test message from rich

Notice that rsyslogd added the timestamp, host, and user account for the message. This is a great troubleshooting tool!

Finding Event Messages

Generally, most Linux distributions create log files in the /var/log directory. Depending on the security of the Linux system, many log files are readable by everyone, but some may not be.

As seen in Listing 17.1 and Listing 17.2, most Linux distributions create separate log files for different event message types, although they don’t always agree on the log file names.

It’s also common for individual applications to have a separate directory under the /var/log directory for their own application event messages, such as /var/log/apache2 for the Apache web server.

Since rsyslogd log files are text files, you can use any of the standard text tools available in Linux, such as cat, head, tail, and, of course, vi to view them. One common trick for administrators is to use the -f option with the tail command. That displays the last few lines in the log file but then monitors the file for any new entries and displays those too.

Configuration

The systemd-journald service reads its configuration from the /etc/systemd/journald.conf configuration file. When you examine this file, you’ll notice that there aren’t any rules defined, only settings that control how the application works:

• The Storage setting determines how systemd-journald stores event messages. When the setting is set to auto, it will look for the /var/log/journal directory and store event messages there. If that directory doesn’t exist, it stores the event messages in the temporary /run/log/journal directory, which is deleted when the system shuts down. You must manually create the /var/log/journal directory for the event messages to be stored permanently. If you set the Storage setting to persistent, systemd-journald will create the directory automatically. When set to volatile, it only stores event messages in the temporary directory.
• The Compress setting determines whether to compress the journal files.
• There are several file maintenance settings that control how much space the journal is allowed to use as well as how often to split journal files for archive, based on either time or file size.
• The ForwardToSyslog setting determines if systemd-journald should forward any received messages to a separate syslog program, such as rsyslogd, running on the system. This provides two layers of logging capabilities.

There are quite a few settings that allow you to customize exactly how systemd- journald works in your system. For a full list and explanation of all the settings, type man journald.conf at the command prompt.

Viewing Logs

The systemd-journald program doesn’t store journal entries in text files. Instead it uses its own binary file format that works like a database. While this makes it a little harder to view journal entries, it does provide for quick searching for specific event entries.

The journalctl program is our interface to the journal files. The basic format for the journalctl command is as follows:

journalctl [options] [matches]

The options control data returned by the matches is displayed. Table 17.3 lists the options available.

The matches parameter defines what type of journal entries to display. Table 17.4 lists the matches available.

The journalctl command is great for when you are looking for specific event entries in the journal; it allows you to filter out events using the matches and determine how to display them using the options, as shown in Listing 17.3.

Listing 17.3: Output from the journalctl command

[rich@localhost ~]$ journalctl -r _TRANSPORT=kernel
-- Logs begin at Fri 2019-02-08 12:47:04 EST, end at...
Feb 08 12:48:36 localhost.localdomain kernel: TCP: lp registered
Feb 08 12:48:17 localhost.localdomain kernel: fuse init (API version 7.22)
Feb 08 12:47:43 localhost.localdomain kernel: virbr0: port 1(virbr0-nic)
Feb 08 12:47:43 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP)
Feb 08 12:47:43 localhost.localdomain kernel: virbr0: port 1(virbr0-nic)
Feb 08 12:47:35 localhost.localdomain kernel: e1000: enp0s8 NIC Link is Up
Feb 08 12:47:32 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE)
Feb 08 12:47:32 localhost.localdomain kernel: ip_set: protocol 6
Feb 08 12:47:32 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP)
...

That makes digging through journal files much easier!

Exercise 17.1 walks through how to monitor the system logs in real time and how to watch as you send a message to the logging facility on your Linux system.