System Logs
The system log used by Unix-like operating systems has become the industry standard for logging, but that’s not necessarily a good thing, because the log mechanism can be cantankerous. Once you properly configure log collection and rotation, however, OpenBSD’s logging system mostly manages itself.
OpenBSD uses the standard logging system for Unix-like (and many embedded) systems, syslog(3). The syslog protocol marks messages with a facility and a priority, and hands those messages to a daemon.
Any program can write to the local syslogd(8) server, but the key in log management is deciding how those messages are sorted and stored. OpenBSD’s syslogd can sort messages based on facility, priority, and source program.
Facilities
A facility indicates the source of a message. In most cases, each program that needs a separate log file uses a different facility. Many programs or protocols, such as FTP, have facilities dedicated to them. The syslog protocol also has a variety of generic facilities that you can use as you wish.
Table 15-1 lists the standard facilities and provides some notes on their usage.
|
Facility |
Usage |
|
|
Public information about authentication, such as when someone logged on or when someone uses |
|
|
Private information about user authentication, normally accessible only to privileged users. |
|
|
Messages from the system scheduler |
|
|
A catchall for processes that neither need nor require a dedicated facility. |
|
|
Messages from FTP and Trivial File Transfer Protocol (TFTP) servers. |
|
|
Kernel-generated messages. |
|
|
These facilities are provided for the sysadmin. Many programs let sysadmins configure their facility. Use these eight facilities for such programs. |
|
|
Messages from the printing system. |
|
|
Messages from mail servers. |
|
|
This special facility writes a message every 20 minutes. |
|
|
Messages from Usenet news servers. |
|
|
Messages from the syslog server itself. |
|
|
The catchall message facility. If a userland program doesn’t specify a logging facility, the messages wind up here. |
|
|
Messages from the Unix-to-Unix Copy Protocol (UUCP) servers. You will probably never encounter this pre-Internet email protocol. |
While most programs have sensible defaults, it’s your job as the system administrator to manage which programs log to which facilities. If possible, use the local facilities for your server-specific daemons. While it’s entirely possible to use facilities for purposes other than originally intended, try not to reassign the uucp facility to some other daemon unless you really have no other option.
Priority
A log message’s priority represents its importance. Programs usually send their logging data to syslogd, but syslogd decides what to retain and what to discard. You get to decide how much detail you want in your logs. Use the following nine syslog levels to decide what to record and what to discard (in order from most to least important):
-
emerg. System emergency. This message appears on every active terminal. The computer might be crashing, or it may have some other error that requires immediate attention. -
alert. An emergency. The system can continue to function, but attend to this error very soon. -
critical. Critical problems. These indicate serious errors, such as hard-drive failures. -
err. Errors. These are in regard to problems that require attention but won’t destroy your system. -
warning. Miscellaneous warnings. These could be attended to, but will not prevent the process that generated them from running normally. -
notice. Important information, such as daemon startup and shutdown notifications. -
info. Basic information. This usually includes transactional data, such as individual messages in a mail server or individual queries to a web server. -
debug. Trivia. This level is usually of interest only to programmers, but occasionally useful to sysadmins trying to figure out why a program is behaving in a certain way. Debugging logs can contain anything, including information that violates user privacy, such as plaintext passwords. -
none. Don’t log anything from this facility here. This is most commonly used to exclude information from log files, as discussed shortly.
By combining levels with priority, you can sort log messages into individual files or other targets.
Sorting Messages via syslogd(8)
syslogd compares received messages to entries in /etc/syslog.conf. This file has two columns: the first (the selector) describes a type of log message, and the second (the action) tells syslogd what to do when a message matches the description. Neither column can have whitespace; whitespace can appear only between the columns. For example, here’s a line from the default syslog.conf:
daemon.info /var/log/daemon
Any log message that has a facility of daemon and a priority of info or higher is appended to the file /var/log/daemon. Of course, if all logs were so easily managed, this would be a short section.
syslogd compares all log messages to all syslog.conf entries. If a log message matches multiple selectors, it is sent to all matching destinations.
Wildcards
You can use wildcards in either the facility or priority. For example, this line logs every message from the mail facility:
mail.* /var/log/maillog
To capture messages of a given priority or higher from all facilities, use an asterisk (*). Here’s how to send all priority err and higher messages to the console:
*.err /dev/console
You can also use a double-wildcard to send all log messages to one place.
*.* /var/log/all.log
Logging everything to one location isn’t terribly useful or wise. You should not send authpriv debugging to a world-readable file.
Excluding Information
Use the none level to exclude information from a log. For example, the following line excludes private authentication information from an otherwise all-inclusive log.
*.*;authpriv.none /var/log/most.log
The semicolon (;) allows you to combine selection criteria on a single line.
Combining Facilities
You can combine multiple facilities in a single entry by using commas. Here’s how to capture all messages of info priority or higher from several facilities:
auth,daemon,syslog,user.info @loghost
Any log message from the auth, daemon, syslog, and user facilities, and of priority info or higher, is sent across the network to the host loghost.
Marking Time
While all log messages have a timestamp, you might want a marker in a log file to indicate that time has passed. The special facility mark creates a message every 20 minutes, letting you add an extra timestamp to a file. Here’s how to add a timestamp to the mail log every 20 minutes:
mail.info;mark.info /var/log/maillog
Local Facilities
The eight facilities local0 through local7 are for your use. Many programs can be configured to use a specific facility, so you can aim them at a particular file. I’ve configured a daemon to use the facility local7. Here, I send messages from that facility to a file:
local7.* /var/log/postgres.log
Some programs have a hard-coded preference for a specific facility. For example, the flow-tools package (see my book Network Flow Analysis, No Starch Press, 2010) has facility local6 hard-wired into the code. Don’t be shocked when you see something like this. Fortunately, OpenBSD’s syslogd can filter based on program name, so you can easily filter your logs despite this sort of daftness.
Selecting by Program Name
If you’re out of facilities, you can use the name of the program generating the syslog messages as a selector. Using a program name requires two lines: The first contains the program name with a leading exclamation mark, and the second sets up logging. OpenBSD offers the following example of sudo(8) logging:
!sudo *.* /var/log/sudo
All log messages from sudo go to the specified log file.
You can also select by program name and stop all subsequent selections of matching messages by using two exclamation points (!!) before the program name. This example sends all messages from sudo to /var/log/sudo, but prevents sudo messages from going to any other log.
!!sudo *.* /var/log/sudo !* …
The !* after the end of the sudo entry is a way to say “all programs”—in other words, don’t sort by program name anymore. You need this only if you use the double-exclamation-point “stop processing matching messages here” syntax.
Log Actions
Now that you know how to sort your log messages into different buckets, let’s see how to take different actions with those messages. Messages can be written to a file, piped to a program, sent to another host, or written to users.
Logging to Files
Most of our examples so far send log messages to a file, giving the full path to the file as the action. Here’s how to send all of the messages from facility local6 to a log file:
local6.* /var/log/flowtools
You can also send the messages to a device by giving the full path to the device node, but this will make sense to very few devices, such as the console. This is because writing the log message to the disk device /dev/wd0d will not store the message on disk.
Logging to a Program
To send selected logs to a program, use a pipe (|) and the full path to the program, like this:
*.* |/usr/local/bin/logsurfer
The logging system should start the destination program, and then feed log messages into the program’s standard input.
Notifying Users
You can also direct log messages to logged-in users by listing multiple users in a comma-separated list. For example, to send a message to all users, use an asterisk.
*.emerg * *.info lasnyder
This example will notify all logged-in users of real emergencies, but deeply annoy lasnyder.[41]
Logging to a Remote Host
I usually have a logging host that collects log messages from everywhere—not only from my OpenBSD boxes, but from all my other Unix-like systems, as well as routers, switches, and anything else that speaks syslog. This reduces my maintenance needs and conserves disk space. And, since, every log message includes a hostname, I can easily sort them out later.
To send messages to another host, use the @ symbol.
*.info @loghost.blackhelicopters.org
This dumps everything of priority info and above to my logging host.
Your logging host must accept syslog messages from the network. If your host is an OpenBSD machine, run syslogd with the -u flag. And be sure to protect your log host with a packet filter, so random hosts can’t write logs to it and fill up your disks.
Customizing syslogd
OpenBSD runs syslogd by default, and you can customize how syslogd behaves. Common customizations include adding more log sockets and listening to the network.
Adding Extra Log Sockets
Programs write log messages to the socket /dev/log, but software inside a chroot won’t be able to access that device. To have a program that’s locked inside a chroot send messages to syslogd, you must put an additional log socket at /dev/log inside the chroot.
For example, since the integrated BIND DNS server is chrooted into /var/named, the DNS server expects to find the log socket at /dev/log, which means that the new log socket should be at /var/named/dev/log. To create this log socket, use syslogd’s -a option, and give the full path to the log socket in /etc/rc.conf.local.
syslogd_flags="-a /var/named/dev/log"
You can use about 20 additional logging sockets.
Listening to the Network
If you want your OpenBSD box to act as a log host, accepting logs from remote hosts, use the -u flag.
syslogd_flags="-u"
Because the syslog protocol has no access control, anyone with access to port 514/UDP on the log host can write to your log files.
Note
Filling a host’s logs with junk to fill the hard disk is an old attack. Use OpenBSD’s packet filtering system (discussed in Chapter 21 and Chapter 22) to protect your logging host.
Syslog and Embedded Systems
OpenBSD supports writing log messages to an in-memory buffer, which allows logging on systems that have no writable disk, such as diskless systems and embedded routers and firewalls. syslogd retains these logs in a memory buffer, and clients can connect to syslogd through a reporting socket and read the logs. As you would expect, logs in memory disappear when syslogd is shut down.
To use syslogd for reporting, first provide a reporting socket with the -s option and give it a full path to a reporting socket. Here’s an rc.conf.local entry for a reporting socket in /var/run/syslog:
syslogd_flags="-s /var/run/syslog"
To log to the buffer, make a syslog.conf action. Specify logging to a buffer with a colon (:), the number of kilobytes to give the buffer, another colon, and the name of the memory buffer. (The maximum buffer size is 256KB.)
For example, here we capture all log messages of err priority or higher and write them to the 128KB memory buffer called errors:
*.err :128:errors
Use syslogc(8) to read a memory buffer, and use the -s option to tell syslogc where to find syslogd’s reporting socket, and provide the name of the log buffer. Here’s how to read the reporting socket /var/run/syslog and read the errors buffer:
$ syslogc -s /var/run/syslog errors
If you’ve forgotten the name of the buffer you want to read, ask syslogc to query the list of available memory logs with -q. Be sure to provide the reporting socket.