Chapter 19. Logs and Audit Trails

If I had to list ten advantages that Linux offers, logging would be in the top five. Logging is an essential component of any network operating system. This chapter focuses on logging tools and techniques that help you keep your finger on your system's daily pulse.

What Is Logging, Exactly?

If you're just now migrating to Linux, you may not be familiar with logging. (Most desktop-oriented operating systems offer minimal logging or sometimes, none at all.)

Briefly, logging is any procedure by which an operating system or application records events as they happen and preserves those records for later perusal.

It's difficult to say when logging first became a staple procedure in computing, but it hails from the discipline of programming. Even when you write a relatively simple program, it's useful to have diagnostic information on hand. For example:

  • Whether the program faulted and if so, when and why.

  • The program's UID and PID.

  • Who has used the program, and when did they use it?

  • Does the program perform tasks in the way you want it to?

You may also have other reasons to incorporate logging into your programs. Suppose you're hired to write a CGI program that creates and manages a contact database. It's not a bad idea to track changes (and deletions in particular), as in the following:

open(DELETELOG, ">>deletelog");
    $date='/bin/date';
$linenumber = $.;
$linerecord = $_;
@fields=split('!:!', $linerecord);
    select(DELETELOG);
    print "On or about $date, you deleted line number $linenumber:";
        print "$fields[0] : $fields[1] : $fields[2]
";
    close(DELETELOG);

This way, if your client inadvertently deletes an irreplaceable record, he or she can later recover it from the log.

In a security context, logging serves a different purpose: to preserve a record of an attacker's evil deeds. Logs provide the only real evidence that a crime has occurred.

Logging in Linux

Logging in Linux is pervasive and occurs at the system, application, and even protocol levels. And, though there are exceptions (third party software, for example), most Linux services output log information to standard or even shared log files.

Most of these reside in /var/log:

[root@linux6 log]# ls -lF
total 19
drwxr-xr-x   3 root     root         1024 Jul  1 11:35 httpd/
-rw-r--r--   1 root     root         3232 Jul  1 12:12 lastlog
-rw-r--r--   1 root     root          185 Jul  1 12:02 mail
drwxr-xr-x   2 majordom majordom     1024 Aug 19  1998 majordomo/
-rw-------   1 root     root         3132 Jul  1 13:02 messages
-rw-r--r--   1 root     root            0 Jul  1 12:02 news.all
drwxr-xr-x   3 news     news         1024 Jul  1 11:40 news.d/
-rw-r--r--   1 root     root            0 Jul  1 12:02 nwamd.log
-rw-r--r--   1 root     root            0 Jul  1 12:02 nwclientd.log
drwxr-xr-x   2 postgres database     1024 Jul  1 11:57 postgres.d/
drwxr-xr-x   2 root     root         1024 Oct  4  1996 promondia/
drwxr-xr-x   2 root     root         1024 Aug 19  1998 samba.d/
-rw-------   1 root     root         1055 Jul  1 12:14 secure
-rw-r--r--   1 root     root            0 Jul  1 12:02 spooler
drwxrwxr-x   2 uucp     uucp         1024 Aug 19  1998 uucp/
-rw-r--r--   1 root     root         1232 Jul  1 12:15 wtmp
-rw-------   1 root     root           91 Jul  1 12:15 xferlog
[root@linux6 log]#

Let's look at these files and the utilities that generate them.

lastlog

lastlog tracks user logins. As explained in the lastlog manual page:

  • lastlog formats and prints the contents of the last login log, /var/log/lastlog. The login-name, port, and last login time will be printed. The default (no flags) causes lastlog entries to be printed in UID order.

By default, lastlog reports on all users listed in /etc/passwd, as shown in the following example:

[root@linux6 log]# lastlog
Username         Port     From       Latest
root             tty1                Thu Jul  1 12:12:12 1999
bin                                  **Never logged in**
daemon                               **Never logged in**
adm                                  **Never logged in**
lp                                   **Never logged in**
sync                                 **Never logged in**
shutdown                             **Never logged in**
halt                                 **Never logged in**
mail                                 **Never logged in**
news                                 **Never logged in**
uucp                                 **Never logged in**
operator                             **Never logged in**
games                                **Never logged in**
gopher                               **Never logged in**
ftp                                  **Never logged in**
man                                  **Never logged in**
majordom                             **Never logged in**
postgres                             **Never logged in**
hapless          ttyp0    172.16.0.1 Thu Jul  1 12:11:40 1999
[root@linux6 log]#

You can single out a specific user by using the -u command-line option, as shown in the following (syntax is lastlog -u user ):

[root@linux6 log]# lastlog -u root
Username         Port     From    Latest
root             tty1             Thu Jul  1 12:12:12 1999
[root@linux6 log]#

lastlog pulls its information from /var/log/lastlog. (If you examine /var/log/lastlog, you'll find that it's a data file, so don't try to concatenate it from a shell prompt.)

Note

Unlike some other logging systems, lastlog entries are only temporary. Therefore, you should take steps to preserve lastlog data on a daily basis.

last

last reports the last login of users. As explained in the last manual page:

  • last searches back through the file /var/log/wtmp (or the file designated by the -f flag) to show a list of all users logged in (and out) since that file was created.

Data reported includes the following:

  • Users

  • The terminal (or service) they used to log in

  • Their IP address (or hostname) during the specified session

  • The date and time

  • The duration of their sessions

The following is an example last query:

[root@linux6 log]# last
hapless  ftp          172.16.0.1  Thu Jul  1 12:15 - 12:15  (00:00)
hapless  ftp          172.16.0.1  Thu Jul  1 12:14 - 12:14  (00:00)
root     tty1                     Thu Jul  1 12:12   still logged in
hapless  ttyp0        172.16.0.1  Thu Jul  1 12:11 - 12:14  (00:02)
hapless  tty1                     Thu Jul  1 12:10 - 12:10  (00:00)
reboot   system boot              Thu Jul  1 12:10
root     tty1                     Thu Jul  1 12:03 - crash  (00:07)
wtmp begins Thu Jul  1 12:03:07 1999
[root@linux6 log]#

I pulled this last report from a freshly installed system, so the last output is meager. When your machine has been running a while, last reports can be several pages long. In these cases, you'll probably want to pull last reports on particular users (as opposed to all users). To do so, issue the last command plus your desired user, as shown in the following:

last root
[root@linux6 log]# last root
root     tty1                 Thu Jul  1 12:12   still logged in
root     tty1                 Thu Jul  1 12:03 - crash  (00:07)
wtmp begins Thu Jul  1 12:03:07 1999
[root@linux6 log]#

last supports several command-line options that control the output format and length. These are summarized in Table 19.1.

Table 19.1.  last Command-Line Options

Option Function
-a Use the -a option to specify that last should display the hostname information in the last field.
-d Use the -d option to specify that last should display not only the target's hostname but also its IP address.
-n [ number ] Use the -noption to specify how many lines lastshould output.
-num [ number ] Use the -numoption to specify how many lines lastshould output.
-R Use the -R option to specify that last should omit the hostname field from the output.
-x Use the -x option to specify that last should display system reboots and run level changes.

Don't underestimate the value of last reports. last can help you investigate intrusions. Here's an example: I remember one case where an authorized local user had apparently logged in to his ISP and used a shell machine there to attack an ISP in Canada. I had a problem with that scenario, though, because when I examined the user's last report, I saw that in the two years during which he had an account, he had never used Telnet to connect to shell (or any other machine within his ISP's domain). That simply didn't jibe. (It turned out that another local user—a Linux user, incidentally—had commandeered the account).

Note

You can also use last to detect other bogus activity. A new co-worker once asked me why I ran automated scripts that pulled w, who, and last reports every few minutes on several machines and redirected them to a separate log server. He figured it out pretty quickly when we were later faced with a spoofing attack. The attacker had a legitimate account on one shell box and used it to spoof another local machine by impersonating a user from still another. While the attacker did his homework, he apparently didn't do it well enough. For, while the two affected machines both reported supposedly legitimate addresses, my w, who, and last queries caught his real IP and processes on another. When matched against all the other system logs on the affected machines, it became clear who the culprit was. This was so, even though the attacker had altered some logs.

Circumventing lastlog, last, and wtmp

Attackers are well aware that /var/log/lastlog and /var/log/wtmp can give them away. Hence, every cracker keeps an up-to-date cache of sweepers and cleaners (programs that circumvent default logging systems).

The following are a few with which you can experiment:

Log cleaners are concrete examples of how legitimate programming techniques can be used to circumvent system security. (In other words, much like scanners, log access utilities are tools that can be used equally effectively by well-intentioned and not-so-well intentioned users).

If you want to learn more about log cleaners (or perhaps write your own), examine the source of the utilities mentioned previously. Most log cleaners require either

  • utmp.h—A header library that you can use to catch run levels, boot time events, init processes, login processes, user processes, the type of login, originating hostname, and so on. (See the utmp or wtmp man pages for more information.)

  • unistd.h—A header library that you can use to catch system messages about error conditions, warning conditions, debugging information, and so on.

The attacker writes code that opens utmp, and, using something like strncpy (string copy), replaces the current line with user-specified data (or simply uses strncpy to replace the current line with whitespace or nothing).

To hedge your bets against crackers tampering with your log entries, you should use at least one third-party or proprietary logging tool. This approach offers two powerful advantages. First, few crackers will know (or bother to verify) that you are running special logging tools. Second, such tools will derive their logs independently, without using operating system logs as a starting index. If you later compare this information to default system logs and find a discrepancy, you'll instantly know that an intrusion has taken place.

Also consider insulating your logs from tampering. For example, write them to write-once media or a remote log server. This is a little more expensive, but it does guarantee that you'll have one set of reliable logs and reliability is everything.

xferlog

xferlog records FTP file transfers. As explained in the xferlog manual page:

  • The xferlog file contains logging information from the FTP server daemon, ftpd(8). This file usually is found in /usr/adm, but can be located anywhere by using a option to ftpd(8). Each server entry is composed of a single line…

Output fields include the following:

  • The current time

  • The duration of the file transfer

  • The remote host (hostname/IP)

  • The size of the file transferred

  • The filename

  • The transfer type (binary/ASCII)

  • Any special action taken (if the file was compressed or tarred)

  • The direction of the transfer (incoming, outgoing)

  • The access mode (anonymous, guest, or authenticated user)

  • The username

  • The service

  • Authentication method

  • The authenticated user ID

The following shows some sample output:

 [root@linux6 log]# more xferlog
Thu Jul  1 12:15:14 1999 1 172.16.0.1 694 /home/hapless/index.html a _ i r hapless ftp 0 *
Thu Jul  1 13:20:17 1999 1 172.16.0.1 694 /home/hapless/index.html a _ o r hapless ftp 0 *
[root@linux6 log]#

These entries show that user hapless (from 172,16.0.1) conducted two transfers—one incoming (i), one outgoing (o)—of index.html as an authenticated user (r) at the specified times.

httpd Logs

httpd stores its logs in /var/log/httpd/apache in two files:

  • access_log—access_log stores general access information: who contacted the server, when, how, and what actions they took.

  • error_log—error_log stores access (and other) errors.

Let's look at the format of these files now.

access_log: The HTTP Access Log File

access_log stores the following values:

  • The visitor's IP address

  • The event's time and date

  • The command or request

  • The status code

The following shows some sample output:

[root@linux6 apache]# more access_log
172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET / HTTP/1.0" 200 1879
172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET / HTTP/1.0" 200 1879
172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET /mmback.gif HTTP/1.0" 404 204
172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET /mmback.gif HTTP/1.0" 404 204
172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET /head.gif HTTP/1.0" 200 17446
172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET /head.gif HTTP/1.0" 200 17446
172.16.0.1 - - [01/Jul/1999:13:09:57 -0700] "GET /mmback.gif HTTP/1.0" 404 204
172.16.0.1 - - [01/Jul/1999:13:09:57 -0700] "GET /mmback.gif HTTP/1.0" 404 204
172.16.0.1 - - [01/Jul/1999:13:10:04 -0700] "POST / HTTP/1.0" 405 228
172.16.0.1 - - [01/Jul/1999:13:10:04 -0700] "POST / HTTP/1.0" 405 228
172.16.0.1 - - [01/Jul/1999:13:10:06 -0700] "GET /mmback.gif HTTP/1.0" 404 204
172.16.0.1 - - [01/Jul/1999:13:10:06 -0700] "GET /mmback.gif HTTP/1.0" 404 204

Table 19.2 provides a quick reference for HTTP status codes.

Table 19.2.  httpd Status Codes

Code Description
200 The 200 code indicates that everything went well; the transfer was successful and occurred without error.
201 The 201 code indicates that a POST command was issued and satisfied successfully without event.
202 The 202 code indicates that the client's command was accepting by the server for processing.
203 The 203code indicates that the server could only partially satisfy the client's request.
204 The 204 code indicates that the client's request was processed but that the server couldn't return any data.
300 The 300code indicates that the client requested data that has recently been moved.
301 The 301 code indicates that the server found the client's requested data at an alternate, temporarily redirected URL.
302 The 302 code indicates that the server suggested an alternate location for the client's requested data.
303 The 303 code indicates that there was a problem because the server could not modify the requested data.
400 The 400 code indicates that the client made a malformed request that could therefore not be processed.
401 The 401 code indicates that the client tried to access data that it is not authorized to have.
402 The 402code indicates that a payment scheme has been negotiated.
403 The 403code indicates that access is forbidden altogether.
404 The 404code (the most often-seen code) indicates that the document was not found.
500 The 500 code indicates that an internal server error occurred from which the server could not recover. (This is a common error when a client calls a flawed CGI script).
501 The 501 code indicates that the client requested an action that the server cannot perform (or does not support).
502 The 502code indicates that the server is overloaded.
503 The 503 code indicates that httpd was waiting for another gateway service to return data but that the external service hung or died.

error_log: The Error Message Log

error_log stores the following fields by default:

  • The date and time

  • The type of report (error)

  • The reason for the error

  • The service

  • The action taken (sometimes)

The following shows some sample output:

[root@linux6 apache]# more error_log
[Thu Jul  1 12:03:01 1999] [notice] Apache/1.3.1 (Unix) configured -- resuming normal operations
[Thu Jul  1 13:09:46 1999] [error] File does not exist: /home/httpd/html/mmback.gif
[Thu Jul  1 13:09:57 1999] [error] File does not exist: /home/httpd/html/mmback.gif
[Thu Jul  1 13:10:06 1999] [error] File does not exist: /home/httpd/html/mmback.gif
[Thu Jul  1 13:33:30 1999] [notice] httpd: caught SIGTERM, shutting down
[Thu Jul  1 13:35:04 1999] [notice] Apache/1.3.1 (Unix) configured -- resuming normal operations
[Thu Jul  1 13:51:39 1999] [notice] httpd: caught SIGTERM, shutting down
[Thu Jul  1 21:23:28 1999] [notice] Apache/1.3.1 (Unix) configured -- resuming normal operations

Customizing httpd Logs

Apache allows you to customize your logs with the LogFormat directive. The following is the default:

LogFormat "%h %l %u %t "%r" %s %b"

This indicates that by default, Apache logs

  • The remote host address

  • The remote logname (unreliable and available only if the client box is running ident)

  • The remote user (unreliable also)

  • The time (standard log format, (Thu Jul 1 13:10:06 1999), for example)

  • The client's first request

  • The status

  • The bytes sent

Table 19.3 summarizes LogFormat directives.

Table 19.3.  httpd LogFormat Directives

Directive Function
%{ env_variable }e The %e directive will define the specified environment variable.
%b The %b directive records the total number of bytes sent (not including headers).
%f The %fdirective records the filename requested.
%h The %hdirective records the remote host's address.
%l The %l directive records the logname (username) of the client's user (if they're running ident).
%P The %P directive records the PID of the process that satisfied the client's request.
%p The %pdirective records the port to which the server directed the response.
%r The %rdirective records the first line of the client's request.
%s The %sdirective records the status of the client's request.
%t The %tdirective records the time of the request.
%T The %Tdirective records the time taken to satisfy the client's request.
%u The %udirective records the remote user (using auth).
%U The %Udirective records the URL that the client initially requested.
%v The %vdirective records the virtual host's hostname.

System and Kernel Messages

System and kernel messages are handled by two daemons:

  • syslogd—syslogd records the type of logging that many programs use. Typical values that syslogd traps include the program name, facility type, priority, and stock log message.

  • klogd—klogd intercepts and logs kernel messages.

To see syslogd and klogd in action, you must turn to /var/log/messages.

/var/log/messages: Recording System and Kernel Messages

/var/log/messages receives message output from syslogd and klogd.

Note

If your Linux system is antiquated, you'll find messages in /var/adm.

System and kernel diagnostic messages appear in the order in which they were received:

[root@linux6 log]# more messages
Jul  1 12:02:50 linux6 syslogd 1.3-3: restart.
Jul  1 12:02:52 linux6 kernel: klogd 1.3-3, log source = /proc/kmsg started.
Jul  1 12:02:52 linux6 kernel: Loaded 4122 symbols from /boot/System.map-2.0.35.
Jul  1 12:02:52 linux6 kernel: Symbols match kernel version 2.0.35.
Jul  1 12:02:52 linux6 kernel: Loaded 95 symbols from 16 modules.
Jul  1 12:02:52 linux6 kernel: VFS: Mounted root (ext2 filesystem) readonly.
Jul  1 12:02:52 linux6 kernel: lp0 at 0x03bc, (polling)
Jul  1 12:02:52 linux6 kernel: CSLIP: code copyright 1989 Regents of the University of California
Jul  1 12:02:52 linux6 kernel: SLIP: version 0.8.4-NET3.019-NEWTTY- MODULAR (dynamic channels, max=256).
Jul  1 12:02:52 linux6 kernel: PPP: version 2.2.0 (dynamic channel allocation)
Jul  1 12:02:52 linux6 kernel: PPP Dynamic channel allocation code copyright 1995 Caldera, Inc.
Jul  1 12:02:52 linux6 kernel: PPP line discipline registered.
Jul  1 12:02:52 linux6 kernel: Swansea University Computer Society IPX 0.34 for NET3.035
Jul  1 12:02:52 linux6 kernel: IPX Portions Copyright (c) 1995 Caldera, Inc.
Jul  1 12:02:52 linux6 kernel: sysctl: ip forwarding off
Jul  1 12:02:52 linux6 amd[23101]: My ip addr is 0x100007f
Jul  1 12:02:52 linux6 amd[23102]: file server localhost type local starts up
Jul  1 12:02:53 linux6 amd[23102]: /etc/amd.localdev mounted fstype toplvl on /

In addition to standard syslog and kernel messages, you'll also find messages from network services:

Jul  1 12:10:38 linux6 syslog: LOGIN ON tty1 BY hapless
Jul  1 12:11:36 linux6 syslog: FAILED LOGIN 1 FROM 172.16.0.1 FOR haples, User not known to the underlying
  authentication module
Jul  1 12:11:36 linux6 syslog: FAILED LOGIN 1 FROM 172.16.0.1 FOR haples, User not known to the underlying
  authentication module
Jul  1 12:11:40 linux6 syslog: LOGIN ON ttyp0 BY hapless FROM 172.16.0.1
Jul  1 12:12:12 linux6 syslog: ROOT LOGIN ON tty1
Jul  1 12:14:37 linux6 ftpd[23622]: FTP LOGIN FROM 172.16.0.1 [172.16.0.1], hapless
Jul  1 12:14:41 linux6 ftpd[23622]: FTP session closed
Jul  1 12:15:07 linux6 ftpd[23625]: FTP LOGIN FROM 172.16.0.1 [172.16.0.1], hapless
Jul  1 12:15:15 linux6 ftpd[23625]: FTP session closed

syslog.conf: Customizing Your syslog

To customize syslog logging, specify your rules in syslog.conf. As explained in the syslog.conf manual page:

  • The syslog.conf file is the main configuration file for the syslogd(8) which logs system messages on *nix systems. This file specifies rules for logging. For special features see the sysklogd(8) man page.

In syslog.conf, you define rules with two fields:

  • The selector field—What to log

  • The action field—Where to log it

The Selector Field

In the Selector field, you must specify at least one of two values:

  • The message type

  • The message priority

The message type is called a facility and must be one of the following:

  • auth—auth is a security facility that tracks user authentication in various services such as FTP, login, and so on. (Essentially, the auth facility tracks any user action that requires a username and password to login or uses the target resource.)

  • authpriv—authpriv is a security facility that tracks security/authorization messages.

  • cron—cron tracks messages from the cron system. cron is a daemon that executes scheduled commands. (See the cron man page for more information.)

  • daemon—daemon tracks additional system daemon messages.

  • kern—kern tracks kernel messages.

  • lpr—lpr tracks line printer system messages.

  • mail—mail tracks mail system messages.

  • news—news tracks news system messages.

  • uucp—uucp tracks UNIX-to-UNIX Copy subsystem messages.

You can specify blanket logging using only the facility and no priority . For example, here's a rule that specifies that the system should send all kernel messages to the console:

kern.*                               /dev/console

Here, the facility is kernel and the action is to log to /dev/console. Or, if you wanted to log all kernel messages to /var/log/messages, you could establish a rule such as the following:

kern.*                    /var/log/messages

The second half of the Selector field is the Priority , which is not always necessary unless you want to refine your output. The Priority must be one of the following:

  • alert—Alerts indicate serious malfunctions that demand immediate attention.

  • crit—crit (critical) messages indicate fatal problems.

  • debug—debug messages provide debugging information on running processes.

  • emerg—emerg (emergency) messages indicate emergency conditions.

  • err—err (error) messages consist of typical STDERR.

  • info—info (informational messages) are plain old for-your-information messages from programs.

  • notice—notice messages are standard messages.

  • warning—warning messages are standard warnings (for example, the system or resource couldn't perform the requested task).

So, for example, if you wanted to log error messages from your news system, you might create a rule such as the following:

# Save news errors of level err and higher
# in a special file.
news.err                          /var/log/spooler

Here, your values are

  • Your facility = news

  • Your priority = err (error messages)

  • Your action = log these to /var/log/spooler

The Action Field

In the action field, you specify what syslog should do with the messages you've requested. As seen earlier, one possible choice is to log the messages to a particular file. Other choices include the following:

  • Named pipes

  • The terminal or console

  • A remote machine (if it's running syslogd)

  • Specified users

  • All users

For example, suppose you wanted to send your kernel messages to the remote host linux3 (running syslogd). You might create a rule such as the following:

kern.*                    @linux3

Or, perhaps you want to send all alerts to user support. You could create a rule such as the following:

*.alert                      support

The sample syslog.conf file provided with Linux offers several pre-fabricated possibilities:

 [root@linux6 conf]# more /etc/syslog.conf

syslog.cong

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                      /dev/console

# Log everything (except mail and news) of level info or higher.
# Hmm--also don't log private authentication messages here!
*.info;news,mail,authpriv,auth.none             -/var/log/messages
# Log debugging too
#*.debug;news,mail,authpriv,auth.none        -/var/log/debug

# The authpriv file has restricted access.
authpriv.*;auth.*                             /var/log/secure
# true, 'auth' in the two previous rules is deprecated,
# but nonetheless still in use…

# Log all the mail messages in one place.
mail.*                                        /var/log/mail
# As long as innd insists on blocking /var/log/news
# (instead of using /var/log/news.d) we fall back to …
news.*                                     /var/log/news.all

# Save uucp and news errors of level err and higher
# in a special file.
uucp,news.err                     /var/log/spooler

# Everybody gets emergency messages, plus log them on
# another machine.
*.emerg                            *
#*.emerg                            @loghost

If you plan to build a large Linux network, I recommend logging to both local and remote locations. This will ensure some level of redundancy. (It's always a good idea to have several versions. You never now when disaster might strike.)

Writing to syslog from Your Own Programs

Eventually, you'll write your own daemons and logging utilities. Here, I thought it would be useful to briefly address how to write syslog from within your programs.

In C, include the syslog library syslog.h. As explained on the syslog(3) manual page:

  • syslog() generates a log message, which will be distributed by syslogd(8). Priority is a combination of the facility and the level… The remaining arguments are a format, as in printf(3) and any arguments required by the format, except that the two character %m will be replaced by the error message string (strerror) corresponding to the present value of errno.

In your syslog call, include a syslog level. Table 19.4 summarizes the levels.

Table 19.4.  syslog Levels

Level Function
LOG_CRIT Logs a critical message.
LOG_DEBUG Logs a debug-level message.
LOG_EMERG Logs an emergency message.
LOG_ERR Logs an error message.
LOG_INFO Logs an informational message.
LOG_NOTICE Logs a notice message.
LOG_WARNING Logs a warning condition.

You can also include a syslog facility, if appropriate. Table 19.5 summarizes the possible facilities.

Table 19.5.  syslog Facilities

Facility Function
LOG_AUTHPRIV Specifies that the current message is type AUTH (a security, authentication, or authorization notification).
LOG_CRON Specifies a clock daemon (cron/at) message.
LOG_DAEMON Specifies a system daemon message.
LOG_KERN Specifies a kernel message.
LOG_LPR Specifies a line printer daemon message.
LOG_MAIL Specifies a mail subsystem message.
LOG_NEWS Specifies a Usenet news message.
LOG_SYSLOG Specifies an internal syslogmessage.
LOG_USER Specifies a generic user message.
LOG_UUCP Specifies a UUCP message.

To write syslog, call syslog() with at least a level and a message. For example, compile and run the following code:

#include <syslog.h>
void main(int argc,char *argv[])
{
 syslog(LOG_ALERT,"This is my alert.
");
 syslog(LOG_CRIT,"This is my critical message.
");
 syslog(LOG_DEBUG,"This is my debug-level message.
");
 syslog(LOG_EMERG,"This is my emergency message.
"));
 syslog(LOG_ERR,"This is my error.
");
 syslog(LOG_INFO,"This is my informational message.
");
 syslog(LOG_NOTICE,"This is my notice.
");
 syslog(LOG_WARNING,"This is my warning.
");
}

When you check syslog, you'll see a corresponding run of messages:

July 23 11:15:55 linux6 syslog: This is my alert message.
July 23 11:15:55 linux6 syslog: This is my critical message.
July 23 11:15:55 linux6 syslog: This is my debug-level message.
July 23 11:15:55 linux6 syslog: This is my emergency message
July 23 11:15:55 linux6 syslog: This is an error.
July 23 11:15:55 linux6 syslog: This is my information message.
July 23 11:15:55 linux6 syslog: This is my notice.
July 23 11:15:55 linux6 syslog: This is my warning.

How you integrate this functionality into your program will depend on what the program does, but typically you'd construct a separate block for sending the message:

if(some-condition)   /* If some operation fails… */
report_error();  /* Jump to report_error() to write syslog */

void report_error(const char str) {
syslog(LOG_WARN, "%s failed: %d (%m),", str, errno);
syslog(LOG_WARN, "Warning: the user has run amok..n");
exit(1); 
}

In Perl, use the Sys::Syslog module (a Perl interface to syslog), like this:

use Sys::Syslog;

if(some operation fails) {
   &report_error;
  }

sub report_error {
syslog(LOG_WARN, "Something terrible has happened.
");
exit 0;
}

Note

In both C and Perl, instances may arise where you must use openlog() and closelog(), or explicitly set the mask or socket type. Please see the syslog.h and Sys::Syslog manual pages for more details. Also, note that depending on your installation, you may not have Sys::Syslog. If not, find updated modules at CPAN, the Comprehensive Perl Archive Network, at http://www.cpan.org/ , or at http://www.perl.com.

Finally, if you prefer Java, Acme Labs offers Acme.Syslog, a ready-made syslog manipulation Java class. Find it at http://www.acme.com/java/software/Acme.Syslog.html.

Backing and Handling Logs

On a single Linux box, log files grow slowly, but in a Linux network with a dozen users or more, heavy logging can result in massive files. If you anticipate generating massive logs, you should arrange to back up or rotate them. For this, one solution is Erik Troan's logrotate.

logrotate

logrotate rotates, compresses, and mails system logs. As explained in the manual page:

  • logrotate is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.

You run logrotate as a cron job. Each time it runs, it reads options from a user-specified configuration file. The following is a typical configuration file entry:

errors [email protected]
       compress

/var/log/messages {
           rotate 5
           weekly
           postrotate
                                     /sbin/killall -HUP syslogd
           endscript
       }

The first two lines define global options. In this case, they specify that all errors should be mailed to , and that all log files should be compressed for transport:

errors [email protected]
       compress

After defining your global options, you must set rules for each log file. To do so, you construct special sections using a directive-based language.

Each section begins with the log filename (in the earlier example, it's /var/log/messages). From there, everything between the open and close brackets ({ and }) are directives . Let's take another look at the earlier example, this time commented:

/var/log/messages { # Take the log file /var/log/messages
           rotate 5 # Rotate it five times before removing it
           weekly   # Perform rotations once a week
           postrotate # Finally, after rotating the file…
                      /sbin/killall -HUP syslogd # Execute this command
           endscript # …and close up this section
       }

You can control many aspects of the log file rotation using directives (either globally or on a per-section basis). Table 19.6 summarizes these directives and what they do.

Table 19.6.  logrotate Configuration File Directives

Directive Function
compress The compress directive specifies that logrotate should compress old log files using gzip.
create mode owner group When logrotate rotates a given file, it creates a new one in its place. Use the create option to specify the new file's mode (a la chmod()), the file's owner, and the file's group.
daily Use the daily directive to specify that logrotate should rotate the specified log file on a daily basis.
endscript Use the endscript directive to indicate that your script (for the current logfile section) has ended.
errors [ email_address ] Use the errors directive to specify an email address to which logrotate will send all error messages.
ifempty Use the ifempty directive to specify that logrotate should rotate the specified log file, even if that file is empty.
mail [ email_address ] Use the mail directive to specify the address to which logrotate will mail final files (those that have been rotated to the end of their cycle).
monthly Use the monthly directive to specify that logrotate should rotate the specified file monthly.
nocompress Use the nocompress directive to specify that logrotate should not gzip old log files.
nocreate Use the nocreate directive to specify that logrotate should not create a new log file after it rotates an old one.
nomail Use the nomail directive to specify that logrotate needn't mail the old log files anywhere.
noolddir Use the noolddir directive to specify that logrotate should rotate log files in the same directory where they reside.
notifempty Use the notifempty directive to specify that logrotate should not rotate empty log files.
olddir [ directory ] Use the olddir directive to specify that logrotate should move log files into directory during rotation.
rotate [ n ] Use the rotate directive to specify that logrotate should rotate the specified log file n times before mailing it out (or removing it).
size [size M/K] Use the size directive to specify how large a log file can grow before logrotate should rotate it. You can express this value in kilobytes or megabytes.
weekly Use the weekly directive to specify that logrotate should rotate the specified log file weekly.

Note

Note that logrotate runs as root and does execute shell scripts. Also, some versions of Apache (when used with caching) will crash if you use logrotate to rotate the logs.

Other Interesting Logging and Audit Tools

Finally, this next section covers several interesting and useful logging and audit tools that don't ship with Linux (see Table 19.7).

Table 19.7. Tools to Enhance Your Logging Security

Tool Description and Location
Ippl ippl is a multi-threaded tool that logs incoming IP packets. You can establish rules for which packet types you'd like to filter. Location: http://www.via.ecp.fr/~hugo/ippl/.
Log Scanner Log Scanner is a Perl-based tool that works in concert with TCP wrappers. It enables you to set up a log parser that will contact you (or others) when predefined anomalies are discovered in a log file. Location: http://logscanner.tradeservices.com/.
Logcheck Logcheck is one component of the Abacus Project. Logcheck processes logs generated by the Abacus Project tools, system daemons, TCP Wrapper, logdaemon, and the TIS Firewall Toolkit. Location: http://www.psionic.com/abacus/logcheck/.
LogWatch LogWatch analyzes your logs for a user-specified time period and generates customizable reports. Location: http://www.kaybee.org/~kirk/html/linux.html.
netlog netlog is a collection of network monitoring and logging utilities (tcplogger, udplogger, netwatch, and extract). netlog can log all TCP connections (and UDP sessions) on a subnet and provides real-time monitoring and reporting. Location: http://net.tamu.edu/ftp/security/TAMU/netlog.README.
PIKT PIKT is the Problem Informant/Killer Tool. PIKT monitors multiple workstations for problems and, if appropriate, automatically fixes those problems. Example problems include disk failures, log failures, queue overflows, erroneous or suspicious permission changes. Location: http://pikt.uchicago.edu/pikt/.
Plugshot's TSTTST is the Tagged Shell Toolkit, which enables you to log and audit user shell commands. Check it out at http://www.plugslot.com/.
Secure Syslog Secure Syslog is a new cryptographically secure system-logging tool. Designed to replace the syslog daemon, Secure Syslog implements a cryptographic protocol called PEO-1 that allows the remote auditing of system logs. Auditing remains possible, even if an intruder gains superuser privileges in the system. Location: http://www.core-sdi.com/english/index.html.

Also, there are several useful utilities that border on being both intrusion detection and logging analysis systems, including the following:

  • Swatch

  • Watcher

  • NOCOL

  • Pinglogger

  • LogSurfer

  • Netlog

  • Analog

SWATCH (The System Watcher)

The authors wrote SWATCH to supplement logging capabilities of out-of-the-box UNIX systems. SWATCH, consequently, has logging capabilities that far exceed your run-of-the-mill syslog. SWATCH provides real-time monitoring, logging, and reporting. And, because SWATCH is written in Perl, it's both portable and extensible.

SWATCH has several unique features, including the following:

  • A "backfinger" utility that attempts to grab finger information from attacking host

  • Support for instant paging (so you can receive up-to-the-minute reports)

  • Conditional execution of commands (if this condition is found in a log file, do this)

Lastly, SWATCH relies on local configuration files. Conveniently, multiple configuration files can exist on the same machine. Therefore, while originally intended only for system administrators, any local user with adequate privileges can use SWATCH.

Learn more about SWATCH in Chapter 20, "Intrusion Detection."

Watcher

Ingham developed Watcher while at the University of New Mexico Computing Center. He explains that, at the time, the Computing Center was expanding and the logging process they were using was no longer adequate. Therefore, Ingham was looking for a way to automate scanning of logs. Watcher was the result of his labors.

Watcher analyzes various logs and processes, looking for radically abnormal activity. (The author sufficiently fine-tuned this process so Watcher can interpret the widely variable output of commands like ps without setting off alarms.)

Watcher runs on UNIX systems and requires a C compiler.

NOCOL/NetConsole v4.0

NOCOL/NetConsole v4.0 is a suite of standalone applications that performs a wide variety of monitoring tasks. This suite offers a Curses interface, which is great for running on a wide range of terminals (it does not require X to work). It is extensible, has support for a Perl interface, and operates on networks running AppleTalk and Novell.

NOCOL/NetConsole v.4.0 is available online at ftp://ftp.navya.com/pub/vikas/nocol.tar.gz.

PingLogger

PingLogger logs ICMP packets to an outfile. Using this utility, you can reliable determine who is ping flooding you. The utility was originally written and tested on Linux (it requires a C compiler and IP header files) but may work on other UNIX systems.

LogSurfer

LogSurfer is a comprehensive log analysis tool. The program examines plain text log files and, based on what it finds (and the rules you provide), it can perform various actions. These might include creating an alert, executing an external program, or even taking portions of the log data and feeding that to external commands or processes. LogSurfer requires C.

Netlog

Netlog, developed at Texas A&M University, can log all TCP and UDP traffic. This tool also supports logging of ICMP messages (though the developers report that performing this logging activity soaks up a great deal of storage). To use this product, you must have a C compiler.

Netlog is available online at ftp://coast.cs.purdue.edu/pub/tools/unix/TAMU/.

Analog

Analog is a truly cross-platform log file analyzer. In addition to Linux, Analog currently runs on the following operating systems:

  • Macintosh

  • OS/2

  • Windows 95/NT

  • Vax/VMS

  • RiscOS

  • BeOS

  • BS2000/OSD

Analog also has built-in support for a wide variety of languages, including English, Portuguese, French, German, Swedish, Czech, Slovak, Slovene, Romanian, and Hungarian.

And, as if that weren't enough, Analog also does reverse DNS lookups (slowly), has a built-in scripting language (similar to the shell languages), and has at least minimal support for AppleScript.

Finally, Analog supports most of the well-known Web server log formats, including Apache, NCSA, WebStar, IIS, W3 Extended, Netscape, and Netpresenz. For these reasons, Analog is a good tool to have around (especially in heterogeneous networks).

Summary

Never underestimate the importance of keeping detailed logs. Not only are logs essential when you're investigating a network intrusion, they're also a requisite for bringing charges against an attacker. Now that you know a bit about logs, the next step is to learn how you can use them to detect intrusions. That's what Chapter 20 is all about.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.138.106.233