If I had to list ten advantages that Linux offers, logging would be in the top five. Logging is an essential component of any network operating system. This chapter focuses on logging tools and techniques that help you keep your finger on your system's daily pulse.
If you're just now migrating to Linux, you may not be familiar with logging. (Most desktop-oriented operating systems offer minimal logging or sometimes, none at all.)
Briefly, logging is any procedure by which an operating system or application records events as they happen and preserves those records for later perusal.
It's difficult to say when logging first became a staple procedure in computing, but it hails from the discipline of programming. Even when you write a relatively simple program, it's useful to have diagnostic information on hand. For example:
Whether the program faulted and if so, when and why.
The program's UID and PID.
Who has used the program, and when did they use it?
Does the program perform tasks in the way you want it to?
You may also have other reasons to incorporate logging into your programs. Suppose you're hired to write a CGI program that creates and manages a contact database. It's not a bad idea to track changes (and deletions in particular), as in the following:
open(DELETELOG, ">>deletelog"); $date='/bin/date'; $linenumber = $.; $linerecord = $_; @fields=split('!:!', $linerecord); select(DELETELOG); print "On or about $date, you deleted line number $linenumber:"; print "$fields[0] : $fields[1] : $fields[2] "; close(DELETELOG);
This way, if your client inadvertently deletes an irreplaceable record, he or she can later recover it from the log.
In a security context, logging serves a different purpose: to preserve a record of an attacker's evil deeds. Logs provide the only real evidence that a crime has occurred.
Logging in Linux is pervasive and occurs at the system, application, and even protocol levels. And, though there are exceptions (third party software, for example), most Linux services output log information to standard or even shared log files.
Most of these reside
in /var/log
:
[root@linux6 log]# ls -lF total 19 drwxr-xr-x 3 root root 1024 Jul 1 11:35 httpd/ -rw-r--r-- 1 root root 3232 Jul 1 12:12 lastlog -rw-r--r-- 1 root root 185 Jul 1 12:02 mail drwxr-xr-x 2 majordom majordom 1024 Aug 19 1998 majordomo/ -rw------- 1 root root 3132 Jul 1 13:02 messages -rw-r--r-- 1 root root 0 Jul 1 12:02 news.all drwxr-xr-x 3 news news 1024 Jul 1 11:40 news.d/ -rw-r--r-- 1 root root 0 Jul 1 12:02 nwamd.log -rw-r--r-- 1 root root 0 Jul 1 12:02 nwclientd.log drwxr-xr-x 2 postgres database 1024 Jul 1 11:57 postgres.d/ drwxr-xr-x 2 root root 1024 Oct 4 1996 promondia/ drwxr-xr-x 2 root root 1024 Aug 19 1998 samba.d/ -rw------- 1 root root 1055 Jul 1 12:14 secure -rw-r--r-- 1 root root 0 Jul 1 12:02 spooler drwxrwxr-x 2 uucp uucp 1024 Aug 19 1998 uucp/ -rw-r--r-- 1 root root 1232 Jul 1 12:15 wtmp -rw------- 1 root root 91 Jul 1 12:15 xferlog [root@linux6 log]#
Let's look at these files and the utilities that generate them.
lastlog
tracks
user logins. As explained in the lastlog
manual page:
lastlog
formats and prints the contents of the last login log, /var/log/lastlog
. The login-name, port, and last login time will be printed. The default (no flags) causes lastlog
entries to be printed in UID order.
By default, lastlog
reports on all users listed in /etc/passwd
, as shown in the following
example:
[root@linux6 log]# lastlog Username Port From Latest root tty1 Thu Jul 1 12:12:12 1999 bin **Never logged in** daemon **Never logged in** adm **Never logged in** lp **Never logged in** sync **Never logged in** shutdown **Never logged in** halt **Never logged in** mail **Never logged in** news **Never logged in** uucp **Never logged in** operator **Never logged in** games **Never logged in** gopher **Never logged in** ftp **Never logged in** man **Never logged in** majordom **Never logged in** postgres **Never logged in** hapless ttyp0 172.16.0.1 Thu Jul 1 12:11:40 1999 [root@linux6 log]#
You can single out a specific user by using the -u
command-line option, as shown in
the following (syntax is lastlog -u
user
):
[root@linux6 log]# lastlog -u root Username Port From Latest root tty1 Thu Jul 1 12:12:12 1999 [root@linux6 log]#
lastlog
pulls its information from /var/log/lastlog
. (If you examine /var/log/lastlog
, you'll find that it's a data file, so don't try to concatenate it from a
shell prompt.)
last
reports the
last login of users. As explained in the last
manual page:
last
searches back through the file /var/log/wtmp
(or the file designated by the -f
flag) to show a list of all users logged in (and out) since that file was created.
Data reported includes the following:
Users
The terminal (or service) they used to log in
Their IP address (or hostname) during the specified session
The date and time
The duration of their sessions
The following is an
example last
query:
[root@linux6 log]# last hapless ftp 172.16.0.1 Thu Jul 1 12:15 - 12:15 (00:00) hapless ftp 172.16.0.1 Thu Jul 1 12:14 - 12:14 (00:00) root tty1 Thu Jul 1 12:12 still logged in hapless ttyp0 172.16.0.1 Thu Jul 1 12:11 - 12:14 (00:02) hapless tty1 Thu Jul 1 12:10 - 12:10 (00:00) reboot system boot Thu Jul 1 12:10 root tty1 Thu Jul 1 12:03 - crash (00:07) wtmp begins Thu Jul 1 12:03:07 1999 [root@linux6 log]#
I pulled this last
report from a freshly installed system, so the last
output is meager. When your machine has been running a while, last
reports can be several pages long. In these cases, you'll probably want to pull last
reports on particular users (as opposed to all users). To do so, issue the last
command plus your desired user, as shown in the
following:
last root [root@linux6 log]# last root root tty1 Thu Jul 1 12:12 still logged in root tty1 Thu Jul 1 12:03 - crash (00:07) wtmp begins Thu Jul 1 12:03:07 1999 [root@linux6 log]#
last
supports several command-line options that control the output format and length. These
are summarized in Table 19.1.
Table 19.1.
last
Command-Line Options
Don't underestimate
the value of last
reports. last
can help you investigate intrusions. Here's an example: I remember one case where an authorized local user had apparently logged in to his ISP and used a shell machine there to attack an ISP in Canada. I had a problem with that scenario, though, because when I examined the user's last
report, I saw that in the two years during which he had an account, he had never used Telnet to connect to shell (or any other machine within his ISP's domain). That simply didn't jibe. (It turned out that another local user—a
Linux user, incidentally—had commandeered the account).
You can also use last
to
detect other bogus activity. A new co-worker once asked me why I ran automated scripts that pulled w
, who
, and last
reports every few minutes on several machines and redirected them to a separate log server. He figured it out pretty quickly when we were later faced with a spoofing attack. The attacker had a legitimate account on one shell box and used it to spoof another local machine by impersonating a user from still another. While the attacker did his homework, he apparently didn't do it well enough. For, while the two affected machines both reported supposedly legitimate addresses, my w
, who
, and last
queries caught his real IP and processes on another. When matched against all the other system logs on the affected machines, it became clear who the culprit was. This was so, even though the attacker
had altered some logs.
Attackers are well aware that /var/log/lastlog
and /var/log/wtmp
can give them away. Hence, every cracker keeps an up-to-date cache of sweepers
and cleaners (programs that circumvent default logging systems).
The following are a few with which you can experiment:
Cloak
—This works
not simply on Linux but also SCO, BSD, Ultrix, and HP/UX. Get Cloak
at
http://agape.trilidun.org/~wart/hack/program-hiders/clear_log/cloak.c
.
cloak2—
This is a powerful cloaking tool. As the author puts it, "Now you can
attribute all YOUR CPU usage to others when playing hack!!!" Check out cloak2
at
http://www.k-elektronik.org/arsip/eksploit/stealth-tools/cloak2.c .
utclean
—This is a
utility that erases any evidence of your presence in wtmp
, wtmpx
, utmp
, utmpx
, and lastlog
. Check out utclean
at
http://www.hoobie.net/security/exploits/hacking/utclean.c.
remove
—This will
clean utmp
, wtmp
, and lastlog
, erasing any evidence of your presence. remove
is superior to many competitors, because it actually removes the entries and does not leave gaps in files. (Gaps are a sure giveaway is the system administrator looks more closely). Check out remove
at
http://nmrc.org/files/unix/remove.c.
utmpedit
(by Anon E. Mouse)—This is a simple, quick utmp
editor. Get
it at
http:// www.k-elektronik.org/arsip/eksploit/stealth-tools/utmpedit.c.
SYSLOG Fogger (by
[email protected]
)—This is a versatile tool for
adding bogus syslog
entries. (This tool works remotely). Check it out at
http://www.k-elektronik.org/ arsip/eksploit/stealth-tools/sysfog.c.
marry
(by Proff)—This is
a powerful (and perhaps, the ultimate) utmp
, wtmp
, and lastlog
editor. Check out marry
at
>http://www.society-of-shadows.com/exploits/bin/marry.c.
Log cleaners are concrete examples of how legitimate programming techniques can be used to circumvent system security. (In other words, much like scanners, log access utilities are tools that can be used equally effectively by well-intentioned and not-so-well intentioned users).
If you want to learn more about log cleaners (or perhaps write your own), examine the source of the utilities mentioned previously. Most log cleaners require either
utmp.h
—A header library that you can use to catch run levels, boot time events, init
processes, login processes, user processes, the type of login, originating hostname, and so on. (See the utmp
or wtmp
man pages for more information.)
unistd.h
—A header library that you can use to catch system messages about error conditions, warning conditions, debugging information, and so on.
The attacker writes code that opens utmp
, and, using something like strncpy
(string copy), replaces
the current line with user-specified data (or simply uses strncpy
to replace the current line with whitespace or nothing).
To hedge your bets against crackers tampering with your log entries, you should use at least one third-party or proprietary logging tool. This approach offers two powerful advantages. First, few crackers will know (or bother to verify) that you are running special logging tools. Second, such tools will derive their logs independently, without using operating system logs as a starting index. If you later compare this information to default system logs and find a discrepancy, you'll instantly know that an intrusion has taken place.
Also consider insulating your logs from tampering. For example, write them to write-once media or a remote log server. This is a little more expensive, but it does guarantee that you'll have one set of reliable logs and reliability is everything.
xferlog
records
FTP file transfers. As explained in the xferlog
manual page:
The xferlog
file contains logging information from the FTP server daemon, ftpd(8). This file usually is found in /usr/adm, but can be located anywhere by using a option to ftpd(8). Each server entry is composed of a single line…
Output fields include the following:
The current time
The duration of the file transfer
The remote host (hostname/IP)
The size of the file transferred
The filename
The transfer type (binary/ASCII)
Any special action taken (if the file was compressed or tarred)
The direction of the transfer (incoming, outgoing)
The access mode (anonymous, guest, or authenticated user)
The username
The service
Authentication method
The authenticated user ID
The following shows some sample output:
[root@linux6 log]# more xferlog Thu Jul 1 12:15:14 1999 1 172.16.0.1 694 /home/hapless/index.html a _ i r hapless ftp 0 * Thu Jul 1 13:20:17 1999 1 172.16.0.1 694 /home/hapless/index.html a _ o r hapless ftp 0 * [root@linux6 log]#
These entries show that user hapless
(from 172,16.0.1
) conducted two transfers—one incoming (i
), one outgoing (o
)—of index.html
as an authenticated user
(r
) at the specified times.
httpd
stores its
logs in /var/log/httpd/apache
in two files:
Let's look at the format of these files now.
access_log
stores the
following values:
The visitor's IP address
The event's time and date
The command or request
The status code
The following shows some sample output:
[root@linux6 apache]# more access_log 172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET / HTTP/1.0" 200 1879 172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET / HTTP/1.0" 200 1879 172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET /mmback.gif HTTP/1.0" 404 204 172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET /mmback.gif HTTP/1.0" 404 204 172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET /head.gif HTTP/1.0" 200 17446 172.16.0.1 - - [01/Jul/1999:13:09:46 -0700] "GET /head.gif HTTP/1.0" 200 17446 172.16.0.1 - - [01/Jul/1999:13:09:57 -0700] "GET /mmback.gif HTTP/1.0" 404 204 172.16.0.1 - - [01/Jul/1999:13:09:57 -0700] "GET /mmback.gif HTTP/1.0" 404 204 172.16.0.1 - - [01/Jul/1999:13:10:04 -0700] "POST / HTTP/1.0" 405 228 172.16.0.1 - - [01/Jul/1999:13:10:04 -0700] "POST / HTTP/1.0" 405 228 172.16.0.1 - - [01/Jul/1999:13:10:06 -0700] "GET /mmback.gif HTTP/1.0" 404 204 172.16.0.1 - - [01/Jul/1999:13:10:06 -0700] "GET /mmback.gif HTTP/1.0" 404 204
Table 19.2 provides a quick reference for HTTP status codes.
Table 19.2.
httpd
Status Codes
error_log
stores the
following fields by default:
The date and time
The type of report (error)
The reason for the error
The service
The action taken (sometimes)
The following shows some sample output:
[root@linux6 apache]# more error_log [Thu Jul 1 12:03:01 1999] [notice] Apache/1.3.1 (Unix) configured -- resuming normal operations [Thu Jul 1 13:09:46 1999] [error] File does not exist: /home/httpd/html/mmback.gif [Thu Jul 1 13:09:57 1999] [error] File does not exist: /home/httpd/html/mmback.gif [Thu Jul 1 13:10:06 1999] [error] File does not exist: /home/httpd/html/mmback.gif [Thu Jul 1 13:33:30 1999] [notice] httpd: caught SIGTERM, shutting down [Thu Jul 1 13:35:04 1999] [notice] Apache/1.3.1 (Unix) configured -- resuming normal operations [Thu Jul 1 13:51:39 1999] [notice] httpd: caught SIGTERM, shutting down [Thu Jul 1 21:23:28 1999] [notice] Apache/1.3.1 (Unix) configured -- resuming normal operations
Apache allows you
to customize your logs with the LogFormat
directive. The following is the default:
LogFormat "%h %l %u %t "%r" %s %b"
This indicates that by default, Apache logs
Table 19.3 summarizes LogFormat directives.
Table 19.3.
httpd
LogFormat
Directives
System and kernel messages are handled by two daemons:
To see syslogd
and klogd
in action, you must turn to /var/log/messages
.
/var/log/messages
receives
message output from syslogd
and klogd
.
System and kernel diagnostic messages appear in the order in which they were received:
[root@linux6 log]# more messages Jul 1 12:02:50 linux6 syslogd 1.3-3: restart. Jul 1 12:02:52 linux6 kernel: klogd 1.3-3, log source = /proc/kmsg started. Jul 1 12:02:52 linux6 kernel: Loaded 4122 symbols from /boot/System.map-2.0.35. Jul 1 12:02:52 linux6 kernel: Symbols match kernel version 2.0.35. Jul 1 12:02:52 linux6 kernel: Loaded 95 symbols from 16 modules. Jul 1 12:02:52 linux6 kernel: VFS: Mounted root (ext2 filesystem) readonly. Jul 1 12:02:52 linux6 kernel: lp0 at 0x03bc, (polling) Jul 1 12:02:52 linux6 kernel: CSLIP: code copyright 1989 Regents of the University of California Jul 1 12:02:52 linux6 kernel: SLIP: version 0.8.4-NET3.019-NEWTTY- MODULAR (dynamic channels, max=256). Jul 1 12:02:52 linux6 kernel: PPP: version 2.2.0 (dynamic channel allocation) Jul 1 12:02:52 linux6 kernel: PPP Dynamic channel allocation code copyright 1995 Caldera, Inc. Jul 1 12:02:52 linux6 kernel: PPP line discipline registered. Jul 1 12:02:52 linux6 kernel: Swansea University Computer Society IPX 0.34 for NET3.035 Jul 1 12:02:52 linux6 kernel: IPX Portions Copyright (c) 1995 Caldera, Inc. Jul 1 12:02:52 linux6 kernel: sysctl: ip forwarding off Jul 1 12:02:52 linux6 amd[23101]: My ip addr is 0x100007f Jul 1 12:02:52 linux6 amd[23102]: file server localhost type local starts up Jul 1 12:02:53 linux6 amd[23102]: /etc/amd.localdev mounted fstype toplvl on /
In addition to
standard syslog
and kernel messages, you'll also find messages from network services:
Jul 1 12:10:38 linux6 syslog: LOGIN ON tty1 BY hapless Jul 1 12:11:36 linux6 syslog: FAILED LOGIN 1 FROM 172.16.0.1 FOR haples, User not known to the underlying authentication module Jul 1 12:11:36 linux6 syslog: FAILED LOGIN 1 FROM 172.16.0.1 FOR haples, User not known to the underlying authentication module Jul 1 12:11:40 linux6 syslog: LOGIN ON ttyp0 BY hapless FROM 172.16.0.1 Jul 1 12:12:12 linux6 syslog: ROOT LOGIN ON tty1 Jul 1 12:14:37 linux6 ftpd[23622]: FTP LOGIN FROM 172.16.0.1 [172.16.0.1], hapless Jul 1 12:14:41 linux6 ftpd[23622]: FTP session closed Jul 1 12:15:07 linux6 ftpd[23625]: FTP LOGIN FROM 172.16.0.1 [172.16.0.1], hapless Jul 1 12:15:15 linux6 ftpd[23625]: FTP session closed
To customize syslog
logging, specify
your rules in syslog.conf
. As explained in the syslog.conf
manual page:
The syslog.conf
file is the main configuration file for the syslogd(8)
which logs system messages on *nix systems. This file specifies rules for logging. For special features see the sysklogd(8)
man page.
In syslog.conf
, you define rules with two fields:
In the Selector
field, you
must specify at least one of two values:
The message
type
The message
priority
The message
type
is called a
facility
and must be one of the following:
auth
—auth
is a security facility that tracks user authentication in various services such as FTP, login, and so on. (Essentially, the auth
facility tracks any user action that requires a username and password to login or uses the target resource.)
authpriv
—authpriv
is a security facility that tracks security/authorization messages.
cron
—cron
tracks messages from the cron
system. cron
is a daemon that executes scheduled commands. (See the cron
man page for more
information.)
daemon
—daemon
tracks additional system daemon messages.
kern
—kern
tracks kernel messages.
lpr
—lpr
tracks line printer system messages.
mail
—mail
tracks mail system messages.
news
—news
tracks news system messages.
uucp
—uucp
tracks UNIX-to-UNIX Copy subsystem messages.
You can specify blanket logging using only the
facility
and no
priority
. For example, here's a rule that specifies that the system should send all kernel
messages to the console:
kern.* /dev/console
Here, the
facility
is kernel and the
action
is to log to /dev/console
. Or, if you wanted to log all kernel messages to /var/log/messages
, you could establish a rule such as the following:
kern.* /var/log/messages
The second half of the Selector
field is the
Priority
, which is not always necessary unless you want to refine your output. The Priority must be one of the following:
alert
—Alerts indicate serious malfunctions that demand immediate attention.
debug
—debug
messages provide debugging information on running processes.
emerg
—emerg
(emergency) messages indicate emergency conditions.
err
—err
(error) messages consist of typical STDERR
.
info
—info
(informational messages) are plain old for-your-information messages from programs.
notice
—notice
messages are standard messages.
warning
—warning
messages are standard warnings (for example, the system or resource couldn't perform the requested task).
So, for example, if you wanted to log error messages from your news system, you might create a rule such as the following:
# Save news errors of level err and higher # in a special file. news.err /var/log/spooler
Here, your values are
In the action field, you
specify what syslog
should do with the messages you've requested. As seen earlier, one possible choice is to log the messages to a particular file. Other choices include the following:
Named pipes
The terminal or console
A remote machine (if it's running syslogd
)
Specified users
All users
For example, suppose you wanted to send your kernel messages to the remote
host linux3
(running syslogd
). You might create a rule such as the following:
kern.* @linux3
Or, perhaps you want to send all alerts to user support
. You could create a rule such as the following:
*.alert support
The sample syslog.conf
file provided with Linux offers several pre-fabricated possibilities:
[root@linux6 conf]# more /etc/syslog.conf syslog.cong # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log everything (except mail and news) of level info or higher. # Hmm--also don't log private authentication messages here! *.info;news,mail,authpriv,auth.none -/var/log/messages # Log debugging too #*.debug;news,mail,authpriv,auth.none -/var/log/debug # The authpriv file has restricted access. authpriv.*;auth.* /var/log/secure # true, 'auth' in the two previous rules is deprecated, # but nonetheless still in use… # Log all the mail messages in one place. mail.* /var/log/mail # As long as innd insists on blocking /var/log/news # (instead of using /var/log/news.d) we fall back to … news.* /var/log/news.all # Save uucp and news errors of level err and higher # in a special file. uucp,news.err /var/log/spooler # Everybody gets emergency messages, plus log them on # another machine. *.emerg * #*.emerg @loghost
If you plan to build a large Linux network, I recommend logging to both local and remote locations. This will ensure some level of redundancy. (It's always a good idea to have several versions. You never now when disaster might strike.)
Eventually, you'll write your own daemons and logging utilities. Here, I thought it would be
useful to briefly address how to write syslog
from within your programs.
In C, include the syslog
library syslog.h
. As explained on the syslog(3)
manual page:
syslog()
generates a log message, which will be distributed by syslogd(8)
. Priority is a combination of the facility and the level… The remaining arguments are a format, as in printf(3)
and any arguments required by the format, except that the two character %m will be replaced by the error message string (strerror
) corresponding to the present value of errno
.
In your syslog
call, include
a syslog
level. Table 19.4 summarizes the levels.
Table 19.4.
syslog
Levels
Level | Function |
---|---|
LOG_CRIT
| Logs a critical message. |
LOG_DEBUG
| Logs a debug-level message. |
LOG_EMERG
| Logs an emergency message. |
LOG_ERR
| Logs an error message. |
LOG_INFO
| Logs an informational message. |
LOG_NOTICE
| Logs a notice message. |
LOG_WARNING
| Logs a warning condition. |
You can also include a syslog
facility, if appropriate. Table 19.5 summarizes the possible
facilities.
Table 19.5.
syslog
Facilities
Facility | Function |
---|---|
LOG_AUTHPRIV
| Specifies that the current message is type AUTH (a security, authentication, or authorization notification). |
LOG_CRON
| Specifies a clock daemon (cron/at ) message. |
LOG_DAEMON
| Specifies a system daemon message. |
LOG_KERN
| Specifies a kernel message. |
LOG_LPR
| Specifies a line printer daemon message. |
LOG_MAIL
| Specifies a mail subsystem message. |
LOG_NEWS
| Specifies a Usenet news message. |
LOG_SYSLOG
| Specifies an internal syslog message. |
LOG_USER
| Specifies a generic user message. |
LOG_UUCP
| Specifies a UUCP message. |
To write syslog
, call syslog()
with at least a level and a message. For example, compile and run the
following code:
#include <syslog.h> void main(int argc,char *argv[]) { syslog(LOG_ALERT,"This is my alert. "); syslog(LOG_CRIT,"This is my critical message. "); syslog(LOG_DEBUG,"This is my debug-level message. "); syslog(LOG_EMERG,"This is my emergency message. ")); syslog(LOG_ERR,"This is my error. "); syslog(LOG_INFO,"This is my informational message. "); syslog(LOG_NOTICE,"This is my notice. "); syslog(LOG_WARNING,"This is my warning. "); }
When you check syslog
, you'll see
a corresponding run of messages:
July 23 11:15:55 linux6 syslog: This is my alert message. July 23 11:15:55 linux6 syslog: This is my critical message. July 23 11:15:55 linux6 syslog: This is my debug-level message. July 23 11:15:55 linux6 syslog: This is my emergency message July 23 11:15:55 linux6 syslog: This is an error. July 23 11:15:55 linux6 syslog: This is my information message. July 23 11:15:55 linux6 syslog: This is my notice. July 23 11:15:55 linux6 syslog: This is my warning.
How you integrate this functionality into your program will depend on what the program does, but typically you'd construct a separate block for sending the message:
if(some-condition) /* If some operation fails… */ report_error(); /* Jump to report_error() to write syslog */ void report_error(const char str) { syslog(LOG_WARN, "%s failed: %d (%m),", str, errno); syslog(LOG_WARN, "Warning: the user has run amok..n"); exit(1); }
In Perl, use the Sys::Syslog
module
(a Perl interface to syslog
), like this:
use Sys::Syslog; if(some operation fails) { &report_error; } sub report_error { syslog(LOG_WARN, "Something terrible has happened. "); exit 0; }
In both C and Perl, instances
may arise where you must use openlog()
and closelog()
, or explicitly set the mask or socket type. Please see the syslog.h
and Sys::Syslog
manual pages for more details. Also, note that depending on your installation, you may not have Sys::Syslog
. If not, find updated modules at CPAN, the Comprehensive Perl Archive Network, at
http://www.cpan.org/
, or at http://www.perl.com.
Finally, if you prefer Java, Acme Labs offers Acme.Syslog
, a ready-made syslog
manipulation
Java class. Find it at
http://www.acme.com/java/software/Acme.Syslog.html.
On a single Linux box, log files grow slowly, but in a Linux network with a dozen users or more, heavy logging can result in massive files. If you anticipate generating massive logs, you should arrange to back up or rotate
them. For this, one solution is Erik Troan's logrotate
.
logrotate
rotates, compresses, and
mails system logs. As explained in the manual page:
logrotate
is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.
You run logrotate
as a cron
job. Each time it runs, it reads options from a user-specified configuration file. The following is a typical configuration file
entry:
errors [email protected] compress /var/log/messages { rotate 5 weekly postrotate /sbin/killall -HUP syslogd endscript }
The first two lines define global options. In this case, they specify that all errors should be mailed to
[email protected]
, and that all log files
should be compressed for transport:
errors [email protected] compress
After defining your global options, you must set rules for each log file. To do so, you construct special sections using a directive-based language.
Each section begins with the log filename (in the earlier example, it's /var/log/messages
). From there, everything between the open and close brackets ({
and }
) are
directives
. Let's take another look at the earlier example, this time commented:
/var/log/messages { # Take the log file /var/log/messages rotate 5 # Rotate it five times before removing it weekly # Perform rotations once a week postrotate # Finally, after rotating the file… /sbin/killall -HUP syslogd # Execute this command endscript # …and close up this section }
You can control many aspects of the log file rotation using directives (either globally or on a per-section basis). Table 19.6 summarizes these directives and what they do.
Table 19.6.
logrotate
Configuration
File Directives
Finally, this next section covers several interesting and useful logging and audit tools that don't ship with Linux (see Table 19.7).
Table 19.7. Tools to Enhance Your Logging Security
Tool | Description and Location |
---|---|
Ippl
|
ippl is a
multi-threaded tool that logs incoming IP packets. You can establish rules for which packet types you'd like to filter. Location:
http://www.via.ecp.fr/~hugo/ippl/. |
Log Scanner |
Log Scanner is
a Perl-based tool that works in concert with TCP wrappers. It enables you to set up a log parser that will contact you (or others) when predefined anomalies are
discovered in a log file. Location:
http://logscanner.tradeservices.com/. |
Logcheck
|
Logcheck is one
component of the Abacus Project. Logcheck processes logs generated by the Abacus Project tools, system daemons, TCP Wrapper, logdaemon , and the TIS Firewall Toolkit. Location:
http://www.psionic.com/abacus/logcheck/. |
LogWatch |
LogWatch analyzes
your logs for a user-specified time period and generates customizable reports. Location:
http://www.kaybee.org/~kirk/html/linux.html. |
netlog
|
netlog is a
collection of network monitoring and logging utilities (tcplogger , udplogger , netwatch , and extract ). netlog can log all TCP connections (and UDP sessions) on a subnet and provides real-time monitoring and reporting. Location:
http://net.tamu.edu/ftp/security/TAMU/netlog.README. |
PIKT |
PIKT is the
Problem Informant/Killer Tool. PIKT monitors multiple workstations for problems and, if appropriate, automatically fixes those problems. Example problems include disk failures, log failures, queue overflows, erroneous or suspicious permission changes. Location:
http://pikt.uchicago.edu/pikt/. |
Plugshot's TST | TST is the Tagged Shell Toolkit, which enables you to log and audit user shell commands. Check it out at http://www.plugslot.com/. |
Secure Syslog
|
Secure Syslog is
a new cryptographically secure system-logging tool. Designed to replace the syslog daemon, Secure Syslog implements a cryptographic protocol called PEO-1 that allows the remote auditing of system logs. Auditing remains possible, even if an intruder gains superuser privileges in the system. Location:
http://www.core-sdi.com/english/index.html. |
Also, there are several useful utilities that border on being both intrusion detection and logging analysis systems, including the following:
Platform: UNIX (Perl is required)
The authors wrote SWATCH to supplement logging capabilities of out-of-the-box UNIX systems. SWATCH, consequently, has logging capabilities that far exceed your run-of-the-mill syslog
. SWATCH provides real-time monitoring, logging, and reporting. And, because SWATCH is written in Perl, it's both
portable and extensible.
SWATCH has several unique features, including the following:
A "backfinger" utility that attempts to grab finger information from attacking host
Support for instant paging (so you can receive up-to-the-minute reports)
Conditional execution of commands (if this condition is found in a log file, do this)
Lastly, SWATCH relies on local configuration files. Conveniently, multiple configuration files can exist on the same machine. Therefore, while originally intended only for system administrators, any local user with adequate privileges can use SWATCH.
Learn more about SWATCH in Chapter 20, "Intrusion Detection."
Kenneth Ingham
Kenneth Ingham Consulting
1601 Rita Dr. NE
Albuquerque, NM 87106-1127
Phone: (505) 262 0602
Email:
[email protected]
URL: http://www.i-pi.com/
Ingham developed Watcher while at the University of New Mexico Computing Center. He explains that, at the time, the Computing Center was expanding and the logging process they were using was no longer adequate. Therefore, Ingham was looking for a way to automate scanning of logs. Watcher was the result of his labors.
Watcher analyzes various logs and processes, looking for radically abnormal activity. (The author sufficiently fine-tuned this process so Watcher can interpret the widely variable output of commands like ps
without setting off alarms.)
NOCOL/NetConsole v4.0 is a suite of standalone applications that performs a wide variety of monitoring tasks. This suite offers a Curses interface, which is great for running on a wide range of terminals (it does not require X to work). It is extensible, has support for a Perl interface, and operates on networks running AppleTalk and Novell.
NOCOL/NetConsole v.4.0 is available online at ftp://ftp.navya.com/pub/vikas/nocol.tar.gz.
Author: Jeff Thompson
PingLogger logs
ICMP packets to an outfile. Using this utility, you can reliable determine who is ping
flooding you. The utility was originally written and tested on Linux (it requires a C compiler and IP header files) but may work on other UNIX
systems.
University of Hamburg, Dept. of Computer Science
DFN-CERT
Vogt-Koelln-Strasse 30
22527 Hamburg, Germany
Location: ftp://ftp.cert.dfn.de/pub/tools/audit/logsurfer/logsurfer-1.41.tar.gz.
LogSurfer
is a comprehensive log analysis tool. The program examines plain text log files and, based on what it finds (and the rules you provide), it can perform various actions. These might include creating an alert, executing an external program, or even taking portions of the log data and feeding that to external commands or processes. LogSurfer
requires
C.
Netlog, developed at Texas A&M University, can log all TCP and UDP traffic. This tool also supports logging of ICMP messages (though the developers report that performing this logging activity soaks up a great deal of storage). To use this product, you must have a C compiler.
Netlog is available online at ftp://coast.cs.purdue.edu/pub/tools/unix/TAMU/.
University of Cambridge Statistical Laboratory
Analog is a truly cross-platform log file analyzer. In addition to Linux, Analog currently runs on the following operating systems:
Macintosh
OS/2
Windows 95/NT
Vax/VMS
RiscOS
BeOS
BS2000/OSD
Analog also has built-in support for a wide variety of languages, including English, Portuguese, French, German, Swedish, Czech, Slovak, Slovene, Romanian, and Hungarian.
And, as if that weren't enough, Analog also does reverse DNS lookups (slowly), has a built-in scripting language (similar to the shell languages), and has at least minimal support for AppleScript.
Finally, Analog supports most of the well-known Web server log formats, including Apache, NCSA, WebStar, IIS, W3 Extended, Netscape, and Netpresenz. For these reasons, Analog is a good tool to have around (especially in heterogeneous networks).
Never underestimate the importance of keeping detailed logs. Not only are logs essential when you're investigating a network intrusion, they're also a requisite for bringing charges against an attacker. Now that you know a bit about logs, the next step is to learn how you can use them to detect intrusions. That's what Chapter 20 is all about.
18.218.171.212