Logs are another set of double-edged swords that lie quietly behind the scenes. They can completely save your butt, or completely overwhelm you, depending on the situation. Their importance, however, is frequently underestimated.

Logs are useful for a number of things. They can help you troubleshoot problems. They can be used for tracking down network anomalies. They can help trace an intruder’s steps, or help solidify your case in a court of law. However, if you don’t have a logging strategy, rest assured you will eventually come to regret it.

If your operating system already supports logging, you might be tempted to forgo additional logging tools. Try to resist that temptation. You can’t always trust your logs. In fact, altering logs to cover one’s tracks is one of the first things crackers learn. The practice has become so common that there are tools that automate the process. A list of the many available tools can be found at http://www.apocalypseonline.com/security/tools/tools.asp?exp_category=Log%20Cleaners, but here are a few of the more well-known ones:

root      console  Fri Jun 19 17:01 - down   (00:01) 
root      console  Fri Jun 12 12:26 - down  (4+02:16) 
root      console  Tue May 19 10:45 - down   (01:50) 
root      console  Fri May  1 11:23 - down   (00:02) 
root      console  Fri Apr 24 09:56 - 09:56  (00:00) 
root      console  Mon Mar 23 02:53 - down   (00:01) 
root      console  Mon Mar 23 02:43 - down   (00:01) 

To hedge your bets against crackers tampering with your log entries, you should create a logging strategy that’s difficult to circumvent. The easiest way to achieve this is to write your logs to a one-way write-once device, or to copy your logs to a secured logging server. Some administrators have their Unix machines write their logs to a serial port that is attached to a standalone machine. Although this is certainly quite secure, the model doesn’t scale very well.

One model that is a little more scalable revolves around using the syslog protocol. syslog is a native service on almost every Unix platform, and recently add-on products have made it available on other platforms (such as Windows NT) as well. Although there are more secure alternatives to syslog, it is now common across most router and firewall products. This ubiquity gives administrators a common denominator with which to centralize all logging. For example, administrators can configure all hosts to log to a protected and centralized syslog-based logging server—giving security teams a single point at which to coordinate log data (see Figure 13.1).

Figure 13.1. Centralizing logging.

When configured properly, the only traffic allowed to the syslog server is traffic destined for UDP port 514 (the syslog port). By sending system logs to a separate, secure machine, you make it a LOT more difficult for intruders to clean their tracks.

In addition to centralizing your logs, you might want to consider using at least one third-party logging or parsing tool. This approach has several advantages. First, although the cracker community is familiar with operating system-based logs, few crackers have the knowledge or the means to circumvent third-party logging software. Second, good third-party software packages derive their logs independently of the operating system logs. You’ll know that intruders have penetrated your system when you compare this information and discover a discrepancy between your third-party logs and your regular logs.

This is especially true if you insulate your third-party logs. For example, suppose you use a third-party logging tool to later verify the integrity of operating system-based logs. Although expensive, writing those third-party logs to write-once media guarantees you one set of reliable logs, and reliability is everything.

Using third-party products is prudent in case your out-of-the-box logging utilities fail. For example, on some versions of Solaris, the tmpx file will truncate incoming hostnames, rendering any data obtained via last erroneous and incomplete.

Coming at this from a different angle, it’s now a pretty common procedure for crackers to kill your logging capabilities prior to launching a real attack. If the target is running an unpatched version of Solaris 2.5.x, for instance, you can kill syslogd simply by sending it an external message from a nonexistent IP address. Similarly, if syslogd accepts remote messages, anyone can make a false entry in the log.

You should consider an alternative logging system for all these reasons. The next section briefly covers several good ones.

The following tools not only report data from logs, they also collect data from diverse sources. Note that some of these tools are starting to tread pretty close to the intrusion detection space, which we covered in detail in Chapter 12, “Intrusion Detection Systems.” It will be interesting to see whether the two types of utilities will interoperate over time.

The following tools examine log files, extract the data, and generate reports.

Never underestimate the importance of keeping detailed logs. Not only are logs essential when you’re investigating a network intrusion, they’re also a requisite for bringing charges against an attacker. Sparse logs simply won’t do.

In recent years, many criminal cracking cases have ended in plea bargains. One of the primary reasons for this is because perpetrators were often kids—kids who were just “having a little fun.” However, plea bargains are becoming less prevalent as real criminal elements migrate to the Net. Real criminals know that proving a case before a judge or jury is very difficult (especially if the prosecution has little Internet experience). When judges and jurors are asked to send a human being to prison, they need substantial proof. The only way you can offer substantial proof is by having several fail-safe methods of logging.

Crimes perpetrated over the Internet are unlike most other crimes. For example, in a robbery case, crooks are placed in a lineup so the victim can identify the culprit. In burglary cases, fingerprints will generally reveal the identity of the perpetrator. On the Internet, however, you have neither a physical description nor fingerprints. Therefore, without logs, making a case against a cracker is almost impossible.