Chapter 11. System monitoring: Working with log files

This chapter covers

• Filtering log entries to maintain system health
• The care and feeding of your Linux logging system
• Filtering text streams using grep, awk, and sed
• Deploying intrusion detection systems

If all you had to work with was just the things you’ve learned so far in this book, I’d say you’re ready to put together a pretty respectable server. It’ll be connected, automated, backed up, open for remote clients requesting data and other services, and at least reasonably secure. All the comforts of home.

Time to put your feet up and enjoy the view? Not yet. Your server may be properly configured, but you’ll also need to keep an eye on the way it handles the road once it enters its production environment. How does that work? As you’ll soon see, most Linux system monitoring consists of reading log files.

A log entry is a text-based record of some system event. When a user enters authentication credentials, a remote client requests data from a web server, an application crashes, or a new hardware device is connected, a descriptive note is appended to one or more log files.

Even a lightly used system can generate many thousands of lines of log files between the time it’s booted up and shut down, and busy applications can easily produce millions of lines a day. Because log files tend to be long and boring, you’ll probably want to outsource their reading to software that can intelligently filter only urgent entries, like warnings of imminent failures, and notify you only when absolutely necessary. The better you get at configuring your system’s log behavior and managing the ever-growing log files, the better you’ll understand your system’s strengths and weaknesses—and the more reliable your system will become.

For those who can read them, log files are a treasure chest of valuable insights. They can tell you about weaknesses in your security defenses and past unauthorized intrusions. Log entries can help you anticipate problems with your system security and performance, and diagnose them once everything has already blown up.

You could say that this chapter is about providing you with an inventory of the log resources at your disposal and describing the best-practice approaches to configuring, consuming, and managing your logs. You’ll also learn about intrusion detection tools that can be set to regularly scan your server and network environments looking for telltale signs of suspicious activity.

But the chapter is really focused on trouble-proofing your system to protect it from security breaches and performance outages. If you’re an admin responsible for a critical public-facing server, then this kind of trouble-proofing is for you.

11.1. Working with system logs

For decades, Linux logging has been managed by the syslogd daemon. Syslogd would collect the log messages that system processes and applications sent to the /dev/log pseudo device. Then it would direct the messages to appropriate plain text log files in the /var/log/ directory. Syslogd (figure 11.1) would know where to send the messages because each one includes headers containing metadata fields (including a timestamp, and the message origin and priority).

Figure 11.1. The flow of log data from sample sources through the syslogd daemon

In the wake of systemd’s relentless, world-conquering juggernaut, Linux logging is now also handled by journald. I say also because syslogd hasn’t gone anywhere, and you can still find most of its traditional log files in /var/log/. But you need to be aware that there’s a new sheriff in town whose (command line) name is journalctl.

Like everything else connected to systemd, the move to journald has been contentious. No one will deny the value behind journald’s new functionality (illustrated in figure 11.2). You’ll soon see for yourself how it introduces some powerful filtering and searches. But the fact that journald stores log data in a binary file rather than plain text is definitely an issue.

Figure 11.2. The journald logs (including those generated by syslogd) are consumed using the journalctl command-line tool.

Practically, because they’re binary, it might sometimes be difficult or even impossible to access your logs. Think about it: when you most need it (while struggling to recover from a system crash, perhaps), journalctl will probably be out of service. Syslogd logs, on the other hand, will always be accessible to you as long as you can mount their drive. This can be immensely helpful when you’re trying to figure out what brought down your system and what has to be done to get it back up. The good news is that both systems will continue to coexist for the foreseeable future.

11.1.1. Logging with journald

Here’s one that came my way last week. A worried developer brought me his laptop running Ubuntu, complaining that it was dying. The boot had, in fact, stalled, leaving a black screen displaying a single message that read something like this:

/dev/sda1: clean, 127663/900212 files, 709879/3619856 blocks

The fact that I was getting a reference to /dev/sda1 told me that the hard drive was alive, the boot process had made it past GRUB, and Linux was at least partially loaded. In fact, that particular screen message turned out to be misleading, because it represented the last successful stage the boot process cleared before stalling and had nothing to do with the problem itself.

After a bit of trial and error, including booting the laptop to a live USB drive and mounting the laptop’s drive (the way you saw back in chapter 6), it turned out that Linux had fully loaded, but it had failed to launch the GUI desktop. How did I discover that gem? Back in a failed boot (with the /dev/sda1:... showing onscreen), I pressed Alt-F1 and was taken to the login of a virtual console. From there I had full command-line shell access.

Tip

It turns out that you can open multiple virtual consoles using Alt-F2, Alt-F3, and so forth. You can also switch back and forth between them through Alt-<the appropriate F-key>. Keep in mind that you can return to the host shell (usually your GUI session) using Alt-F7.

Why was the desktop GUI not loading? And why am I even telling you about this whole business in the first place? The answer to both questions can be found, you guessed it, in the laptop’s logs.

To figure it all out, let’s take a small detour to learn a bit more about journalctl and then get back to that poor laptop. Typing journalctl by itself returns a screen full of the oldest log entries currently on the system. The first line, however, displays the start and end date of the available entries. In the following, there are nearly nine months of log entries, which would require a great many screens to display in order:

# journalctl
-- Logs begin at Thu 2016-12-15 08:46:17 EST,
 end at Mon 2017-09-04 21:14:54 EDT. --

As you’re likely to be interested in more recent activity, the -n 20 argument will display only the last 20 entries:

# journalctl -n 20

It’s always a good idea to check your logs every now and then. I ran that command on my own workstation to confirm that it works as expected and discovered an orphaned OpenVPN client. Bless its little heart. Every 5 seconds, the client was faithfully trying to connect with a VPN server that, sadly, no longer existed, and then sending off new entries to tell me about it. To resolve the problem, I used systemctl to first stop and then disable the OpenVPN service.

As I wrote earlier, the more accurately you can narrow down results, the quicker you’ll get the information you’re after. One useful way to filter logs is to display results by priority. Adding -p emerg, for instance, shows you only those log entries categorized as an emergency. If you know something has broken on your system but can’t isolate the problem, emerg would be a good first place to look:

# journalctl -p emerg

Besides emerg, you can also filter for debug, info, notice, warning, err, crit, and alert messages (see syslogd’s priority levels in the next section). Adding the -f flag (for follow) displays the 10 most recent entries and any subsequent entries as they’re created. This allows you to watch events in real time as they occur:

# journalctl -f

You can also filter logs by date and time. And here’s where we arrive back at the GUI-less laptop. If you’ve got a pretty good idea when the boot process ground to a halt, you can narrow down your search to return only events from that time frame. Fortunately, the --since and --until arguments take dates and times. If you don’t specify a date, you’ll get the most recent time meeting those criteria.

In my case, I could specify a 2-minute bracket within which I figured the failure occurred. Two minutes, especially during a system boot, can still produce a lot of log entries—but it will be a whole lot easier to browse than 20 minute’s worth:

# journalctl --since 15:50:00 --until 15:52:00

Had I run that command on the laptop, I might have saved myself some time. As it happened, however, I went with the older syslog log files instead. Let’s see how those work.

11.1.2. Logging with syslogd

All the logs generated by events on a syslogd system are added to the /var/log/syslog file. But, depending on their identifying characteristics, they might also be sent to one or more other files in the same directory.

With syslogd, the way messages are distributed is determined by the contents of the 50-default.conf file that lives in the /etc/rsyslog.d/ directory. This example from 50-default.conf shows how log messages marked as cron-related will be written to the cron.log file. In this case, the asterisk (*) tells syslogd to send entries with any priority level (as opposed to a single level like emerg or err):

cron.*    /var/log/cron.log

Working with syslogd log files doesn’t require any special tools like journalctl. But if you want to get good at this, you’ll need to know what kind of information is kept in each of the standard log files. Table 11.1 lists the most common syslogd log files and their purposes. (To get you up to speed on the others, check out their man pages. man lastlog is one example.)

Table 11.1. Commonly used syslogd facilities

Filename	Purpose
auth.log	System authentication and security events
boot.log	A record of boot-related events
dmesg	Kernel-ring buffer events related to device drivers
dpkg.log	Software package-management events
kern.log	Linux kernel events
syslog	A collection of all logs
wtmp	Tracks user sessions (accessed through the who and last commands)

In addition, individual applications will sometimes write to their own log files. You’ll often also see entire directories like /var/log/apache2/ or /var/log/mysql/ created to receive application data. Log redirection can also be controlled through any one of eight priority levels, in addition to the * symbol you saw before. Table 11.2 lists syslogd’s priority levels.

Table 11.2. Syslogd priority levels

Level	Description
debug	Helpful for debugging
info	Informational
notice	Normal conditions
warn	Conditions requiring warnings
err	Error conditions
crit	Critical conditions
alert	Immediate action required
emerg	System unusable

Now, curious about what I found on that laptop? Well, it’s not really relevant to this chapter, but because you’ve been such a nice reader I’ll tell you anyway. This one appeared in the syslog log file itself:

xinit: unable to connect to X server: connection refused

X server is the Linux system that handles desktop graphical interfaces. If the user was unable to connect, it sounds like one of two things:

• The X server system had become corrupted.
• There was some kind of authentication problem.

It wasn’t the former because, by this time, I’d already successfully logged in to a desktop session using a different user account. But it was difficult to see how the laptop’s owner would have been able to authenticate in a command-line shell and still be refused access to the desktop. Nevertheless, it seemed worth exploring further, so I opened up the auth.log file, and here’s what I saw (with username used here in place of the developer’s name):

lightdm: pam_succeed_if(lightdm:auth): requirement
   "user ingroup nopasswdlogin" not met by user "username"

I know that lightdm is the desktop manager used by Ubuntu computers, and pam is a module that handles Linux user authentication. But I’ll admit that I didn’t understand the full significance of the message until I fed it into my favorite internet search engine.

There I learned that other people facing that message had traced it to incorrect ownership of the .Xauthority file kept in the user’s home directory. Apparently X server can only load a GUI session if the .Xauthority file belongs to the user. I checked, and this user’s .Xauthority file actually belonged to root. Fixing the problem was as simple as running chown to change the ownership back to username:

$ ls -al | grep Xauthority
-rw-------  1 root root  56 Sep  4 08:44 .Xauthority
# chown username:username .Xauthority

How did the ownership get mixed up in the first place? Who knows. But it happened. And the smart use of log files helped me fix it.

11.2. Managing log files

With hundreds of system processes spitting out thousands of log messages each hour, an unmanaged log system quickly fills all the storage space available to it. At that point logging will break down, along with any other system processes relying on that space. What to do? Continue reading.

11.2.1. The journald way

Journald deals with the problem by automatically limiting the maximum disk space the journald system will be allowed to use. Once a limit is reached, older messages are removed and deleted. This setting is controlled by the SystemMaxUse= and RuntimeMaxUse= settings in the /etc/systemd/journal.conf file.

What’s the difference between these settings? By default, journald builds and maintains its journal in the /run/log/journal file, which is a volatile file that’s destroyed each time the system is shut down. You can, however, direct journald to maintain a persistent journal file at /var/log/journal. You’d use either one or the other of those two journal.conf settings depending on your system setup. Converting to a persistent journal file requires only that you create a /var/log/journal/ directory, and use systemd-tmpfiles to direct log traffic appropriately:

# mkdir -p /var/log/journal
# systemd-tmpfiles --create --prefix /var/log/journal

11.2.2. The syslogd way

By default, syslogd handles log rotation, compression, and deletion behind the scenes without any help from you. But you should know how it’s done in case you ever have logs needing special treatment.

What kind of special treatment could a simple log ever require? Well, suppose your company has to be compliant with the transaction reporting rules associated with regulatory or industry standards like Sarbanes-Oxley or PCI-DSS. If your IT infrastructure records must remain accessible for longer periods of time, then you’ll definitely want to know how to find your way through the key files.

To see the logrotate system in action, list some of the contents of the /var/log/ directory. The auth.log file, for instance, appears in three different formats:

• auth.log—The version that’s currently active, with new auth messages being written to it.
• auth.log.1—The most recent file to have been rotated out of service. It’s maintained in uncompressed format to make it easier to quickly call it back into action should it be necessary.
• auth.log.2.gz—An older collection (as you can see from the .gz file extension in the following listing) that, because it’s less likely to be needed, has been compressed to save space.

Listing 11.1. Contents of the /var/log/ directory

$ ls /var/log | grep auth
auth.log                     1
auth.log.1                   2
auth.log.2.gz                3
auth.log.3.gz
auth.log.4.gz

• 1 The currently active version of auth.log
• 2 The most recently retired version
• 3 Subsequent versions are compressed using gzip.

When, after seven days, the next rotation date arrives, auth.log.2.gz will be renamed auth .log.3.gz, auth.log.1 will be compressed and renamed auth.log.2.gz, auth.log will become auth.log.1, and a new file will be created and given the name auth.log. The default log rotation cycle is controlled in the /etc/logrotate.conf file. The values illustrated in this listing rotate files after a single active week and delete old files after four weeks.

Listing 11.2. Some common settings from the /etc/logrotate.conf file

# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# packages drop log rotation information into this directory
include /etc/logrotate.d

The /etc/logrotate.d/ directory also contains customized configuration files for managing the log rotation of individual services or applications. Listing the contents of that directory, you see these config files:

$ ls /etc/logrotate.d/
apache2  apt  dpkg  mysql-server  rsyslog  samba  unattended-upgrade

Here’s what the apt config file looks like on my system.

Listing 11.3. Contents of the /etc/logrotate.d/apt log rotate configuration file

/var/log/apt/term.log {
  rotate 12                   1
  monthly                     2
  compress                    3
  missingok
  notifempty
}
/var/log/apt/history.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

• 1 Files will be rotated 12 times before being deleted.
• 2 Rotations will take place once a month.
• 3 Rotated files will be immediately compressed.

Tip

Many admins choose to redirect log entries to purpose-built remote log servers where the data can receive all the specialized attention it deserves. This can free up application servers for their immediate tasks and consolidate log data in a single, easily accessible central location.

11.3. Consuming large files

You know you’ve got better things to do with your time than read through millions of lines of log entries. The following sections describe three text-processing tools that can do it for you—faster and better.

11.3.1. Using grep

Based on my experience, at least, the most versatile and straightforward tool for filtering text is our old friend grep. Here’s an obvious example that will search through the auth.log file for evidence of failed login attempts. Searching for the word failure will return any line containing the phrase authentication failure. Checking this once in a while can help you spot attempts to compromise an account by guessing at the correct password. Anyone can mess up a password once or twice, but too many failed attempts should make you suspicious:

$ cat /var/log/auth.log | grep 'Authentication failure'
Sep  6 09:22:21 workstation su[21153]: pam_authenticate: Authentication failure

If you’re the kind of admin who never makes mistakes, then this search might come up empty. You can guarantee yourself at least one result by manually generating a log entry using a program called logger. Try it out doing something like this:

logger "Authentication failure"

You could also pre-seed a genuine error by logging in to a user account and entering the wrong password.

As you can tell, grep did the job for you, but all you can see from the results is that there was an authentication failure. Wouldn’t it be useful to know whose account was involved? You can expand the results grep returns by telling it to include the lines immediately before and after the match. This example prints the match along with the lines around it. It tells you that someone using the account david (that would be me, I guess) tried unsuccessfully to use su (switch user) to log in to the studio account:

$ cat /var/log/auth.log | grep -B 1 -A 1 failure
Sep  6 09:22:19 workstation su[21153]: pam_unix(su:auth): authentication
   failure; logname= uid=1000 euid=0 tty=/dev/pts/4 ruser=david rhost=
   user=studio
Sep  6 09:22:21 workstation su[21153]: pam_authenticate:
   Authentication failure
Sep  6 09:22:21 workstation su[21153]: FAILED su for studio by david

By the way, you can also use grep to search within multiple files, which I’ve personally found useful while writing this book. My problem was that each chapter is saved as a separate file in its own directory, so searching, say, for each reference to the word daemon can get tedious. Because copies of the plain text chapter files are also saved to a single manuscript directory, I can move to that directory and search using grep -nr. There it is, in chapter 3:

$ grep -nr daemon                    1
linux-admin-chapter3.adoc:505:       2
[...]

• 1 The n argument tells grep to include line information; the r argument returns recursive results.
• 2 The number 505 in the result tells me that the match appeared on line 505 in the specified file.

11.3.2. Using awk

There’s a lot that grep can do, but it’s not omnipotent. Let’s take these entries from the /var/log/mysql/error.log file as an example.

Listing 11.4. A selection of entries from a /var/log/mysql/error.log file

2017-09-05T04:42:52.293846Z 0 [Note] Shutting down plugin 'sha256_password'
2017-09-05T04:42:52.293852Z 0 [Note]
 Shutting down plugin 'mysql_native_password'
2017-09-05T04:42:52.294032Z 0 [Note] Shutting down plugin 'binlog'
2017-09-05T04:42:52.294647Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

2017-09-05T12:24:23.819058Z 0 [Warning] Changed limits:
 max_open_files: 1024 (requested 5000)
2017-09-05T12:24:23.819193Z 0 [Warning] Changed limits:
 table_open_cache: 431 (requested 2000)

As you can see, entries are categorized by a priority level enclosed in brackets ([Warning], for example). Suppose you’d like to know how many warning messages there have been since the log was last rotated. Perhaps you regularly compare the numbers with an accepted baseline frequency to know when you might have a problem needing investigation. You can grep for each time [Warning] shows up and then pipe that to wc, which will count the lines, words, and characters in the output:

# cat error.log | grep [Warning] | wc
   4219   37292  360409

It would seem that [Warning] shows up 4,219 times. Someone is trying to get your attention here. But try the same thing using the awk tool. wc will return only 204 lines! What’s going on?

# cat error.log | awk '$3 ~/[Warning]/' | wc
    204    3213   27846

It seems that grep uses brackets to contain a list of characters and searches the text stream for any one of those. Because there are six unique letters in Warning, pretty much every line in the log file was a match. You could always remove the brackets and search for Warning, but there might be cases where that returns too many false positives.

As long as the search string is enclosed in forward slashes (/[Warning]/), awk will have no trouble with the brackets. And the sophisticated syntax of awk comes with its own distinct benefits. Take that $3 in the previous example, for instance. By default, awk divides a line of text into individual fields, each separated by one or more spaces. This log entry, for instance, has seven fields:

2017-09-05T04:42:52.294032Z 0 [Note] Shutting down plugin binlog

grep interprets the final four words (Shutting down plugin binlog) as four distinct data fields, so it would be hard to figure out a way to usefully search those. But the third field represents the entry’s priority level, so you can run awk with $3 to return only matching results from that field.

11.3.3. Using sed

In case typing wc to return the number of lines in a stream doesn’t appeal to you, you could always do the same thing with sed, in whose universe = will print the current line number (-n tells sed to not print the text itself):

# cat error.log | awk '$3 ~/[Warning]/' | sed -n '$='
204

OK, but you’re not likely to ever need to drag out an industrial-strength stream editor like sed just to do something simple like count lines. I only brought it up as an excuse to introduce you to sed and, in particular, its ability to make complex substitutions on text streams.

Here’s a simple example where sed is given the text hello world and told to substitute (s) the first instance of the word world with the word fishtank:

$ echo "hello world" | sed "s/world/fishtank/"
hello fishtank

Adding a g before the final quotation mark would tell sed to substitute every instance of world, assuming there were more than one.

That’s nice, but not very useful. Where sed shines is in cleaning up text as part of some larger process. Perhaps the next step of a Bash script you’re working on needs to operate on only one part of the input it’s receiving. Or maybe you’re preparing text so it will be more readable for an end user.

Let’s say you’ve got a file called numbers.txt that contains some code with line numbers. You need to remove the line numbers without messing with the code itself, including the lines that are part of the code. Here’s some text you can use (paste it into a file on your own computer to follow along):

Listing 11.5. A sample code snippet with unwanted line numbers

4 <menuitem action='Item 1'/>
5 <menuitem action='Item 2'/>
6 <menuitem action='Item 3'/>
7 <menuitem action='Exit'/>
8   <separator/><

Now feed the numbers.txt file to sed: use the caret character (^) to point sed to the start of each line. Specify any number, and the instance of any number at that first position will be deleted. Note how the line numbers have disappeared, but the Item numbers are still there:

$ sed "s/^ *[0-9]* //g" numbers.txt
<menuitem action='Item 1'/>
<menuitem action='Item 2'/>
<menuitem action='Item 3'/>
<menuitem action='Exit'/>
  <separator/><

Alternatively, you could redirect the output to a new file:

$ sed "s/^ *[0-9]* //" numbers.txt > new-numbers.txt

Finally, sed can selectively print only the subdirectories (and not individual files) from a directory listing. Here’s what an unfiltered listing would look like:

$ ls -l
total 28
drwxrwxr-x 2 dbclinton dbclinton 4096 Sep  7 14:15 code
-rw-rw-r-- 1 dbclinton dbclinton   55 Sep  6 11:53 file
-rw-rw-r-- 1 dbclinton dbclinton   76 Sep  6 11:54 file2
-rw-rw-r-- 1 dbclinton dbclinton 4163 Sep  6 00:02 mysql.log
-rw-rw-r-- 1 dbclinton dbclinton  137 Sep  7 13:58 numbers
drwxrwxr-x 2 dbclinton dbclinton 4096 Sep  7 14:15 old-files

And here’s how that would go using sed with ^d to print (p) only those lines starting with a d (which, as you know, designates a directory):

$ ls -l | sed -n '/^d/ p'
drwxrwxr-x 2 dbclinton dbclinton 4096 Sep  7 14:15 code
drwxrwxr-x 2 dbclinton dbclinton 4096 Sep  7 14:15 old-files

11.4. Monitoring with intrusion detection

Besides logs, there’s another way to monitor your system’s health and welfare: intrusion detection. The idea is to create a baseline profile of your system state, the way it’s supposed to look, and then scan the system periodically looking for the kinds of changes that suggest there’s mischief afoot.

One way to go about this is by implementing a network-based intrusion detection System (NIDS), which relies on software (like Snort, Nmap, and Wireshark) to regularly “sniff” the network neighborhood looking for devices, open ports, and hosts that shouldn’t be there. We’re not going to spend time on NIDS here (although my “Linux Network Security” course on Pluralsight does).

Closer to home, you can also build a host-based intrusion detection system (HIDS) to keep a steady eye on your server. That’s what we’re going to learn how to do using an open source package called Tripwire.

Tripwire scans your server and adds key attributes of important system files (like the file size) to its own database. If any one of those files should be edited or deleted, or if new files are added to directories being monitored, those attributes will change. When you later tell Tripwire to check the system, it compares the current values to the ones stored in the database and reports any discrepancies.

Note

The Tripwire company also provides a commercial version that offers centralized management for multiple installations, policy compliance, support, and compatibility with Windows (not that we’ll be needing that one). It’s not cheap, but for enterprise-scaled deployments, it can be worth the cost.

To get Tripwire going, you’ll first install a plain vanilla mail server to send email reports to any admins you specify. You’ll install and configure Tripwire itself, and then edit and encrypt its policy and configuration files (tw.cfg and tw.pol, respectively). Finally, you’ll simulate a system change to see how it looks in an emailed report.

11.4.1. Setting up a mail server

Creating a basic Linux mail server is a lot easier than you might think. Install the postfix (and, for Ubuntu, mailutils) package. During installation (for Ubuntu, at least), select Internet Site when prompted for a mail configuration type and localhost.localdomain for your system mail name. You’ll also want to make sure the inet_interfaces setting in the /etc/postfix/main.cf file is set to localhost. All that’s left is to make sure port 25 is open for SMTP on any firewall you’ve got running and to restart postfix:

# systemctl restart postfix

That wasn’t so bad, was it? You should now have an active mail server that can send emails to remote recipients. It’s not ready to handle incoming mail, but you don’t need that for this project.

11.4.2. Installing Tripwire

Installation is straightforward, although there are some minor variations between Ubuntu and CentOS. I’ll cover each in its own section.

Ubuntu

As apt installs the Tripwire package, you’ll be prompted to create new passphrases for two sets of signing keys. This process will briefly leave your passphrase unencrypted and exposed to anyone else who happens to be logged in to your system (figure 11.3).

Figure 11.3. Warning screen from the Debian/Ubuntu installation process for Tripwire

After creating passphrases from both sets of keys, you’ll be asked whether you want to rebuild the encrypted Tripwire configuration file (figure 11.4). Rebuilding the configuration and policy files is required after any edits to their source text files. You’ll see how to do it manually later, but this first time the Tripwire installation does it for you.

Figure 11.4. Final screen before Tripwire rebuilds the encrypted configuration file

When the installation is complete, you’ll be shown a screen (figure 11.5) containing information about the locations of key Tripwire files. Make a note of those, and in particular note the location of the documentation in /usr/share/. As you might expect, there are also configuration files in the /etc/tripwire/ directory.

Figure 11.5. When installation completes, Tripwire displays the locations of important files.

CentOS

CentOS installation lacks the fancy, cutting-edge, vintage 1980’s graphics accompanying the Ubuntu wizard. In fact, assuming you already have the epel-release repository installed (install epel-release), it’s a matter of installing the Tripwire package and watching the lines of text fly by.

In CentOS, the signing keys are created by running a program called tripwire-setup-keyfiles after the main installation. There you’ll be prompted to create two passphrases. Once the installation is done (for both Ubuntu and CentOS), you’ll need to run tripwire --init to initialize the database:

[...]
Wrote database file: /var/lib/tripwire/tripwire.twd
The database was successfully generated.

If you’re setting up Tripwire on an LXC container, be aware that you’ll likely see a lot of errors like these:

The object: "/proc/cpuinfo" is on a different file system...ignoring.
The object: "/proc/diskstats" is on a different file system...ignoring.
The object: "/proc/meminfo" is on a different file system...ignoring.

This is happening because some of the system files normally monitored by Tripwire are being shared with the container’s host. This means that processes running in the container may not always have full access. Don’t worry, though. Other than producing lots of nasty-looking error messages, this won’t have any impact on the way Tripwire works.

11.4.3. Configuring Tripwire

You control the way Tripwire behaves through two files kept in the /etc/tripwire/ directory: tw.cfg and tw.pol. The thing is, those files are encrypted and are not only impossible to edit but, unless you’ve got some special superhero skills, they’re unreadable, too. The files are built from the information in two plain text files: twcfg.txt and twpol.txt.

The twcfg.txt file contains some basic environment variables, all of which you can change to fit your needs. Most of them will work perfectly well right out of the box. But changing some of the file locations can add a layer of security through obscurity, especially if you’re concerned that the bad guys might be looking to identify and disable the alarm system before getting down to do their work. If you do decide to move things around, twcfg.txt is the file you’ll edit.

Listing 11.6. The default contents of the twcfg.txt file

ROOT                   =/usr/sbin
POLFILE                =/etc/tripwire/tw.pol
DBFILE                 =/var/lib/tripwire/$(HOSTNAME).twd
REPORTFILE             =/var/lib/tripwire/report/$(HOSTNAME)-$(DATE).twr  1
SITEKEYFILE            =/etc/tripwire/site.key
LOCALKEYFILE           =/etc/tripwire/$(HOSTNAME)-local.key
EDITOR                 =/bin/vi                                           2
LATEPROMPTING          =false
LOOSEDIRECTORYCHECKING =false
MAILNOVIOLATIONS       =true
EMAILREPORTLEVEL       =3                                                 3
REPORTLEVEL            =3
MAILMETHOD             =SENDMAIL
SYSLOGREPORTING        =false
MAILPROGRAM            =/usr/sbin/sendmail -oi -t

• 1 Note how the report filename reflects the server’s hostname for easy identification.
• 2 You can change the default text editor to Nano if you’re more comfortable with it.
• 3 You may find changing the verbosity setting to 1 will make reports easier to read.

One thing you’ll want to add to this file is the email address (or addresses) where you’d like reports to be sent. You do this by adding a GLOBALEMAIL line pointing to your address:

GLOBALEMAIL   [email protected]

The twpol.txt file sets the policies Tripwire will use to categorize and scan your file system. The file gives you a working generic policy, but you’ll almost certainly need to make at least some customizations to accommodate your particular server setup. You can add those customizations gradually over time, as you get a feel for the kinds of false positives you’re seeing in reports, and for the things Tripwire might be missing.

You’ll probably get a feel for the kinds of false positives your system will produce pretty quickly. Likely culprits include process ID (PID) files and configuration files for applications you’re actively working on. Once you’ve run Tripwire a couple of times and start to recognize the false positives, look through the twpol.txt file for the line referencing the offending file (the line for the bashrc file, for instance, would look like /etc/bashrc -> $(SEC_CONFIG) ;) and comment it out with the # character (meaning, #/etc/bashrc -> $(SEC_CONFIG) ;).

Browse through the twpol.txt file you’ve installed on your own server, and note the way policies are defined by rules with names like Invariant Directories. Each rule is assigned a severity level (like SIG_MED, for medium significance, in the following listing). An Invariant designation means that you don’t expect such objects to change permission or ownership, and that you should therefore be alerted if such events do occur.

Listing 11.7. A sample Tripwire policy rule from the twpol.txt file

# Commonly accessed directories that should remain static with  regards
# to owner and group.

(
  rulename = "Invariant Directories",
  severity = $(SIG_MED)
)
{
  /                                    -> $(SEC_INVARIANT) (recurse = 0) ;
  /home                                -> $(SEC_INVARIANT) (recurse = 0) ;
  /etc                                 -> $(SEC_INVARIANT) (recurse = 0) ;
}

Obviously, you’re free to edit the contents and values of any of the dozen or so rules that make up Tripwire policy to suit your needs. The file is well documented, so spending 10 or 15 minutes reading through it should make the kinds of things you can do with it quite clear.

After editing the plain text files, and assuming you’re in the same directory, update the encrypted versions using twadmin --create-cfgfile and twadmin --create-polfile:

# twadmin --create-cfgfile --site-keyfile site.key twcfg.txt
Please enter your site passphrase:
Wrote configuration file: /etc/tripwire/tw.cfg
#
# twadmin --create-polfile twpol.txt
Please enter your site passphrase:
Wrote policy file: /etc/tripwire/tw.pol

Because the source files are in plain text, you should delete those as soon as their settings have been successfully incorporated into their encrypted cousins:

$ cd /etc/tripwire
# rm twcfg.txt
# rm twpol.txt

If, sometime in the future, you need to update your configuration, you can recover the original values by running twadmin --print-cfgfile or twadmin --print-polfile. Those commands give you a file in which you can make any necessary edits:

# twadmin --print-cfgfile > twcfg.txt
# twadmin --print-polfile > twpol.txt

It’s time to take Tripwire out for a test drive. On the command line, tripwire takes many arguments with a -m prefix, where m stands for module, so -m c would load the check module. Running a check prints a report to the screen that will (probably) mostly consist of dozens of file system error messages about files and directories that aren’t there:

# tripwire -m c
[...]
176. File system error.
     Filename: /proc/pci
     No such file or directory
------------------------------
*** End of report ***
Integrity check complete.

Now you can update the Tripwire database based on the results of the previous scan:

# tripwire -m u
Please enter your local passphrase:
Wrote database file: /var/lib/tripwire/localhost.localdomain.twd

It’s possible that Tripwire will complain that it’s unable to open the report file. If that happens, run --update -r against the most recent file currently in the /var/lib/ tripwire/report/ directory. You’ll be shown the report in a text editor. Exit, and the update will continue (:q! should do the trick if you find yourself in the vi editor):

tripwire -m u -r  /var/lib/tripwire/report/
  localhost.localdomain-20170907-102302.twr            1

• 1 You don’t need to include the full filename, including its date/time stamp, as long as you include enough to be unique.

11.4.4. Generating a test Tripwire report

Let’s make some trouble and see whether Tripwire will notice. Adding a new user to the system will leave fingerprints all over the place. At the very least, the passwd, shadow, and group files in /etc/ will be updated. Give your friend Max an account and password:

# useradd max
# passwd max
Changing password for user max.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

Now run tripwire, specifying that an email report should be sent using level 1 verbosity:

tripwire --check --email-report --email-report-level 1

If everything works out, you should receive an email that includes text like this:

Modified:       "/etc"
Modified:       "/etc/group"
Modified:       "/etc/group-"
Modified:       "/etc/gshadow"
Modified:       "/etc/gshadow-"
Modified:       "/etc/passwd-"
Modified:       "/etc/shadow-"
Modified:       "/etc/subgid"
Modified:       "/etc/subgid-"
Modified:       "/etc/subuid"
Modified:       "/etc/subuid-"
Modified:       "/etc/tripwire"
Modified:       "/etc/tripwire/tw.pol"
Modified:       "/etc/tripwire/tw.cfg"
Modified:       "/etc/passwd"
Modified:       "/etc/shadow"

What’s left? Once you’re done fine-tuning your policies, you can add tripwire --check and tripwire -m u commands to cron to automate the process. Happy hunting!

Summary

• Journald logging is built around a binary journal file that can be finely parsed using journalctl to target specific matches.
• Log files grow quickly and must be controlled through a rotation regimen that takes business and regulatory needs into account.
• The grep, awk, and sed text-filtering and -formatting tools can be used to manage large volumes of data: grep, by string matches; awk, by breaking strings down into fields; and sed, through string substitution.
• Tripwire can be used for ongoing server monitoring to alert admins should it sense the presence of suspicious behavior.

Key terms

• Journald stores its log data in a binary file, which allows for more flexible retrieval but requires a running host system.
• On a syslogd system, all log events are written to the /var/log/syslog file.
• Log rotation for syslogd log files is controlled through the /etc/logrotate.conf file and through individual files in the /etc/logrotate.d/ directory.
• A text stream filter (like grep) searches text for matching strings and, usually, returns the entire line containing the match.
• A network-based intrusion detection system (NIDS) monitors a private network for evidence of infiltration.
• A host-based intrusion detection system (HIDS) monitors servers for evidence system files have been maliciously altered.
• A mail server allows emails to be sent and received by applications or at the command-line level.

Security best practices

• Regularly scan the auth.log file for evidence of repeated failed attempts to access user accounts.
• Configure scanning software like an intrusion detection system to push alerts to admins when potentially compromising events occur.
• The original plain text files used to create encrypted Tripwire configuration and policy files should be deleted immediately after use.

Command-line review

• Alt-F<n> opens a virtual console from a non-GUI shell.
• journalctl -n 20 displays the 20 most recent log entries in the journal.
• journalctl --since 15:50:00 --until 15:52:00 displays only events between the since and until times.
• systemd-tmpfiles --create --prefix /var/log/journal instructs systemd to create and maintain a persistent journal rather than a file that’s destroyed with every boot.
• cat /var/log/auth.log | grep -B 1 -A 1 failure displays matching lines along with the lines immediately before and after.
• cat /var/log/mysql/error.log | awk '$3 ~/[Warning]/' | wc searches the MySQL error log for events classified as a warning.
• sed "s/^ [0-9] //g" numbers.txt removes numbers at the start of each line of a file.
• tripwire --init initializes the database of a Tripwire installation.
• twadmin --create-cfgfile --site-keyfile site.key twcfg.txt generates a new encrypted tw.cfg file for Tripwire.

Test yourself

1

The /dev/log pseudo device is used for

1. Storing the journald ephemeral journal file
2. Storing the journald persistent journal file
3. Collecting log event data for syslogd
4. Storing log even data collected by syslogd

2

Which of the following commands will display the five most recent log entries in the journal file?

1. journalctl -l 5
2. journalctl -n 5
3. journalctl -f 5
4. journalctl --since 5

3

Which of the following directives would send only kernel-related emergency messages to the existing kern.log file?

1. kern.emerg -/var/log/kern in the /etc/rsyslog.d/50-default.conf file
2. kern.* -/var/lib/kern.log in the /etc/rsyslog.d/50-default.conf file
3. *.emerg -/var/log/kern.log in the /etc/rsyslog.d/30-default.conf file
4. kern.emerg -/var/log/kern.log in the /etc/rsyslog.d/50-default.conf file

4

Which configuration file is used to control the log-rotation policies for files in /var/log/?

1. /etc/logrotate.conf
2. /etc/systemd/journal.conf
3. /etc/logrotate.d
4. /etc/rsyslog.conf

5

What arguments will tell grep to include surrounding lines of text along with the matching line it displays?

1. cat /var/log/auth.log | grep --B 1 --A 1 failure
2. cat /var/log/auth.log | grep --since 1 --until 1 failure
3. cat /var/log/auth.log | grep -B 1 -A 1 failure
4. cat /var/log/auth.log | grep -b 1 -a 1 failure

6

Which of the following sed commands will remove line numbers from lines of text?

1. sed "s/^ [0-9] //g"
2. sed -n '/^d/ p'
3. sed -n '/^d/ [0-9] p'
4. sed "s/^ [0-9] //"

7

Which of the following commands will prepare the Tripwire database for operation?

1. tripwire-setup-keyfiles
2. tripwire --init
3. twadmin --create-polfile twpol.txt
4. tripwire -m c

8

Which of these commands contains the appropriate syntax to encrypt and update the Tripwire configuration file?

1. wadmin --create-polfile --site-keyfile site.key twcfg.txt
2. wadmin --create-cfgfile --site-keyfile site.key twcfg.txt
3. wadmin --create-cfgfile --local-keyfile local.key twcfg.txt
4. wadmin --create-cfgfile --site-keyfile site.key twpol.txt

Answer key

1.

c

2.

b

3.

d

4.

a

5.

c

6.

a

7.

b

8.

b