MRTG has a few simple dependencies that you may need to fulfill before you dive right into the installation. For starters, you need to have a web server up and running. Apache is typically recommended, but you may be able to get it to work with other web servers. You’ll also need Perl installed and working on your system, and MRTG will require three libraries to build its graphs. The first, gd, is used to generate the graphs that make MRTG what it is. The second is libpng, which is used to generate the images of the graphs. Finally, to compress these images, you’ll need the zlib library. Download locations for all three of these libraries can be found at the MRTG home page (http://people.ee.ethz.ch/~oetiker/webtools/mrtg/).

Once you have the dependencies installed, you can begin the MRTG installation. First, download and untar the source to your build location. Start the MRTG installation with the following command:

	$ ./configure–prefix=/usr/local/mrtg-2

If this produces an error message, you may have to specify where you installed the previously mentioned libraries:

	# ./configure–prefix=/usr/local/mrtg-2 --with-gf=/path/to/gd
	--with-z=/path/to/z --with-png=/path/to/png

If you need help determining where those libraries were installed, run the following command for each library to find its location:

	# find / -type f–name libpng

Once configuration is complete, follow it up with a typical make install:

	# make && make install

The next step is to create the mrtg.cfg file that MRTG will use to determine which devices on your network to query. If you had to create this by hand, things could get a little hairy. Fortunately for us, however, MRTG comes with a command-line configuration tool called cfgmaker that greatly simplifies the creation of the .cfg. Detailed documentation on cfgmaker is available at the MRTG home page, but the following example should be enough to get you started:

	# cfgmaker–global 'WorkDir: /path/to/web/root/mrtg' 
	--output=/etc/mrtg.cfg 
	--global'Options[_]: bits, growright' --output=/etc/mrtg.cfg 
	[email protected] 
	[email protected] 
	Global 'Options[_]: bits, growright' --ifref=descry 
	--ifdescr=alias [email protected]

This will create the configuration file /etc/mrtg.cfg, which will tell MRTG to create bandwidth graphs for router1, router2, and switch1. The graphs will use bits as the primary measurement on the y-axis and will grow toward the righthand side. The–global options add entries that apply to this configuration as a whole, while those that are not specified as global apply only to the devices in which we specify them. The location of the configuration file to create is specified by the–output option.

With a valid config file in hand, we can now run MRTG for the first time. Each time you run MRTG, you’ll need to specify the location from which you want it to read the config file. Also, unless you’ve added it to your path, you’ll need to type out the full path to the executable.

	# /usr/local/bin/mrtg-2/bin/mrtg /etc/mrtg.cfg

You will see some errors the first two times you run MRTG, but pay them no mind—it’s simply complaining because it can’t find any previous MRTG data. After running the command, your MRTG web root should be filled with PNG files. This is great, except it’s a pain to look at them like this, and they’re not exactly labeled in a human friendly format. The solution to this problem can be found in the indexmaker tool. indexmaker works just like the cfgmaker tool, only instead of generating config files, it generates an HTML template with which we can display our MRTG graphs:

	# indexmaker–output=/path/to/web/root/index.html 
	–title="My Network MRTG"–sort=title

This will create an index.html file that sorts and displays our data in a much more user-friendly format, as shown in Figure 9-1. You can then modify the index file just as you would any HTML file to make it display any other information you wish.

Figure 9-1. Network traffic graphs created from MRTG data

The only thing left to do is to automate the process. MRTG wouldn’t be very useful if you had to start it manually every time, so we’ll have to automate it by adding it to cron. Add the following entry to root’s crontab to run MRTG every five minutes:

	*/5 * * * * /usr/local/mrtg-2/bin/mrtg /etc/mrtg.cfg 
		--logging /var/log/mrtg.log

Don’t get impatient waiting to see your pretty new graphs. It will take a day or so for them to begin displaying truly useful data. Once you’ve had them running for a while, though, you’ll be able to pick out all kinds of useful trends in your bandwidth utilization. For instance, you might notice that your bandwidth tends to spike between 8:30 and 9:00 A.M., and then again after lunch. This will help you better understand the utilization of your network, and in turn better serve it. It can be fascinating to simply watch your bandwidth utilization materialize, and then use the information to track and follow trends in network activity. MRTG will create yearly graphs as well as hourly, monthly, and daily graphs. Having such detailed information at your fingertips can help you understand just how much traffic you gained after your web site was Slashdotted, and how your popularity increased even after the story ran.

MRTG has a million uses, and they’re not just limited to tracking bandwidth utilization. With a little modification, you can use it to measure almost anything you want. For more information on modifying MRTG to display other statistics, see the MRTG home page.

—Brian Warshawsky

Even just sticking to Linux boxes offers a wealth of information. I’ve written a script in PHP, runnable from a command line, that gathers basic information and reports on listening TCP ports, using only SNMP. Here’s the script:

	#!/usr/bin/php

	<?php
	snmp_set_quick_print(1);
	$string = "public";
	$host = "livid";
	check_snmp($host);
	spitinfo($host);
	
	function check_snmp($box)//see if this box is running snmp before we throw
				 //requests at it.
	{
		$string="public";
		$infocheck = @snmpget("$box", "$string", "system.sysDescr.0");
		if(! $infocheck)
		{
			die("SNMP doesn't appear to be running on $box");
		}
		else
		{
		return $infocheck;
		}
	}
	function spitinfo($host)//retrieves and displays snmp data.
	{
		   $string = "public";
		   $hostinfo = @snmpget("$host","$string","system.sysDescr.0");
		   list ($k)=array(split(" ", $hostinfo));
		   $os = $k[0];
		   $hostname = @snmpget("$host","$string","system.sysName.0");
		   $user = @snmpget("$host","$string","system.sysContact.0");
		   $location = @snmpget("$host","$string","system.sysLocation.0");
		   $macaddr = @snmpget
				  ("$host","$string","interfaces.ifTable.ifEntry.
				ifPhysAddress.2");
		   $ethstatus =
				   @snmpget("$host","$string","interfaces.ifTable.ifEntry.
					 ifOperStatus.2");
		   $ipfwd = @snmpget("$host","$string","ip.ipForwarding.0");
		   $ipaddr = @gethostbyname("$host");
		   $info=array("Hostname:"=>"$hostname","Contact:"=>"$user",
				"Location:"=>"$location","OS:"=>"$os","MAC Address:"=>
				"$macaddr","IP Address:"=>"$ipaddr","Network Status"=>
				"$ethstatus",
				"Forwarding:"=>"$ipfwd");
		   print "$host
";
		   tabdata($info);
		   print "
TCP Port Summary
";
		   snmp_portscan($hostname);
	}
	function tabdata($data)
	{
		foreach($data as $label=>$value)
			{
				if($label){
				print "$label	";
				}else{
				print "Not Available";
				}
				if($value){
				print "$value
";
				}else{
				print "Not Available";
				}
			}
	}

	function snmp_portscan($target)
	{
			$listen_ports = snmpwalk("$target", "public", ".1.3.6.1.2.1.6.13.1.3.
	0.0.0.0");
			foreach($listen_ports as $key=>$value)
			{
				print "TCP Port $value (" . getservbyport($value, 'tcp') . ")
	listening 
";
			}
	}
	
	?>

Save this script to a file named report.php, and make it executable (chmod 775 report.php). Once that’s done, run it by issuing the command ./report.php.

I’ve hard-coded a value for the target host in this script to shorten things up a bit, but you’d more likely want to feed a host to the script as a command-line argument, or have it read a file containing a list of hosts to prod for data. You’ll also probably want to scan for the number of interfaces, and do other cool stuff that I’ve left out here to save space. Here’s the output when run against my Debian test system:

	Hostname:		livid
	Contact:		jonesy([email protected]
	Location:		Upstairs office
	OS:		Linux
	MAC Address:	0:a0:cc:e7:24:a0
	IP Address:		192.168.42.44
	Network Status	up
	Forwarding:		notForwarding

	TCP Port Summary
	TCP Port 80 (http) listening
	TCP Port 111 (sunrpc) listening
	TCP Port 199 (smux) listening
	TCP Port 631 (ipp) listening
	TCP Port 649 ( ) listening
	TCP Port 2049 (nfs) listening
	TCP Port 8000 ( ) listening
	TCP Port 32768 ( ) listening

You’ll notice in the script that I’ve used numeric values to search for in SNMP. This is because, as in many other technologies, the human-readable text is actually mapped from numbers, which are what the machines use under the covers. Each record returned in an snmpwalk has a numeric object identifier, or OID. The client uses the Management Information Base (MIB) files that come with the Net-SNMP distribution to map the numeric OIDs to names. In a script, however, speed will be of the essence, so you’ll want to skip that mapping operation and just get at the data.

You’ll also notice that I’ve used SNMP to do what is normally done with a port scanner, or with a bunch of calls to some function like (in PHP) fsockopen. I could’ve used function calls here, but it would have been quite slow because we’d be knocking on every port in a range and awaiting a response to see which ones are open. Using SNMP, we’re just requesting the host’s list of which ports are open. No guessing, no knocking, and much, much faster.

There are a couple of tools out there that you can use for log monitoring. One is log-guardian, which is a Perl script that allows you to monitor multiple logfiles for various user-supplied patterns. You can also configure the action that log-guardian takes when a match is found. The downside to using log-guardian is that you must have some Perl knowledge to configure it, since actions supplied by the user are in the form of Perl subroutines, and other configuration parameters are supplied in the form of Perl hashes. All of these are put directly into the script itself or into a separate configuration file. You can grab log-guardian from its web site: http://www.tifaware.com/perl/log-guardian/. Once downloaded, you can put the log-guardian.pl script wherever you store local system tools, such as under /opt or in /var/local. Since it doesn’t come with an init script, you’ll need to add a line similar to this one to your system’s rc.local file:

	/var/local/bin/log-guardian &

The real power of log-guardian comes from Perl’s File::Tail module, which is a fairly robust bit of code that acts just like tail -f. This module is required for log-guardian. To determine whether you have it installed, you can run something like locate perl | grep Tail, or run a quick Perl one-liner like this at the command line:

	$ perl -e "use File::Tail;"

If that returns a big long error beginning with “Can’t find Tail/File.pm” or something similar, you’ll need to install it using CPAN, which should be dead simple using the following command:

	# perl -MCPAN -e shell

This will give you a CPAN shell prompt, where you can run the following command to get the module installed:

	> install File::Tail

The File::Tail module is safe for use on logfiles that get moved, rolled, or replaced on a regular basis, and it doesn’t require you to restart or even think about your script when this happens. It’s dead-easy to use, and its more advanced features will allow you to monitor multiple logfiles simultaneously.

Here’s a simple filter I’ve added to the log-guardian script itself to match on sshd connections coming into the server:

	'/var/log/messages' => [
		{
			label => 'SSH Connections',
			pattern => "sshd",
		   action => sub {
				   my $line = $_[1];
				   print $line;
		   }
		},
	],

That’s about as simple a filter you can write for log-guardian. It matches anything that gets written to /var/log/messages that has the string sshd in it and prints any lines it finds to stdout. From there, you can send it to another tool for further processing or pipe it to the mail command, in which case you could run log-guardian like this:

	# /var/local/bin/log-guardian | mail [email protected]

Of course, doing this will send every line in a separate email, so you might prefer to simply let it run in a terminal. You’ll be able to monitor this output a little more easily than the logfiles themselves, since much of the noise has been filtered out for you.

This sshd filter is just one example—the “pattern” can consist of any Perl code that returns some string that the program can use to match against incoming log entries, and the “action” performed in response to that match can be literally anything you’re capable of inventing using Perl. That makes the possibilities just about endless!

The logcheck utility is not a real-time monitor that will alert you at the first sign of danger. However, it is a really simple way to help weed out the noise in your logs. You can download logcheck from http://sourceforge.net/projects/sentrytools/.

Once downloaded, untar the distribution, cd to the resulting directory, and as root, run make linux. This will install the logcheck files under /usr/local. There are a few files to edit, but the things that need editing are simple one-liners; the configuration is very intuitive, and the files are very well commented.

The main file that absolutely must be checked to ensure proper configuration is /usr/local/etc/logcheck.sh. This file contains sections that are marked with tags such as CONFIGURATION and LOGFILE CONFIGURATION, so you can easily find those variables in the file that might need changing. Probably the most obvious thing to change is the SYSADMIN variable, which tells logcheck where to send output.

	[email protected]

You should go over the other variables as well, because path variables and paths to binaries are also set in this file.

Once this is ready to go, the next thing you’ll want to do is edit root’s crontab file, which you can do by becoming root and running the following command:

	# crontab -e

You can schedule logcheck to run as often as you want. The following line will schedule logcheck to run once an hour, every day, at 50 minutes after the hour:

	50 * * * * /bin/sh /usr/local/etc/logcheck.sh

You can pick any time period you want, but once per hour (or less in smaller sites or home networks) should suffice.

Once you’ve saved the crontab entry, you’ll start getting email with reports from logcheck about what it’s found in your logs that you might want to know about. It figures out which log entries go into the reports by using the following methodology:

These two files are simply lists of strings that logcheck will try to match up against entries in the logs it goes through to create the reports. There is actually a third file as well, /usr/local/etc/logcheck.violations.ignore, which contains strings that are matched only against entries that are already flagged as violations. There’s an example of this in the INSTALL file that comes with the distribution that is more perfect than anything I can think of, so I’ll reiterate it here:

	Feb 28 21:00:08 nemesis sendmail[5475]: VAA05473: to=crowland, ctladdr=root
	(0/0), delay=00:00:02, xdelay=00:00:01, mailer=local, stat=refused

	Feb 28 22:13:53 nemesis rshd: refused connect from [email protected]:1490

	The top entry is from sendmail and is a fairly common error. The stat line
	indicates that the remote host refused connections (stat=refused). This can
	happen for a variety of reasons and generally is not a problem.

	The bottom line however indicates that a person ([email protected]) has tried
	unsuccessfully to start an rsh session on my machine. This is bad (of
	course you shouldn't be running rshd to begin with).

	The logcheck.violations file will find the word 'refused' and will flag it
	to be logged; however, this will report both instances as being bad and you
	will get false alarms from sendmail (both had the word 'refused').

To get around these false positive without also throwing out things you want to know about, you put a line like this in /usr/local/etc/logcheck.violations.ignore:

	mailer=local, stat=refused

This will match only the Sendmail log entry and will be ignored. Any other entries will be caught if they contain the string “refused”.

Of course, it will likely take you some time to fine-tune the reports logcheck sends, but the model of forcing you to tell the tool to explicitly ignore things ensures that it ignores only what you tell it to, instead of making assumptions about your environment.

This script reads syslog output from a FIFO and forwards it as a Jabber message:

	#!/usr/bin/perl
	use Net::Jabber qw(Client);
	use strict;
	# Announce resources
	my %resource = (
	  online => "/announce/online",
	);
	# default options
	my %option = (
	  server => "moocow:5222",
	  user => "admin",
	  type => "online",
	);
	# Default port if none specified
	$option{server} = "moocow:5222";
	# Ask for password if none given
	unless ($option{pass}) {
	  print "Password: ";
	  system "stty -echo";
	  $option{pass} = <STDIN>;
	  system "stty echo";
	  chomp $option{pass};
	  print "
";
	}
	# Connect to Jabber server
	my ($host, $port) = split(":", $option{server}, 2);
	print "Connecting to $host:$port as $option{user}
";
	my $c = new Net::Jabber::Client;
	$c->Connect(
	hostname => $host,
	port => $port,
	) or die "Cannot connect to Jabber server at $option{server}
";
	my @result;
	eval {
	  @result = $c->AuthSend(
	    username => $option{user},
		password => $option{pass},
		resource => "GAIM",
	  );
	};
	die "Cannot connect to Jabber server at $option{server}
" if $@;
	if ($result[0] ne "ok") {
	  die "Authorisation failed ($result[1]) for user $option{user} on
	$option{server}
";
	}
	print "Sending $option{type} messages
";
	# The message. Change the file name in this 'open' line to
	# the name of your fifo.
	open(STATUS, "cat /var/log/log-fifo 2>&1 |")
		|| die "UGH: there's issues: $!";
	while (<STATUS>) {
			   my $xml .= qq[<subject>] .
		($option{type} eq "online" ? "Admin Message" : "MOTD") .
		qq[</subject>];
			   my $to = $host . $resource{$option{type}};
			   $xml .= qq[<message to="$to">];
			   $xml .= qq[<body>];
			   my $message = $_;
		$xml .= XML::Stream::EscapeXML($message);
		$xml .= qq[</body>];
			$xml .= qq[</message>] ;
			$c->SendXML($xml);
			print $xml;
	}

Place this script in a place accessible only by you and/or your admin team (for example, /var/local/adm/bin/jann-log) and change the permissions so that the script is writable and executable only by your admin group. Then open up a Jabber client on your desktop and connect to your Jabber server. Once that’s done, run the script. It should confirm that it has connected to the Jabber server and is awaiting messages from the FIFO.

A simple way to test your auth.warning facility on the server where jann-log is listening for messages is to SSH to the host and purposely use the wrong password to try to log in.

Zabbix is a complicated beast, so there are naturally a few dependencies to note before you rush headlong into the installation. Zabbix is written in PHP, so make sure you have a relatively recent version installed. If you haven’t upgraded in a while, this might be the time to do so. Since Zabbix is completely web-based, you’ll obviously need a web server as well. Par for the course, Apache or Apache2 is the recommended server of choice. Make sure when you install Apache that you configure it with mod-php enabled as well. This ensures that Apache can understand the embedded PHP that makes Zabbix what it is. Then, make sure you have the PHP GD library installed (available from http://www.boutell.com/gd/). While Zabbix will technically run without this, it’s not recommended, as this is the library that generates the network maps and graphs that make Zabbix so useful. Finally, you’ll need a SQL database. While Zabbix supports both PostgreSQL and MySQL, in this example we’ll be using MySQL.

Unfortunately, installing Zabbix isn’t as straightforward as many applications we’ve discussed so far. Some parts of its installation, which I’ll highlight as we go along, are optional.

The first step in getting Zabbix up and running is to download and untar the source code. You can find this at the home page (http://www.zabbix.com). At the time this book was written, the latest version was 1.0. Download the archive file of the latest version, untar it in your normal build location, and navigate to the new directory. First, we’ll need to configure Zabbix to make use of the database choice we’ve selected (MySQL) and to use SNMP. Run the following command to prepare the installation:

	$ ./configure–with-mysql–with net-snmp

This shouldn’t take too long, so don’t grab a beer just yet! Before you move on to the make, you’ll need to take a second to prepare the MySQL database for Zabbix. Navigate to the create/ directory and then start MySQL, create the Zabbix database, and concatenate the .sql scripts to populate the tables:

# mysql–u<username> -p<password>
		   Mysql> create database Zabbix;
		   Mysql> quit;
# cd create/mysql
# cat schema.sql |mysql–u<username> -p<password> Zabbix
# cd ../data
# cat data.sql |mysql–u<username> -p<password> Zabbix

You can now jump back to the root of the Zabbix directory and issue the make command.

Once the make completes, take a moment to copy the contents of the bin/directory to somewhere in your path. I tend to use /usr/local/bin.

	# cp bin/* /usr/local/bin

This is a fairly unsophisticated installation mechanism, but you’re almost done. Now we have to set a few variables so that PHP knows how to properly access your database. Navigate to frontend/php/include in your Zabbix source directory and open the file db.inc.php in your favorite text editor. Make the following changes:

	$DB_TYPE ="MySQL";
	$DB_SERVER ="localhost";
	$DB_DATABASE ="Zabbix";
	$DB_USER ="<MySQL username here>"
	$DB_PWD ="<MySQL password here>"

The $DB_DATABASE variable is the name of the database you created in MySQL for Zabbix earlier. Once these changes have been made, copy the PHP files to your web root:

	# cp–R frontends/php/* /srv/www/htdocs/

Now make the directory /etc/zabbix and copy the sample configuration files to it:

	# mkdir /etc/zabbix
	# cp misc/conf/zabbix_suckerd.conf /etc/zabbix/zabbix_suckerd.conf
	# cp misc/conf/zabbix_trapperd.conf /etc/zabbix/zabbix_trapperd.conf

These sample configuration files are fine for small-time applications, but if you’re planning on deploying Zabbix on a large-scale or enterprise rollout you should read the configuration files section of the online Zabbix manual, available at http://www.zabbix.com/manual/v1.1/config_files.php. Doing so will save you many headaches in the future. Once these files are moved, you’re done with the installation! All that’s left is to fire up the Zabbix daemons and ensure that they work:

	# zabbix_suckerd
	# zabbix_trapperd

Assuming everything went as planned, you can now point your web browser to http://127.0.0.1 and see your new Zabbix installation. When you get to the login screen, enter Admin for your username and leave the password field blank. Once logged in, take a moment to change the default password.

After that installation, you certainly deserve to do something easy now! Fortunately, Zabbix seems to be designed with ease in mind. Let’s start adding some hosts to monitor. The upper section of the screen has the navigation bars that you’ll use to navigate around Zabbix. Click Hosts to add a new host to your monitoring. Figure 9-3 shows the fields available when adding a new host on the Hosts tab in Zabbix.

Figure 9-3. The Zabbix Hosts tab

You’ll see here that you have several options when adding your new hosts. Fill in the options to suit your needs and click Add. Note that if you’d rather monitor by hostname than by DNS (which is often an excellent idea), checking the Use IP Address box will give you an additional box to provide the IP address to monitor. For example, let’s assume we want to configure Zabbix to notify us if 192.168.2.118 ever stops serving FTP traffic. To do so, on the Hosts tab, we would enter 192.168.2.118 in the Host field. We’d then change the port to 22 since we’re interested in FTP traffic. Next, move over to the Items tab. We’ll need to type in a description for this item, so we’ll call it Home-FTP. Under Type, select “Simple check.” In the Key field, enter “ftp.” The rest we can leave as it is. Now wait a few minutes, and check the Latest Values tab. You should see an option there for 192.168.2.118 (or the hostname if you gave it one). Since the FTP server is running, we get a return value of 1. Had the server not been running, we would see 0 in that field. Notice that to the right you have the option to graph, trend, and compare data collected over time. This allows for detailed data analysis on the uptime and availability of your servers. It is also an excellent demonstration of the graphical qualities of Zabbix.

The last aspect of Zabbix that we’ll look at is the mapping feature (shown in Figure 9-4). This is an excellent tool for providing a quick reference map of the network showing detailed status. To begin, click on the lower Network Maps button. Create a new network map by filling in the name you wish to call your new map. If you’ll have a lot of hosts to monitor, change the size of the map to make it bigger. Click Add to continue. Once you’ve created your map, it’s time to add some hosts to it. Select the host we created in the previous example, Home-FTP. You can then select the coordinates you wish for the icon representing Home-FTP to be displayed on. Select the Server icon and click Add. The page will refresh, and when it finishes loading, you’ll see your icon representing Home-FTP on the map. You can continue adding hosts and placing them on the map until you have a full representation of your network.

What we’ve covered here is a fraction of the capabilities of Zabbix. If you’d like to get more in depth with it, you can install the Zabbix agents on the machines you wish to monitor. Once you’ve done that, you can monitor statistics such as CPU utilization, drive space, and anything else that can be monitored via SNMP. You can also define custom triggers to alert you right away to emergency situations. Trigger definition is highly detailed and can get quite elaborate and complex. If you’d like to learn more about this incredibly flexible network monitoring tool, check out the Zabbix web page at http://www.zabbix.com for more information. There is a fairly active forum there dedicated to helping users in need and sharing configuration tips and tricks.

—Brian Warshawsky

Figure 9-4. The mapping features of Zabbix

A configuration file called /etc/syslog.conf controls the syslog daemon. As unimaginative as the config file’s name might be, learn it well because this is a file you’ll need to become very familiar with if you want to master the intricacies of Linux system logging. The file may not make a whole lot of sense upon first glance, but here’s a simple syslog.conf file that I’ll use to explain the syntax further:

	# Log all kernel messages to the console.
	# Logging much else clutters up the screen.
	# kern.*				/dev/console
	# Log anything (except mail) of level info or higher.
	# Don't log private authentication messages!
	*.info;mail.none;authpriv.none;cron.none	/var/log/messages
	# The authpriv file has restricted access.
	authpriv.*				/var/log/secure
	# Log all the mail messages in one place.
	mail.*				-/var/log/maillog
	# Log cron stuff
	cron.*				/var/log/cron
	# Everybody gets emergency messages
	*.emerg				  *
	# Save news errors of level crit and higher in a special file.
	uucp,news.crit			/var/log/spooler
	# Save boot messages also to boot.log
	local7.*				/var/log/boot.log

As you can see in the noncommented lines in this example, there are three main parts to each active line of the configuration file. The first entry on a line is called the facility, which is the underlying subsystem that creates the log messages for the logfiles. There are 13 predefined system facility values: auth, authpriv, cron, daemon, ftp, kern, lpr, mail, mark, news, syslog, user, and uucp. In addition to these, there are also eight others, named local0 through local7, which are for programs to use when implementing their own syslog messages. Each of the predefined facilities refers to a specific aspect of the system. For instance, auth refers to the Linux authorization system, including programs such as login and su. The mark facility is used internally for syslog, and should be left alone for the time being. The daemon facility is for other system daemons that are not listed specifically. You can represent all available facilities by using the asterisk (*) symbol.

The second part to a configuration line is the priority, which is separated from its associated facility by a period. Every time a part of the system sends a message to syslog, that message is coded with a priority. Basically, the program is letting syslog know how important this message is. From lowest to highest, the priority levels are debug, info, notice, warning, err, crit, alert, and emerg. The higher the priority, the more important the message is. Once you hit the emerg priority, the system is rapidly approaching a kernel panic and is probably unusable. You can represent messages of any priority by using the asterisk symbol. For example, local7.* means “messages of any priority from the local7 facility.”

The third and final aspect of the configuration line is the action. This is basically just a short section that tells syslog what to do with the information it has received. To better explain this, let’s look at an example line from the sample configuration file provided above:

	# Log cron stuff
	cron.*				/var/log/cron

Few things are more annoying than scrolling through /var/log/messages and having to wade through all the cron messages, so this kind of configuration option comes in handy. This example means that messages of all priorities issued by the cron facility should be sent to the /var/log/cron logfile. As mentioned previously, the asterisk is a wildcard feature that tells syslog to apply the same rule to every message from cron, regardless of its priority. You can do similar things with the asterisk wildcard for the facility, such as instructing syslog to send every message of priority warning or higher to a specific logfile:

	*.warning				/var/log/problems

Other wildcard features that can be used include the at sign (@), for sending messages to remote syslog hosts; a dash (-), for telling syslog not to sync the disks after every message; and an asterisk in the actions section of the configuration to alert everyone on the system to an issue. For instance, look at the following example from the sample configuration file:

	# Everybody gets emergency messages
	*.emerg				  *

The final asterisk on this line tells syslog to send a message out to every user online via the wall (Write to ALL users) command to let them know of any emergency conditions. These messages will appear in every active terminal window on the system. You can think of configurations like this as Linux’s emergency broadcast system.

Another interesting line in the example syslog.conf file shown earlier in this hack is the line that addresses kernel syslog messages. Rather than being sent to a logfile, all these messages are sent to the console instead. One popular trick using this feature is to direct many of the syslog messages to a virtual console instead of the main console. I often do this on machines that aren’t used much for local work but still have monitors. For example, specifying this line:

	auth,kern.*		/dev/tty5

allows me to see the syslog messages of everyone who logs on—and any issues with the kernel—simply by switching the machine to virtual console 5 (Alt-F5) and leaving it there with the monitor on. Now, whenever I walk by that machine, I can keep track of users logging on and off, or anything else I’ve set it up to do. When I need to work on the server and that would be in the way, I just switch back to my primary console (Alt-F1), and the messages continue to be sent to console 5.

Another interesting syslog option is remote logging. While syslog itself allows for remote logging, there is a more robust solution to be found in syslog-ng [Hack #87] , a new version of syslog. syslog allows you to send messages to remote hosts, but it does so in plain text across the network, so you should use this feature with caution. Here’s how it works: by adding an at sign and a hostname or IP address in the action section of the configuration file, you can specify that syslog send its messages to another waiting remote syslog server. The remote syslog server will need to have the syslog daemon started with the –r option to allow it to listen on port 514 for incoming syslog messages. The following line shows an example of sending all critical kernel messages to the remote machine aardvark for safekeeping.

	kern.crit		@aardvark

Remote logging can be extremely helpful in the event of a system crash, as it allows you to see log messages that you might otherwise be unable to access (since the system that issued them is down). As previously mentioned, these messages are sent in plain text across the network, so be sure to use syslog’s remote logging with caution—and never do it across the Internet. Also, note that if you send certain types of messages to a remote log server, they are not recorded locally unless you create another entry that also sends those same messages to the local log, as in the following example:

	kern.crit		@aardvark
	kern.crit		/var/log/messages

The syslog daemon can be customized in many different ways, but it’s somewhat dated in terms of both capabilities and security. “Centralize System Logs Securely” [Hack #87] provides newer and even more configurable approach to system logging.

—Brian Warshawsky

To implement encrypted remote logging, you’ll need to download and compile three programs. Let’s start with stunnel. Grab the latest instance of the source code from http://www.stunnel.org/download/source.html. Once you’ve got the tarball, unpack it and navigate to your newly created directory. You can now follow the typical installation procedure:

	$ ./configure
	$ make
	# make install

You’ll now need to grab the source for syslog-ng and libol, a library required by syslog-ng. You can download each of these from http://www.balabit.com/downloads/syslog-ng/. Untar and install libol first, then syslog-ng. Installation of these two applications uses the previous typical source install three-step.

Once you’ve successfully installed stunnel, syslog-ng, and libol, you’ll need to create encryption certificates for all the machines between which you want to transfer secure log information.

To transfer log data securely between a remote host and a central log host, communication between the two must be encrypted. In order to successfully use encryption, both hosts must be able to verify their identities and share the encryption keys used for reading and writing the encrypted data. This information is provided by SSL certificates, which can either be granted by a third party or created yourself for use within your organization. (For more than you probably want to know about SSL and certificates, see the SSL HOWTO at http://www.tldp.org/HOWTO/SSL-Certificates-HOWTO/.)

At this point, you must create multiple certificates: one for use by the central log server, and one for each client that sends log information to the server. Later in this section, you’ll install the server certificate on your server and distribute the client certificates to the hosts for which they were created.

The process for creating certificates varies slightly based on the Linux distribution you’re using. For a Red Hat system, it is as follows:

	# cd /usr/share/ssl/certs
	# make syslog-ng-server.pem
	# make syslog-ng-client.pem

As each certificate is generated, the script will ask you several questions regarding your location, hostname, organization, and email address. Once all the questions have been answered, your certificates are generated. Your next step is to verify that only root has access to them:

	[root@aardvark certs]# ls -l *.pem
	-rw-------1 root root 2149 Aug 14 12:12 syslog-ng-client.pem
	-rw-------1 root root 2165 Aug 14 12:12 syslog-ng-server.pem
	[root@aardvark certs]#

There is one last thing you’ll need to do before you start distributing your certificates: extract the CERTIFICATE section from each certificate that is going to a client machine and concatenate the extracted sections into a single file named syslog-ng-client.pem, which you will put on your server along with the server key. The CERTIFICATE key data in a certificate file is the information between the following two lines:

	-----BEGIN CERTIFICATE-----
	-----END CERTIFICATE-----

Copy the syslog-ng-client.pem file over to the /etc/stunnel directory on the server and place a copy of each client’s own certificate in that client’s /etc/stunnel directory. This may sound somewhat complicated, so let’s summarize: all you’re doing here is extracting the CERTIFICATE from each client’s certificate file, concatenating that information into one large client certificate that will reside on your server (along with the server’s certificate), and then copying the individual client certificates to the hosts for which they were intended.

Now, on the server side, edit your stunnel.conf file to read as follows:

	cert = /etc/stunnel/syslog-ng-server.pem
	CAfile = /etc/stunnel/syslog-ng-client.pem
	verify = 3
	[5140]
			accept = your.server.ip:5140
			connect = 127.0.0.1:514

Then make similar changes to stunnel.conf on the client side:

	client = yes
	cert = /etc/stunnel/syslog-ng-client.pem
	CAfile = /etc/stunnel/syslog-ng-server.pem
	verify = 3
	[5140]
			accept = 127.0.0.1:514
			connect = your.server.ip:5140

Once those changes have been made, it’s time to start working on creating your syslog-ng.conf file. The syntax of this file has a steep learning curve and is well beyond the scope of this hack, so use what I’m about to show you as a starting point, and work from there. Far more detail can be found online and in the manpages. On your central log server, add the following to /etc/syslog-ng/syslog-ng.conf:

	options { long_hostnames(off);
				sync(0);
				keep_hostname(yes);
				chain_hostnames(no); };
	source src {unix-stream("/dev/log");
				pipe("/proc/kmsg");
				internal();};
	source stunnel {tcp(ip("127.0.0.1")
				port(514)
				max-connections(1));};				
	destination remoteclient {file("/var/log/remoteclient");};
	destination dest {file("/var/log/messages");};
	log {source(src); destination(dest);};
	log {source(stunnel); destination(remoteclient);};

Then, add the following to your syslog-ng.conf file on each client:

	options {long_hostnames(off);
				sync(0);};
		source src {unix-stream("/dev/log"); pipe("/proc/kmsg");
				internal();};
		destination dest {file("/var/log/messages");};
		destination stunnel {tcp("127.0.0.1" port(514));};
		log {source(src);destination(dest);};
		log {source(src);destination(stunnel);};

Once you’ve done all this, you can start stunnel and syslog-ng to see if everything is working. Before you do so, though, make sure you stop the syslogd service. You don’t want the two of them stepping on each other. To test whether your remote logging is working, use the logger command:

	# logger This is a Test

Then, on your log server, search (or grep) /var/log/messages (or wherever you have remote logs) for “This is a Test”. If you get a response, congratulations—everything is working fine, and you now have encrypted remote logging!

While remote logging has always been a useful and even necessary process, sending valuable system information unencrypted across the void has long been a security risk. Thanks to syslog-ng and stunnel, we no longer have to worry about that. In addition, the flexibility of syslog-ng has moved leaps and bounds beyond what syslogd was ever capable of. It truly is the Next Generation of system logging daemons.

That flexibility comes with a price, though—the syslog-ng configuration file is a complex beast. If you spend a little time getting to know it, however, you’ll find that it’s not quite as hard as it looks. I can assure you that the complexity of the syntax is proportional to its adaptability once you understand it. Listed below are some resources you can consult online for help in configuring your syslog-ng instance to meet your needs.

—Brian Warshawsky

I’ve found that Nagios provides an extremely simple way of taking many of our disparate scripts, notification modules, ping checkers, and other tools, and putting them all under the Nagios umbrella as “plug-ins” without my having to change much of anything. In fact, the monitoring functionality that comes preconfigured with Nagios is all handled through shell, Perl, or C programs that Nagios calls in the background.

The barrier to entry was actually so low that within a day I had a very basic Nagios configuration up and running, with a web interface, email notifications, and basic service and host checks working. By the end of the week, I had configured Nagios to be more discriminating in its notifications (e.g., notify only the DBA if the database service became unavailable, but only the Sun admins if the database server went down). I had also configured host and service dependencies, and told it about our next two scheduled downtimes. I had even found existing plug-ins for Nagios that allowed for the retirement of a couple of our home-grown scripts for monitoring things like a NetApp filer and a MySQL database. Things were looking up!

What’s more is that the Nagios web interface, while it keeps useful enough statistics to help pinpoint when a problem started or predict your disk needs on a file server over the next year, can also easily be integrated with standard tools such as MRTG [Hack #79] or Cacti.

If you want to get really hardcore, you can also use Nagios to collect SNMP traps, or go fully distributed by using Nagios agents, rather than a central polling mechanism, across your machine room.

The only downside to Nagios that I’ve found so far is that, while configuration is pretty brainless, there is no configuration GUI or automation, so it all has to be done by hand (which can be somewhat cumbersome and very time-consuming). The payoff is there, though, so let’s check out some configuration details. I’ll cover only the most basic configuration, because documenting a full-blown Nagios deployment could be another book unto itself!

First you’ll need to install Nagios, either using your distribution’s package management system (for a binary install) or by going to http://www.nagios.org to grab the source and installing according to the plentiful documentation.

We’ll start simple. Your machine room consists of hosts. These hosts run services. If either a host or a service that it runs becomes unavailable, you’ll want Nagios to notify a contact. Thus, the first thing to do is tell Nagios about these entities. To do this, we add entries in the hosts.cfg, services.cfg, and contacts.cfg files. These files may be located under /etc/nagios if your installation was preconfigured (as on a SUSE system or a Red Hat RPM install), or wherever you told it to put configuration files during a source install (/usr/local/etc/nagios, by default).

Here’s a simple hosts.cfg entry that tells Nagios some basic information about a host:

	define host{
		use				generic-host
		host_name		newhotness
		alias			Jonesy's Desktop
		address			128.112.9.52
		parents			myswitch
	}

You’ll notice that all this information is specific to my desktop machine. There’s nothing here about how to check the availability of the host, when to check it, or anything else. This is because Nagios allows you to configure a template host entry to hold all of that information (since it’s likely to be identical for large numbers of hosts). The template used in the above entry is called generic-host, and can be found near the top of the hosts.cfg file. The generic-host template entry looks like this:

	define host{
			name				 generic-host
			notifications_enabled				1
			event_handler_enabled				1
			flap_detection_enabled				1
			process_perf_data				1
			notification_interval				360
			notification_period				24x7
			notification_options				d,u,r
			contact_groups				sysstaff
			check_command				check-host-alive
			max_check_attempts				10
			retain_status_information			1
			retain_nonstatus_information		1
			register				0
			}

This one entry does all the heavy lifting for the rest of the devices that reference this template. They will all be checked using the check-host-alive check command, which is a scripted ping command. Per the notification_period key’s value, they’ll be monitored 24 hours a day, 7 days a week. The notification_options line says to send notifications if the status of the machine is either down (d), unreachable (u), or recovered (r). The flap_detection_enabled option is turned on here, as well. This is a feature of Nagios that seeks to save you from getting pages from services or hosts that change state frequently due to temporary aberrations in network connectivity, host response times, or services that are purposely restarted to pick up automated updates. You have to admit, putting all this detail into one entry is better than putting it into every host entry!

Let’s move on to services. A typical services.cfg entry looks like this:

	define service{
			use				generic-service
			host_name				ftpserver
			service_description			FTP
			is_volatile				0
			check_period				24x7
			max_check_attempts			3
			normal_check_interval		5
			retry_check_interval		1
			contact_groups				sysstaff
			notification_interval		120
			notification_period			24x7
			notification_options		w,u,c,r
			check_command				check_ftp
			}

This is the entry for my FTP server. Again, it includes only the information specific to the FTP server; all the rest of the information comes from the template named generic-service, whose settings are applied to all of the services whose entries refer to it using the use generic-service directive. Notice that I use a service-specific check command called check_ftp. The check_ftp command is just a shell script that attempts to make a connection to the FTP service on ftpserver.

You’ve no doubt noticed that both the host and service checks send mail to sysstaff if there’s a problem. But what is sysstaff? It’s actually not an email alias (although you can use one if you like). Instead, it’s configured within Nagios itself, in the contacts.cfg and contactgroups.cfg files. Let’s have a look! Here’s an entry for a contact from the contacts.cfg file:

	define contact{
			contact_name				jonesy
			alias					Jonesy
			service_notification_period			24x7
			host_notification_period			24x7
			service_notification_options		c,r
			host_notification_options			d,r
			service_notification_commands		notify-by-email
			host_notification_commands			host-notify-by-email
			email					[email protected]
			}

This is my contact entry. It says that I’m to be notified of any host or service failures 24 hours a day, 7 days a week. However, I’ve hacked my entry so that instead of being notified of every change in state, I’m only notified when services (service_notification_options) are critical (c) and when they recover (r), and when hosts (host_notification_options) are down (d) and when they recover (r). There’s an entry like this for everyone who will receive notifications about service or host status from Nagios.

Once all of the contacts are defined, you can group them together to form Nagios-specific groups in contactgroups.cfg. Here’s an example:

	define contactgroup{
			contactgroup_name		sysstaff
			alias The Systems		Guys
			members				jonesy,bill,joe
			}

That wasn’t so hard, was it? Just remember that anyone in a contact group must first be defined as a contact in contacts.cfg.

At this point you have only a very simple configuration, but it’s enough to fire up Nagios and have it monitor the hosts and services you defined and notify those who are defined as contacts. Before you do that, though, you should run the following command to do a syntax check:

	$ nagios -v /etc/nagios/nagios.cfg

This runs Nagios in “verify” mode, and we’ve fed it the main Nagios configuration file, which contains a line for every other configuration file in use. If there’s a problem, Nagios will spit out plenty of information for you to find, check out, and fix the problem. In these early stages, the most common issues will probably be related to configuration files defined in nagios.cfg that are not yet being used. For example, since we haven’t used the dependency configuration file, you’ll want to comment out any references to it in nagios.cfg.

If you received no errors, you’re in good shape. You might see “warnings” that point out possible problems to you during config verification, but in many cases these warnings are for things that are intentional, such as contacts that are not assigned to a contact group (which is not required and not always desirable). Once you’ve verified that the warnings are harmless, or fixed whatever issues existed and reverified things, you can fire up Nagios and begin receiving notifications via email about the hosts and services you’ve configured.