journalctl Filters

The great thing about the journal logging facility is that it is structured, meaning that we can filter information we are interested in easily. Anyone with Linux experience will tell you their favorite incantations to investigate logs. With filters we no longer need to rely on piping logs through commands such as grep and awk so much.

We will start with filtering the journalctl output by just kernel messages by using the –k option.

$ sudo journalctl –k
-- Logs begin at Fri 2016-11-25 22:16:00 AEDT, end at Mon 2016-11-28 22:06:08 AEDT. --
Nov 25 22:16:00 au-mel-centos-1 kernel: Initializing cgroup subsys cpuset
Nov 25 22:16:00 au-mel-centos-1 kernel: Initializing cgroup subsys cpu
...
Nov 28 21:01:01 backup kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Nov 28 22:01:01 backup kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs

Now let’s reduce that long list to only those kernel log messages that are of an ERROR priority (-p).

$ sudo journalctl -p err -k
-- Logs begin at Fri 2016-11-25 22:16:00 AEDT, end at Mon 2016-11-28 22:10:14 AEDT. --
Nov 25 22:16:13 backup kernel: CIFS VFS: Send error in SessSetup = -127
Nov 25 22:16:13 backup kernel: CIFS VFS: cifs_mount failed w/return code = -127
Nov 25 22:17:13 backup kernel: CIFS VFS: Send error in SessSetup = -13

Here we have listed all the kernel logs that are in error. There are seven priorities available to use; either the word or the number will do.

• emerg(0)— emergency

• alert(1)

• crit(2)— critical

• err(3)— error

• warning(4)

• notice(5)

• info(6)—information

• debug(7)

The –k option is the same as using the dmesg command to view logs, but as you can see, journalctl is easier to read and filter.

We can combine filters. We can see messages since the last system boot with the –b option.

$ sudo journalctl –p info –b

This can often still be a lot of messages, so we can further refine this with more specific time filters.

$ sudo journalctl -p info --since "2016-11-28 22:44:00" --until "2016-11-28 22:54:00"

Here we are displaying the logs at priority info for the last ten minutes. The times are in local time. If you want to see UTC times, you can issue the following:

$ sudo journalctl --utc

In Listing 18-2 we saw the full list of fields that are included in our logs. We can use this metadata as filters too. One of the fields from Listing 18-2 is _COMM, which records the command that was used in generating the log. We can list all the different values for that field with the following:

$ sudo journalctl –F _COMM
unix_chkpwd
request-key
freshclam-sleep
run-parts
pickup
usermod
...

Let’s say we want to search for any usermod changes that have happened since the last boot.

$ sudo journalctl _COMM=usermod -b
-- Logs begin at Fri 2016-11-25 22:16:00 AEDT, end at Mon 2016-11-28 23:09:04 AEDT. --
Nov 25 22:48:41 backup usermod[4844]: add 'jsmith' to group 'sales'
Nov 25 22:48:41 backup usermod[4844]: add 'jsmith' to shadow group 'sales'

We can also combine any of these fields with the --since and --until time filters to get a more narrow view. Having two fields listed (separated by a space) provides a logical AND listing (_COMM=usermod  _COMM=useradd). Using a + will give you a logical OR listing (_COMM=usermod + _HOSTNAME=backup), which will provide a listing with usermod and any listing with the hostname backup.

We can also list logs by their systemd unit names. Here we are going to follow (like the tail command) the httpd log using the following:

$ sudo journalctl -f -u httpd.service
-- Logs begin at Fri 2016-11-25 22:16:00 AEDT. --
Nov 28 23:27:11 backup systemd[1]: Starting The Apache HTTP Server...
Nov 28 23:27:11 backup systemd[1]: Started The Apache HTTP Server.

Here you can see the output of the Apache service as it starts up.

Securing Journald with FSS

We said earlier that we can detect whether journal logs have been tampered with. This is done with a feature called Forward Secure Sealing (FSS), which signs the logs with one of a generated key pair. A sealing keywill seal the logs at a specified interval, and the verify keycan be used to detect tampering. The logs are singed, or sealed, at regular configurable intervals. This provides some level of security for your logs.

However, it does not stop people who attack your system from covering their tracks, and they can get around this by either deleting the logs or editing between the sealing time interval. It does not provide any extra information in the event of someone tampering with your logs but will give you a timeframe in which such an event happened. It can be seen as one small piece of your overall system security.

To use FSS, you first need to enable the persistent storage of your journal log files. This is easily done by issuing the following:

$ sudo mkdir /var/log/journal
$ sudo systemctl restart systemd-journald

Then to generate the key pair, we will issue the command in Figure 18-1.

Figure 18-1. FSS key generation and QR code

As the text in Figure 18-1 says, we should store the secret verification key in a safe place, and it gives us the handy ability to use the QR code to store it on our phone. The signing key has been place in the fss file in the /var/log/journal/ e3c7f…db6ab/ directory. That will be rotated every 15 minutes.

Let’s verify the logs with our key to make sure they haven’t been tampered with.

$ sudo journalctl --verify-key 4f5f8f-9eb38b-eff95a-bc0bc8/191914-35a4e900
PASS: /var/log/journal/e3c7fd86ed8b4ef69e569a93e30db6ab/system.journal
PASS: /var/log/journal/e3c7fd86ed8b4ef69e569a93e30db6ab/user-1005.journal

Journal-Remote

Another way to secure our logs from loss or tampering is to send them to a centralized logging host as quickly as we can. Systemd Journal provides a systemd-journal-remote service that can receive journal messages from other hosts and provide a centralized logging service. This is quite new, and it currently has a few detractors since the TLS service isn’t really secure (doesn’t enforce client certificate verification), but it shows great promise as it matures.

The service can act either passively (wait for journal messages) or actively (pull messages from a remote host). It can be configured to listen over HTTP or HTTPS. We are going to set up one host, gateway.example.com, that will upload logs to our main backup.example.com server.

We are going to use https:// transport for our logs, and we will assume that we have already created the TLS keys required with our CA. We are also using two CentOS hosts in this example, but the configuration should be the same for both CentOS and Ubuntu.

First with CentOS you will need to install the systemd-journal-gateway package. On Ubuntu it is the systemd-journal-remote package. Both packages provide the systemd-journal-gateway, systemd-journal-upload, and systemd-journal-remote services.

The gateway service is an HTTP server that can be used to query journal logs. The remote service is used to receive logs from other servers. The upload, of course, is used to upload logs to a remote server. We are only going to show the upload and remote services here.

We will first set up the remote service that will listen for log messages. To do this, we need to edit the following:

$ sudo vi /etc/systemd/journal-remote.conf
[Remote]
ServerKeyFile=/etc/pki/tls/private/backup.example.com.key
ServerCertificateFile=/etc/pki/tls/certs/backup.example.com.cert
TrustedCertificateFile=/etc/pki/tls/certs/cacert.pem

This provides the details of the TLS keys and certs we need. The systemd-journal-remote user will need to be able to read the private key file. Next we need to make some additional filesystem changes.

$ sudo mkdir –p /var/log/journal/remote && sudo chown systemd-journal-remote /var/log/journal/remote

By default the remote journal service will listen on port 19532. We will need to add this to our allowed rules in our firewall.

$ sudo firewall-cmd --permanent --zone public --add-port=19532/tcp
$ sudo firewall-cmd --reload

By default the service is configured to listen with https://. This is configurable in the /usr/lib/systemd/system/systemd-journal-remote.service file. Now we can start the remote journal service with the following:

$ sudo systemctl enable systemd-journal-remote
$ sudo systemctl start systemd-journal-remote
$ sudo systemctl status systemd-journal-remote

We check the status before proceeding to make sure our service has started successfully. The next service we need to configure is on the gateway host. We need to edit the following file:

$ sudo vi /etc/systemd/journal-upload.conf
[Upload]
URL=https://backup.example.com:19532
ServerKeyFile=/etc/pki/tls/private/gateway.example.com.key
ServerCertificateFile=/etc/pki/tls/certs/gateway.example.com.cert
TrustedCertificateFile=/etc/pki/tls/certs/cacert.pem

This is similar to the remote service file with only one significant difference, as you can no doubt see. The URL option points to the backup.example.com host on port 19532. Again, the private key must be readable by the systemd-journal-upload user. We need now to make some similar filesystem changes like we did for the remote service.

First we need to give access to the state file that keeps track of what journals we have sent and give access to the journal logs that are stored in /run/log/journal.

$ sudo chown systemd-journal-upload /var/lib/systemd/journal-upload
$ sudo usermod –aG systemd-journal systemd-journal-upload

We are now ready to start our journal uploader.

$ sudo systemctl enable systemd-journal-upload
$ sudo systemctl start systemd-journal-upload

We should, very shortly, start seeing logs coming into the /var/log/journal/remote directory on the backup.example.com host.

[jsmith@backup ∼]$ sudo ls -l /var/log/journal/remote/
total 16388
-rw-r-----. 1 systemd-journal-remote systemd-journal 8388608 Nov 29 22:58 remote-
gateway@3a016bda55334bcd88d8a6fa52b1dc61-0000000000000001-0005426ea713ed3c.journal

Now we have our remote host sending logs to our backup server. In the next section we will further explore how to get the journal logs into rsyslog as an alternative.

Facilities

The facility identifies the source of the rsyslog message. Some operating system daemons and other common application daemons have standard facilities attached to them. The mail and kern facilities are two good examples, being mail-related event notification messages and all kernel-related messages, respectively.

Other processes and daemons that do not have a specified facility can use the local facilities, which range from local0 to local7. Table 18-1 lists all rsyslog facilities.

There are also two special facilities: *, a wildcard that indicates all facilities , and none, which negates a facility selection.

You can see the wildcard selector in Listing 18-4.

Listing 18-4. rsyslog.conf * Wildcard Selector

*.emerg                                :omusrmsg:*

This will send all messages of the emerg priority, regardless of facility, to everyone who is logged in.

You can use the none wildcard selector to not select messages from a particular facility. The example shown in Listing 18-5 will tell rsyslog to not log any kernel messages to the file/var/log/messages.

Listing 18-5. rsyslog.conf none Wildcard Selector

kern.none                      /var/log/messages

Priorities

Priorities are organized in an escalating scale of importance. They are debug, info, notice, warning, err, crit, alert, and emerg. Each priority selector applies to the priority stated and all higher priorities, so mail.err indicates all mail facility messages of err, crit, alert, and emerg priorities.

Like with facilities, you can use the wildcard selectors * and none. Additionally, you can use two other modifiers: = and !. The = modifier indicates that only one priority is selected; for example, cron.=crit indicates that only cron facility messages of crit priority are to be selected. The ! modifier has a negative effect; for example, cron.!crit selects all cron facility messages except those of crit or higher priority. You can also combine the two modifiers to create the opposite effect of the = modifier so that cron.!=crit selects all cron facility messages except those of critical priority. Only one priority and one priority wildcard can be listed per selector.

Actions

Actions tell rsyslogd what to do with the event notification messages it receives. Depending on the output modules loaded, rsyslog can perform several potential actions.

• Logging to a file

• Logging to a device

• Logging to a named pipe

• Logging to a specific user or the console

• Sending logs to another host

• Logging to a database table

• Executing a command

• Discarding

Listing 18-6 shows examples of the first four actions rsyslogd can take, including logging to a file, device file, named pipes, and the console or a user’s screen.

Listing 18-6. File, Device, and Named Pipe Actions

cron.err          /var/log/cron
auth.!=emerg    /dev/lpr1
news.=notice     |/tmp/pipe
auth-priv         root,jsmith

In the first line, all cron messages of err priority and higher are logged to the file /var/log/cron.

The second line has all auth messages except those of emerg priority being sent to a local printer lpr1.

The third sends all news messages of notice or greater priority to a named pipe called /tmp/pipe.

The fourth and last line sends all auth-priv messages to the users root and jsmith if they are logged in.

There is one last action you can perform, sending logs to another host, as you can see in Listing 18-7.

Listing 18-7. Logging to a Remote System

mail     @backup.example.com

In this example, all mail messages are sent to the host backup.example.com.

To send all logs, we’d use this syntax:

*.*     @backup.example.com

rsyslog uses UDP port 514 to transmit log messages . This assumes the rsyslog daemon on the remote host has been configured to receive logs and that you have suitable firewall rules in place to receive the log entries. Here’s an example:

$ sudo firewall-cmd --permanent --zone public --add-port=514/udp

Here we’ve created a firewall rule that allows the host to receive rsyslog data from the host 192.168.0.254 on UDP port 514. However, since this is UDP, remembering that UDP is a fire-and-forget protocol, there is no guarantee that the server at the other end will receive it.

To get around this and prevent possible message loss, we would use RELP, which is the reliable event logging protocol (see www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html ).

*.*    :omrelp:backup.example.com:2514

Of course, we need to load the module in the module section of our rsyslog.conf file, and we will show how to configure RELP shortly.

Combining Multiple Selectors

You can also combine multiple selectors in your rsyslog.conf file, allowing for more sophisticated selections and filtering. For example, you can list multiple facilities separated by commas in a selector (see Listing 18-8).

Listing 18-8. Multiple Facilities

auth,auth-priv.crit              /var/log/auth

This sends all auth messages and all auth-priv messages with a priority of crit or higher to the file /var/log/auth.

You cannot do this with priorities, though. If you want to list multiple priorities, you need to list multiple selectors separated by semicolons, as shown in Listing 18-9.

Listing 18-9. Multiple Priorities

auth;auth-priv.debug;auth-priv.!=emerg        /var/log/auth

This example shows you how to send all auth messages and all auth-priv messages with a priority of debug or higher, excluding auth-priv messages of emerg priority to the file /var/log/auth.

You can also use multiple lines to send messages to more than one location, as shown in Listing 18-10.

Listing 18-10. Logging to Multiple Places

auth                               /var/log/auth
auth.crit                         jsmith
auth.emerg                    /dev/console

Here all auth messages are logged to /var/log/auth as previously, but auth messages of crit or higher priority are also sent to user jsmith, if he is logged in. Those of emerg priority are also sent to the console.

It is also common to use the omfwd output module, which allows you to send logs to a remote server via TCP or UDP. If we wanted to send log messages to a local file and to a remote server, we could use something like this:

mail.*    action(type="omfile" sync="no" file="/var/log/maillog”)
                   action(type="omfwd" Target="monitor.example.com" Port="10514" Protocol="tcp")

You can read more about this module at www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html .

Configuring RELP Client

We will start with the RELP client configuration. We will assume that the hosts already have a TLS key and public certificate that has been signed by our private CA. We will again use the gateway host, and it will connect and send its logs to the backup.example.com host.

$ sudo vi /etc/rsyslog.d/relp.conf
# make gtls driver the default
$DefaultNetstreamDriver gtls

# certificate files
$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/cacert.pem
$DefaultNetstreamDriverCertFile /etc/pki/tls/certs/gateway.example.com.cert
$DefaultNetstreamDriverKeyFile /etc/pki/tls/private/gateway.example.com.key

$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer backup.example.com
$ActionSendStreamDriverMode 1
*.* @@backup.example.com:6514

We have created a file called relp.conf and have configured it to use an encrypted transport between the client and the backup.example.com host. We do this via a NetStreamDriver called gtls, which implements our TLS transport (GnuTLS). A NetStreamDriver can provide sequenced delivery, authentication, and secure transport.

We need to provide our TLS certificate files, and they are described as shown previously. Remember that on Ubuntu the TLS files will have a different path.

We handle authentications via our certificate names (x509/name). That means we verify our connection to our peer by testing their common name in their TLS certificate. If it matches the peer backup.example.com, then we will permit the connection; if not, we will fail it. The mode 1 signifies that we will use TLS.

The last line says that we will ship all facilities and priorities (*.*) to backup.example.com on port 6514. You will now restart the rsyslog daemon .

$ sudo systemctl restart rsyslog

Configuring RELP Server

Now we can configure our RELP server. This is going to collect our logs from our network and store them. The configuration is similar to the client, and our backup host looks like this:

$ sudo vi /etc/rsyslog.d/relp.conf
$ModLoad imtcp

$DefaultNetstreamDriver gtls

$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/cacert.pem
$DefaultNetstreamDriverCertFile /etc/pki/tls/certs/backup.example.com.cert
$DefaultNetstreamDriverKeyFile /etc/pki/tls/private/backup.example.com.key

$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverPermittedPeer *.example.com
$InputTCPServerStreamDriverMode 1
$InputTCPServerRun 6514

First we will need to load our imtcp module, which will provide TCP connections. Then we have the same configuration options as we had in the client for our TLS keys except we change the names. We accept all peers that are signed by our CA with example.com in their common name. The last line tells rsyslogd to run a TCP server on port 6514.

Now we will need to make sure that port 6514 is open on our backup server, which is a CentOS server, so we use the firewall-cmd command.

$ sudo firewall-cmd --permanent --zone public --add-port=6514/tcp
$ sudo firewall-cmd --reload

Then we restart the rsyslog daemon on the backup server too.

$ sudo systemctl restart rsyslog

On the backup server we can now inspect our /var/log/messages file, and we should see logs similar to these:

Dec  1 12:01:17 gateway chronyd[608]: Selected source 27.124.125.250
Dec  1 12:01:19 gateway chronyd[608]: Selected source 27.124.125.250
Dec  1 12:01:21 backup systemd: Starting user-0.slice.
Dec  1 12:01:01 backup systemd: Started Session 143 of user root.
Dec  1 12:05:51 gateway chronyd[608]: Selected source 202.127.210.37
Dec  1 12:08:22 dc1 systemd[1]: Started CUPS Scheduler.

The logs are coming in with the timestamp, hostname, service, and message being written. There we can see that the host gateway, the host dc1 and the backup server are all now logging to our backup server .

Installing and Configuring Beats

Let’s take a look now at the first step of the path of our logs. With Beats, particularly Filebeat, we are going to gather data from our host and ship it to a Logstash server. In this scenario, the Filebeat will run on our gateway host and send its data to our monitor server. The monitor server we talked about in Chapter 17 had our Graphite service set up and was used to store the metrics we collected from our hosts via Collectd.

For simplicity we are going to run the Logstash service and Elasticsearch on our monitor host. In a real-world scenario you would have at least three Elasticsearch nodes in a cluster, depending on the amount of transforming and data collection, and you may have a few tiers of Logtash services that would run on one or two other separate nodes.

On the gateway node, we will install Filebeat. We can download Filebeat from https://www.elastic.co/downloads/beats/filebeat , and we can choose the most appropriate package format (RPM, DEB, tar.gz). The gateway host happens to be a CentOS host, so we will install an RPM version.

$ sudo yum install -y https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.2-x86_64.rpm

For both Ubuntu and CentOS, the configuration file for Filebeat is stored in /etc/filebeat/filebeat.yml. We are going to send all our logs from /var/log to our Logstash server that will be on the monitor server. We do that with the configuration shown in Listing 18-14.

Listing 18-14. Filebeat.yml

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/messages
    - /var/log/*.log
    - /var/log/audit/audit.log
tags: ["security", "network"]
fields:
  env: production
output.logstash:
  hosts: ["monitor.example.com:5044"]
  ssl.certificate_authorities: ["/etc/pki/tls/certs/cacert.pem"]
  ssl.certificate: "/etc/pki/tls/certs/gateway.example.com.cert"
  ssl.key: "/etc/pki/tls/private/gateway.example.com.key"

It is common to send data to a local Logstash service instead of a remote one as we have done here. This way you can locally transform the data prior to sending across the network. You may want to do this if you require only anonymized data to traverse the network, or you may want to use the distributed processing power of your hosts rather than relying on a centralized Logstash service to transform your data.

The format of Filebeat configuration file is YAML (see the “YAML Format” sidebar for more information). In Listing 18-14 the first line declares our filebeat.prospectors. You can have different types of input, and the prospectors file is where you declare them. In Listing 18-14 you can see we declare an input_type value of log. The other alternative is an input type of stdin. We can then list the paths that we want to ingest our logs from. Of course, we would like to gather logs from /var/log/messages. You can use globs to catch all logs (/var/log/*.log), or you can target specific logs (/var/log/audit/audit.log). Here we are capturing any in the /var/log directory (excluding any .gz or -20161131 rotated logs), and we are not capturing anything in /var/log/*/*.log except for those we have specified (audit.log).

---
key: value
- lista
- listb
keya:
  - valuea
  - valueb

If we wanted a different to capture our Apache logs, we would do that separately like this:

- input_type: log
  paths:
    - /var/log/*.log
- input_type: log
  paths:
    - /var/log/apache/httpd-*.log
  document_type: apache

We don’t run an Apache server on our gateway, but here you can see that we would target the Apache logs as a different input type, and we would apply a document_type of apache. This changes the event type field, one of the metadata tags we apply to our logs, to apache instead of log. This helps us with transformations further down the processing chain.

In Listing 18-14 you will also see that there are global directives we apply to all our data. We tag our data with security and network, and we give it an environment tag of production. This again adds richness to the data we can later use to extract and transform and filter our data.

Our output is going to our Logstash server on the monitor.example.com host. We are going to use TLS-encrypted transport to send our data. We are not anonymizing our logs here, so anything that is sent to the monitoring host can be read in clear text, making us vulnerable to snooping. Encrypting the data prevents this.

There are more configuration options that you can use on your Filebeat prospectors. You can include or exclude specific lines, exclude files, add specific tags, add specific fields, and add multiline pattern matching. You can read more about these here:

• https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html

We will not start our Filebeat service straightaway, but you can start and stop it using systemctl as you would normally.

$ sudo systemctl enable filebeat
$ sudo systemctl start filebeat

Before we start our Filebeat service, we will configure our Logstash service to accept the logs.

Installing and Configuring Logstash

Logstash is a Java process that is used to transform data prior to “stashing” the output in some kind of storage for further analysis or viewing. It is very powerful, and you can do a lot with it. This chapter will not do it justice as we are only able to show the barest essentials of its power. In this section we are going to show you how to install and then do a basic configuration so that we can output our log data to Elasticsearch.

The current version of Logstash requires Java 8. We are going to be installing this service on our Ubuntu monitoring host via the APT repository provided at https://www.elastic.co , a company that helps design and support the open source projects of Kibana, Elasticsearch, Logstash, and Beats.

Let’s first add the APT repository by first adding the public GPG key, then adding the package apt-transport-https (if it is not already installed), and finally adding the actual repository to APT.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo aptitude install –y apt-transport-https
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
sudo aptitude update

Here we have used the aptitude package manager, which we have installed rather than apt-get, which you can also use. Now we can install Logstash.

$ sudo aptitude install –y logstash

For CentOS and the latest installation notes, you can see the instructions here:

• https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

The configuration files for Logstash are kept in /etc/logstash. Mostly you do not need to change any configuration on how the service starts, but if you want, you can do so with the /etc/logstash/startup.options file. That file contains the user that runs the service, the JAVA_OPTS you may like to include, and so on.

We are now going to create a pipeline for processing our logs. We can do this in two ways. We can edit and add to the /etc/logstash/logstash.yml file, which is the main pipeline configuration file, or we can create a file in /etc/logstash/conf.d/, which will be read in by the Logstash service. Let’s see how we can collect our log file from the gateway host and any other Beats service.

$ sudo vi /etc/logstash/conf.d/general.conf
input {
  beats {
    port => 5044
    ssl  => true
    ssl_certificate => “/etc/ssl/certs/monitor.example.com.cert”
    ssl_key => “/etc/ssl/private/monitor.example.com.key”
    ssl_certificate_authorities [ “/etc/ssl/certs/cacert.pem” ]
    ssl_verify_mode => force_peer
  }
}
output {
  stdout { codec => rubydebug }
}

Taking a look at this file, you can see that it is made of two sections: an input and an output. Typically you will see three sections including a filter section.

Input {
  ...
}
filter {
  ...
}
output {
  ...
}

Those of you familiar with Ruby will recognize this as Ruby hash syntax . JRuby, for Java Ruby, is used with Logstash, and the configuration files are in native Ruby syntax .

In our input section, we have included our beats plug-in, and the beats plug-in accepts the port and SSL configuration options like earlier. This should already be familiar to you, but the important thing to note is that here we are specifying a ssl_verify_mode value of force_peer. This means that if the client does not provide the server with a certificate, we will drop the connection immediately. This will make our system more secure and drop unauthorized connections early.

The output section describes where we will send the data after we have processed it. We can see that we will output the data to stdout, and to help us debug that output, we will use a codec called rubydebug. This is good for viewing what Logstash is doing while we begin our journey. We will also specify elasticsearch as an output, but we don’t want to do so at this stage.

We are ready to start our Logstash service and start seeing how it works. We do that via the following command:

$ sudo -u logstash /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/general.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
04:20:40.036 [[main]-pipeline-manager] INFO  logstash.inputs.beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
04:20:40.072 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
04:20:40.075 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
04:20:40.087 [[main]<beats] INFO  org.logstash.beats.Server - Starting server on port: 5044
04:20:40.134 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}

In the first section we will see the startup output giving us details on what is being started with what parameters. You will see that we are starting the Beats input on port 5044, the main pipeline, and a web API on port 9600.

Now, on the gateway host we can start our filebeat service. We do that with the systemctl command like so, and then we can tail the logs that are produced by the service.

$ sudo systemctl start filebeat && tail –f /var/log/filebeat/filebeat

Now on the monitor host , we can start the logstash service, and we should see a rapid amount of data coming in from our Filebeat on the gateway server.

In Listing 18-15 we see the one of the captured log messages from the gateway host. Each Beat is given its own timestamp, and we are given a bunch of other related metadata like input_type, the host it can from, tags we have associated with the Beat, and so on. Each of these Beats will have a host, a source, and a message. The message will have the rsyslog format we have already seen.

Listing 18-15. Beat from Gateway Server Logs

{
    "@timestamp" => 2016-12-04T06:33:33.868Z,
    "offset" => 11979,
    "@version" => "1",
    "input_type" => "log",
    "beat" => {
        "hostname" => "gateway.example.com",
            "name" => "gateway.example.com",
         "version" => "5.0.2"
     },
     "host" => "gateway.example.com",
     "source" => "/var/log/messages",
      "message" => "Dec  4 06:33:24 gateway jsmith: tesing this is a test",
      "fields" => {
           "env" => "production"
       },
       "type" => "log",
       "tags" => [
          [0] "security",
          [1] "network",
          [2] "beats_input_codec_plain_applied"
       ]
}

Let’s briefly take a look at the audit log we are collecting too:

         "host" => "gateway.example.com",
         "source" => "/var/log/audit/audit.log",
         "message" => "type=SERVICE_STOP msg=audit(1480834167.796:997): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'",
         "type" => "log",

Immediately you will notice that audit.log is not is the same format as your normal rsyslog. The auditd file is a set of key/value pairs (key=value). They can appear very different from what we see here, but basically they consist of a type and a message (which includes a timestamp and unique ID). Then, depending on the type, they can have any number of other keys and values. In this example we have stopped the openvpn service, and this is the resultant log notification.

This is not in the format that we would like and is not like an rsyslog message. Let’s change that so that we record this as a different type of event. To do that, we will edit the Filebeat configuration on the gateway host and change the following:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/messages
    - /var/log/*.log
- input_type: log
  paths:
    - /var/log/audit/audit.log
  document_type: auditd

We have moved the audit.log into its own input_type section. To that section we have added the document_type option and set it to auditd. Now let’s show you what that does; go ahead and reload the Filebeat service .

    "host" => "gateway.example.com",
    "source" => "/var/log/audit/audit.log",
    "message" => "type=SERVICE_STOP msg=...terminal=? res=success'",
    "fields" => {
        "env" => "production"
    },
    "type" => "auditd",

Now when we get the same message from stopping the openvpn service on the gateway host, the type is set to auditd. We can now use this in our filter section to make the audit log easier to further process.

Logstash Filters

Logstash filters are a way of parsing and transforming data to make it easier to discover what’s in your logs. We are going to take our auditd log and extract information and add it to our event data.

When we look at a raw audit.log file, we see information like this:

type=SERVICE_STOP msg=audit(1480844911.323:1080): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=openvpn@gateway comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

These are key=value logs. You can see type=SERVICE_STOP, uid=0, and msg='...' are all key/value pairs. Logstash can understand how to deal with key=values. We do this by telling the filter to take each key and value and assign it. You can also see that there we have an audit(1480844911.323:1080), which is the timestamp (1480844911.323, in Unix epoch time) plus a unique ID (1080) that marks the event.

So, how do we tell Logstash to work on these events? In our Beat configuration we marked audit.logs with the document_type of auditd. We saw that the events coming into Logstash from the Filebeat now have the type auditd attached to them. We can now match this value and work on the logs that specifically have that data. We do that with a conditional if statement in our filter section.

filter {
  if [type] == "auditd" {
    # audit.log get matched and worked on here
  }
}

Here we are using a conditional if { ... } statement that matches on the tag [type] if it equals the string "auditd". Now that we are working on the right logs coming through, we can now tell Logstash to create key/value pairs from what it sees. We do this with the following lines:

filter {
  if [type] =∼ "auditd" {
    kv { }
  }
}

This simply will create more labels in our metadata on which we can further work on and use for discovery. Let’s take a look at what this does to our Logstash data:

{
    "msg" => [
        [0] "audit(1480844911.323:1080):",
        [1] "unit=openvpn@gateway comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success"
    ],
    "uid" => "0",
    "ses" => "4294967295",
    "auid" => "4294967295",
    "pid" => "1",
    "source" => "/var/log/audit/audit.log",
    "message" => "type=SERVICE_STOP msg=audit(1480844911.323:1080): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=openvpn@gateway comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'",
    "type" => "SERVICE_STOP",
    "subj" => "system_u:system_r:init_t:s0"
    "tags" => [
        [0] "security",
        [1] "network",
        [2] "beats_input_codec_plain_applied"
    ],
    "offset" => 5738705,
    "input_type" => "log",
    "@timestamp" => 2016-12-04T09:48:31.650Z,
    "@version" => "1",
    "beat" => {
        "hostname" => "gateway.example.com",
        "name" => "gateway.example.com",
        "version" => "5.0.2"
    },
    "host" => "gateway.example.com",
    "fields" => {
        "env" => "production"
    },
}

Now you can see that we are adding further texture to our data by labeling it with more usable data. We have ordered the output to be more human readable, but you can see that the first few lines of the output are the key/value pairs from the message contents. We can make it more readable too if we give greater detail to some of those labels. We do this with the mutate and rename functions .

filter {
  if [type] =∼ "auditd" {
    kv { }
    mutate {
      rename => {
        "type"        => "audit_type"
        "auid"        => "audit_uid"
        "fsuid"       => "audit_fs_uid"
        "suid"        => "audit_set_uid"
        “subj”        => “audit_subject”
        "ses"         => "session_id"
        “hostname => “audit_hostname”
      }
    }
  }
}

Now when we look at our Logstash data, it will look similar to this:

{
...
    "audit_uid"       => "4294967295",
    "audit_subject" => "system_u:system_r:init_t:s0",
    "audit_type"     => "SERVICE_STOP"
    "session_id"     => "4294967295",
...
}

The mutate function has changed the state of the log information. It has allowed us to take certain fields in the log message and rename them to clearer labels.

Lastly, if we look at the timestamp in the log message it, it doesn’t get extracted. We are going to make sure that the data ends up as a label as well. To do that, we use a function called grok. A grok function is made up of two parts, the syntax and the semantic, and is written like this:

%{SYNTAX:SEMANTIC}

The SYNTAX is a name of a pattern that matches your text. There are quite a few that come with Logstash, and you can view them all here:

• https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

The patterns are regular expressions that map to a name. The timestamp we are dealing with is in epoch or Unix epoch and is the number of seconds since 1970. Since it is a number, we can use the native Logstash NUMBER pattern to match it.

The SEMANTIC is just the identifier we will give it so that we add greater value to our metadata labels. We will give it audit_epoch.

As we have said, there are two parts to the timestamp audit(1480844911.323:1080): the time and the unique identifier. We will use grokto search for the string containing the epoch and the unique ID and extract them as labels for us.

    grok {
      match => { "msg" => "audit(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}):" }
    }

Here we are telling grok to match the msg key and find the audit(<somenumber>:<somenumber): string. For the first match we will give it the name audit_epoch, and for the second we will give it audit_counter. Now when we run our Logstash service again, we will see the following appear:

{
...
    "offset" => 5745528,
    "audit_epoch" => "1480846476.689",
   "audit_counter" => "1106",
...
}

We now have our audit log being successfully transformed and adding extra labels to our Logstash data. We can now begin to add that into Elasticsearch.

Elasticsearch Installation and Configuration

Elasticsearch installations require a minimum of 4GB of RAM to run the service and will fail to start without it. The installation is simple once we have the APT repository configured provided by Elastic Co. (the one we configured in the installation of Logstash).

Not surprisingly, the installation is simply using the aptitude package manager:

$ sudo aptitude install –y elasticsearch

In our one server scenario, we do not need much extra configuration. We can start the Elasticsearch service now, and we can start storing our logs in it. However, if we want to create an Elasticsearch cluster, we will need to edit the /etc/elasticsearch/elasticsearch.yml file.

To create an Elasticsearch cluster , you need to supply the cluster name and the node name. Let’s edit the file like so:

cluster.name: monitor
node.name: monitor-node-1

Also within the elasticsearch.yml file, we can set the data path where we store data, the log directory, network options, and so on.

For service discovery, or how Elasticsearch finds other nodes in the cluster, Elasticsearch has a Zen Discovery module. The Zen Discovery module is used to discover other cluster nodes, perform master elections, detect faults, and notify of cluster updates. It does this by sending either network pings to nodes or unicast packets that participating hosts can react to.

Since we are not setting up a cluster, we can just start the service now without changing the configuration.

$ sudo systemctl start elasticsearch

Of course, we can always use the status subcommand of systemctl to check the status of our Elasticsearch service, but we can also issue the following command from the command line:

$ curl http://localhost:9200
{
  "name" : "7mf_JBi",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "69nMGfoYRYiwatKO6Nj1BA",
  "version" : {
    "number" : "5.0.2",
    "build_hash" : "f6b4951",
    "build_date" : "2016-11-24T10:07:18.101Z",
    "build_snapshot" : false,
    "lucene_version" : "6.2.1"
  },
  "tagline" : "You Know, for Search"
}

This curl command to port 9200 shows that our Elasticsearch server is responding. The response is in JSON format. We can query Elasticsearch further with something like this:

curl -H "Content-Type: application/json" "http://localhost:9200/logstash-*/_search" -d '{
   "query": {
     "match": {
       "event_type": "SERVICE_STOP"
     }
   },
   "sort": [ "_doc" ]
 }'

This curl command now queries the _search URI on Elasticsearch to check whether we have any matches for audit_type "SERVICE_STOP". The –d option for curl allows us to send a data POST, which we send as a JSON-formatted query.

The result of this query can look like this:

{
  "took":2,
  "timed_out":false,
  "_shards": {
     "total":5,
     "successful":5,
     "failed":0
   },
  "hits":  {
    "total":74,
    "max_score":null,
    "hits":[...the events...]
  }
}

The first section tells us how long it took to run the query and that we successfully queried five shards. In the hits section, we have a total of 74 hits for SERVICE_STOP audit_type. There will be list of events that match printed also. So, now we can see that our Logstash service is sending data in from our gateway server as we expected.

Rather than using curl commands to search your logs, there is another open source tool that can help visualize our logs, Kibana. We will now show you how to install and configure it.

Installing Nagios on CentOS

On CentOS , we can install our Nagios package with the following:

$ sudo yum install nagios nagios-plugins nagios-plugins-ssh nagios-plugins-smtp nagios-plugins-bacula
nagios-plugins-disk nagios-plugins-fping nagios-plugins-http nagios-plugins-ldap nagios-plugins-mysql

We have chosen a selection of plug-ins to install as well. While we won’t show you how to use each plug-in, you can see that there are plug-ins available for most of the services we have shown in this book.

Installing Nagios on Ubuntu

On Ubuntu, the default version of Ubuntu is version 3. If we want to install version 4, we will have to install from source.

$ sudo apt-get install nagios3 nagios-plugins

This will install Nagios version 3 and all the required supporting packages including the Nagios plug-ins.

If you want to install from source (for either CentOS or Ubuntu, but this is for Ubuntu), you need to do the following.

First we need to create users and groups to run and manage the Nagios service.

$ sudo useradd nagios  && 
sudo groupadd nagcmd && 
sudo usermod -aG nagcmd www-data && 
sudo usermod –aG nagcmd nagios

We have added the nagcmd group to the www-data and nagios users so that the web service can read the Nagios results. Now we need to download the source, untar it, and then enter the directory.

$ wget https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.2.3.tar.gz && 
tar zxf nagios-4.2.3.tar.gz && 
cd nagios-4.2.3/

If you remember when we compiled Nginx in Chapter 8, we will do something similar now to Nagios. We first have to configure any special requirements for our installation, then compile any binaries, and finally install Nagios onto our system.

$ ./configure --with-httpd-conf=/etc/apache2/conf-available 
  --with-nagios-user=nagios 
  --with-nagios-group=nagcmd 
  --sysconfdir=/etc/nagios

Here we have configured our Nagios service to place the Apache configuration into the appropriate place (you may like to choose /etc/apache2/sites-available instead here). We have declared the user and group, and we have placed our --sysconfdir (system configuration directory) in /etc/nagios, which is common in CentOS (in Ubuntu it is normally /etc/nagios<version  number>, but we are not choosing that here).

Now we need to compile the source code and then install the various components.

$ sudo make all
$ sudo bash -c 'make install
make install-init
make install-config
make install-commandmode
make install-webconf'

The configuration files are installed in /etc/nagios, and the Nagios binary is installed in /usr/local/nagios/bin/nagios. If you want to install the latest plug-ins, you can do so like this:

$ wget https://nagios-plugins.org/download/nagios-plugins-2.1.4.tar.gz && 
tar zxf nagios-plugins-2.1.4.tar.gz && 
cd nagios-plugins-2.1.4/ && 
./configure --with-nagios-user=nagios --with-nagios-group=nagcmd && 
make && sudo make install

This will install the plug-ins into the /usr/local/nagios/libexec/ directory. The last thing we need to do is make sure we have a systemd service file to start the service.

$ sudo vi /etc/systemd/system/nagios.service
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/nagios
Description=LSB: nagios host/service/network monitoring and management system
Before=multi-user.target
Before=multi-user.target
Before=multi-user.target
Before=graphical.target
Before=shutdown.target
After=local-fs.target
After=remote-fs.target
After=systemd-journald-dev-log.socket
After=nss-lookup.target
After=network-online.target
After=time-sync.target
Wants=network-online.target
Conflicts=shutdown.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/nagios start
ExecStop=/etc/init.d/nagios stop
ExecReload=/etc/init.d/nagios reload

Because there is no native systemd service file available for Ubuntu just yet, the Nagios installation will deploy the /etc/init.d/nagios file, and we just have to point this service file at it (we spoke about Ubuntu’s adoption of systemd in Chapter 6 and how it still uses some LSB init files to start services).

The nagios.cfg File

The nagios.cfg configuration file contains the base configuration for your Nagios installation. Each option in this file is in the form of an option-value pair. For example, the location of the Nagios log file is specified using the log_file option; on CentOS, this would be done as follows:

log_file=/var/log/nagios/nagios.log

This is usually the first option in your nagios.cfg file followed by the cfg_file and cfg_dir options that specify the location of your object configuration files. The cfg_file option allows you to specify an individual file that contains the Nagios object configuration. Here’s an example:

# Definitions for monitoring the local (Linux) host
cfg_file=/etc/nagios/objects/localhost.cfg

You can specify multiple files; indeed, many people specify each object type in a separate file to organize them.

cfg_file=/etc/nagios/objects/commands.cfg
cfg_file=/etc/nagios/objects/contacts.cfg
cfg_file=/etc/nagios/objects/timeperiods.cfg
cfg_file=/etc/nagios/objects/templates.cfg
...

Further in the configuration file you will find the cfg_dir option, which specifies a directory. Nagios will load any file in this directory with a suffix of .cfg. Here’s an example:

cfg_dir=/etc/nagios/conf.d

The nagios.cfg file contains a number of other useful options, some of which you can see in Table 18-4.

You can also turn on and off checking of hosts and services and the sending of alerts at a global level on the nagios.cfg configuration file. For a full list of the available options, see https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/configmain.html .

Host Configuration

Let’s start examining Nagios’s configuration by opening CentOS’s /etc/nagios/objects/localhost.cfg configuration file and looking at its contents, starting with a host object definition.

We’re going to start with the host object definition in the file that you can see in Listing 18-16.

Listing 18-16. A Host Object

define host {
        use                   linux-server    ; Name of host template to use
                                                       ; This host definition will inherit all variables that are defined
                                                       ; in (or inherited by) the linux-server host template definition.
        host_name       localhost
        alias                 localhost
        address            127.0.0.1
        }

You can see that an object definition starts with define, the type of object to define (in our case a host object), and the definition, which is enclosed in the { } curly braces. Inside the definition are the attributes of the object defined by a series of key/value statements, separated by spaces. Our host object definition has four attributes: use, host_name, alias, and address.

The use attribute tells our host object to refer to a template. A template is a technique Nagios uses to populate an object definition with values that might be the same across many objects. For example, host objects will share many of the same attributes and characteristics. Rather than specify every single attribute in each host object definition, you can instead refer to a template. Nagios then creates the host object with all the attributes in the host definition plus those attributes in the template. We’ll look at the additional attributes defined in the linux-server template in Listing 18-17 in a moment.

In this case, the rest of the attributes of our host object define its identity. The host_name attribute defines the name of the host object. This name must be unique. You can have only one host object called localhost or headoffice.example.com. Nagios also makes the host_name attribute available as a macro called $HOSTNAME$.

The alias attribute is another name for the object; in this case, we’ve used an alias of localhost. This alias is usually used as a longer description of the host and is also available as a macro called $HOSTALIAS$.

The last attribute, address, provides the IP address of the host; in this case we’re monitoring our local host, 127.0.0.1. This IP address must be contactable by Nagios to allow monitoring to take place. It is also available as the macro $HOSTADDRESS$.

define host {
        name                           linux-server
        use                              generic-host
        check_period               24x7
        check_interval             5
        retry_interval               1
        max_check_attempts  10
        check_command         check-host-alive
        notification_period       workhours
        notification_interval     120
        notification_options     d,u,r
        contact_groups           admins
        register                       0
        }

dns-servers <- critical-servers <- linux-servers <- generic-host

define timeperiod{
          timeperiod_name       24x7
          alias                 24 Hours A Day, 7 Days A Week
          Sunday                00:00-24:00
          monday                00:00-24:00
          tuesday               00:00-24:00
          wednesday             00:00-24:00
          Thursday              00:00-24:00
          friday                00:00-24:00
          saturday              00:00-24:00
          }

Sunday    00:00-02:00,17:00-19:00

define command{
          command_name    check-host-alive
          command_line    $USER1$/check_ping -H $HOSTADDRESS$ -w 3000.0,80% -c 5000.0,100% -p 1
          }

$USER1$/check_ping -H $HOSTADDRESS$ -w 3000.0,80% -c 5000.0,100% -p 1

$USER1$=/usr/lib64/nagios/plugins

define timeperiod{
          timeperiod_name      workhours
          alias                "Normal" Working Hours
          monday               09:00-17:00
          tuesday              09:00-17:00
          wednesday            09:00-17:00
          thursday             09:00-17:00
          Friday               09:00-17:00
          }

define contactgroup{
          contactgroup_name       admins
          alias                   Nagios Administrators
          members                 nagios-admin
          }

define contact{
         contact_name                             nagios-admin
         alias                                            Nagios Admin
         service_notification_period         24x7
         host_notification_period            24x7
         service_notification_options       w,u,c,r
         host_notification_options            d,r
         service_notification_commands  notify-by-email
         host_notification_commands       host-notify-by-email
         email                                           nagios-admin@localhost
         }

define command{
          command_name      host-notify-by-email
           command_line    /usr/bin/printf "%b" "***** Nagios *****

Notification Type: $NOTIFICATIONTYPE$
Host: $HOSTNAME$
State: $HOSTSTATE$
Address: $HOSTADDRESS$
Info: $HOSTOUTPUT$

Date/Time: $LONGDATETIME$
" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
   }

***** Nagios *****

Notification Type: PROBLEM
Host: gateway.example.com
State: DOWN

Address: 192.168.0.254
Info: PING CRITICAL - Packet loss = 100%

Date/Time: Fri Feb 13 00:30:28 EST 2009

define host{
          name                                   generic-host
          notifications_enabled              1
          event_handler_enabled         1
          flap_detection_enabled           1
          failure_prediction_enabled    1
          process_perf_data                  1
          retain_status_information        1
          retain_nonstatus_information  1
          notification_period                   24x7
          register                              0
          }

define host{
          use                      linux-server
          host_name          gateway.example.com
          alias                    gateway.example.com
          address               192.168.0.254
          }

Service Configuration

Now that you know something about host objects, we’re going to examine a service object. Services are defined using the service-type object and are linked to their underlying host. For example, based on our existing configuration examples, Listing 18-19 shows a service that checks the disk space of our root partition.

Listing 18-19. A Service Definition

define service{
          use                                local-service
          host_name                    localhost
          service_description       Root Partition
          check_command           check_local_disk!20%!10%!/
          }

Our service definition is simple. The use attribute specifies a template our service is going to use. The host_name attribute specifies what host the service runs on, in our case localhost. The service_description describes the service. Lastly, the check_commandattribute specifies the command that the service uses to check the status of whatever is being monitored. This check_command is slightly different; after the command we want to use, you can see a string.

!20%!10%!/

This string consists of variables we’re passing to the command definition, with each variable value prefixed with an exclamation mark (!). So here we are passing the values 20%, 10%, and / to the command. This allows us to reuse a command for multiple services, as you’ll see in a moment.

Let’s take a quick look at the check_local_disk command:

define command{
          command_name    check_local_disk
          command_line       $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
          }

Like our previous command, with command_line we’ve specified the $USER1$ macro to give us the path to the plug-in being executed. That plug-in is check_disk, which checks the status of a local disk.

You can also see the -w and -c options—which we told you earlier set the threshold values for the WARNING and CRITICAL statuses. Lastly, we have the -p option, which specifies the disk partition we’re monitoring. In this command, however, the value of each option is $ARGx$: $ARG1$, $ARG2$, and $ARG3$, respectively. Each of these arguments represents one of the arguments we passed in our check_command attribute in Listing 18-19, so our command_line command in fact looks like the following:

command_line    $USER1$/check_disk -w 20% -c 10% -p /

This results in a WARNING status being generated when only 20 percent of disk space is available and a CRITICAL status being generated when 10 percent of disk space is free, with both statuses applying to the root filesystem, or /.

To create a service that monitors disks on another partition, for example, /var, we would create a service like the following:

define service{
          use                            local-service
          host_name                localhost
          service_description   Var Partition
          check_command       check_local_disk!20%!10%!/var
          }

Before we discuss some other services, let’s take a quick look at the local-servicetemplate our service is using:

define service{
        name                                     local-service
        use                                        generic-service
        max_check_attempts            4
        normal_check_interval         5
        retry_check_interval              1
        register                                  0
        }

The service template, local-service, is similar to previous templates you’ve seen but with some additional attributes. The first of these new attributes, normal_check_interval, specifies how often Nagios should check that the service is OK, in this case every 5 minutes. The second new attribute, retry_check_interval, is related. If, when checking the service, Nagios discovers that the service is not OK, it retries the check the number of times specified in the max_check_attempts attribute. This is done before it marks the service as not OK. During this retry period, instead of checking once every 5 minutes as specified in normal_check_interval, the check is made every 1 minute as specified in retry_check_interval.

Note that this template also has a parent template, generic-service, which we’re not going to discuss in detail. Suffice to say that options like notification_options, notification_period, contact_groups, and check_period are defined there and can be overridden in downstream templates.  The options used in that template are explained at https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/objectdefinitions.html - service.

Let’s look at another service definition, this one to monitor a network-based service, in Listing 18-20.

Listing 18-20. A Network-Based Service Definition

define service{
          use                               local-service   ; Name of service template to use
          host_name                    gateway.example.com
          service_description       Check SMTP
          check_command           check_smtp!25
          }

In Listing 18-20, we have a new service, called Check SMTP, that uses our local-service template and a check_command of check_smtp!25. This passes the value 25 to a command called check_smtp. Let’s look at that command now:

define command{
         command_name    check_smtp
         command_line    $USER1$/check_smtp -H $HOSTADDRESS$ -p $ARG1$
         }

Here we have a command that runs a plug-in called check_smtp. It accepts the $HOSTADDRESS$ macro, which is the IP address of the SMTP server we want to check. The -p option specifies the port (the plug-in defaults to port 25), and we pass in this value as the $ARG1$ macro.

You can see a service alert generated by this service from the nagios.log log file here:

[1481325559] SERVICE ALERT: gateway.example.com;Check SMTP;CRITICAL;HARD;4;
CRITICAL - Socket timeout after 10 seconds

The nagios.log entry specifies the Unix epoch time (1481325559, or Fri Dec 9 23:19:19 UTC 2016), the type of alert, the host and service, and the nature of the alert including the output from the plug-in.

Simple Remote Monitoring

You can create a variety of network-based services using Nagios plug-ins, but what if you want to monitor services that aren’t network facing or on the local host? One of the methods to do this is a special plug-in called check_by_ssh (for others, see the “Remote Monitoring” sidebar).

The check_by_ssh plug-in uses SSH to connect to a remote host and execute a command. So to make use of the plug-in, you have an SSH daemon running on the remote host, and any intervening firewalls have to allow SSH access to and from the host.

You also need to use key-based authentication between the hosts because Nagios has no capability to input a password when checking the service. So we’re going to start by creating a key to use between our Nagios server and the remote host.

To create this key, we should be the user who runs Nagios, usually nagios. We can do this using the sudo command to execute the ssh-keygen command from the Nagios home directory of /var/spool/nagios.

# sudo -u nagios ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/var/spool/nagios/.ssh/id_rsa):
Created directory '/var/spool/nagios/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/spool/nagios/.ssh/id_rsa.
Your public key has been saved in /var/spool/nagios/.ssh/id_rsa.pub.
The key fingerprint is:
df:fc:d6:d2:50:66:65:51:79:d1:f8:56:a1:78:a4:df [email protected]

We use the -t rsa option to create an RSA key. We are prompted to enter a location for the key, usually in the .ssh directory under the home directory of the nagios user—in this case, /var/spool/nagios/.ssh. The private key is in the id_rsa file, and the public key is in the id_rsa.pub file. Instead of entering a passphrase for the key, we press Enter to specify an empty passphrase, because we need the connection to be made without a passphrase or password prompt.

We then need to copy the public key, id_rsa.pub, to the remote host and store it in the authorized_keys file for the user we’re going to connect to. If you are following along with this example, you should create a user on the remote host and specify a password for it. In our case, we do so on the remote gateway.example.com host as follows:

gateway$ sudo useradd nagios

We also need to create the .ssh directory on the remote host and protect it.

gateway$ sudo mkdir /home/nagios/.ssh
gateway$ sudo chmod 0700 /home/nagios/.ssh

We can then copy the file, assuming jsmith has the appropriate access to the gateway server.

nagios$ scp .ssh/id_rsa.pub [email protected]:/tmp/authorized_keys
nagios$ ssh [email protected]
gateway$ sudo mv /tmp/authorized_keys /home/nagios/.ssh/authorized_keys && 
sudo chown nagios /home/nagios/.ssh/authorized_keys && 
sudo chmod 0644 /home/nagios/.ssh/authorized_keys

If this succeeds, we should now be able to SSH from the Nagios server to the gateway host without requiring a password. As root, issue the following (as the Nagios user account cannot access a shell as it is set to /sbin/nologin for their account; see the /etc/passwd file):

nagios# sudo –u nagios –E ssh nagios@gateway
The authenticity of host 'gateway (192.168.0.254)' can't be established.
ECDSA key fingerprint is 2d:94:d5:bd:3e:40:93:fe:d4:9b:eb:6f:93:4d:f3:a1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gateway,192.168.0.254' (ECDSA) to the list of known hosts.

Here we have used sudo to execute the ssh command as the nagios user. The –E says use the Nagios user’s environment. This is important because this command will write the host key to the Nagios user’s ∼/.ssh/known_hosts file. We now configure Nagios to use this connection to check services. The check_by_ssh plug-in also relies on having the command to be executed installed on the remote host. This command is usually a Nagios plug-in, and hence the easiest way to do this is to install the Nagios plug-in package on the remote host. On CentOS, we issue the following:

gateway$ sudo yum install nagios-plugins nagios-plugins-load

On Ubuntu, you can install the plug-ins like we did on the Nagios server, or you should be able to install them from the APT repository.

gateway$ sudo apt-get install nagios-plugins

We can then define a command that uses the check_by_ssh plug-in to monitor a service on the remote host. For example, to monitor the load on a remote host, we could use the following command:

$ sudo vi /etc/nagios/objects/commands.cfg
# ssh_check_commands
define command{
         command_name       check_load_ssh
         command_line       $USER1$/check_by_ssh -H $HOSTADDRESS$ -l nagios  
-C "/usr/lib64/nagios/plugins/check_load -w $ARG1$ -c $ARG2$"
        }

We have added the previous command definition to the /etc/nagios/objects/commands.cfg file. We call our command check_load_ssh. The command_line part specifies that we’re executing the check_by_ssh plug-in and connecting to the host specified by the -H option.

The -l option specifies the name of the user we want to connect to on the remote host; here we’re using the nagios user we just created.

The -C specifies the command we want to run on the remote host. In this case, we’re running another, locally installed, Nagios plug-in called check_load and passing two arguments to it as the values of the -w and -c (WARNING and CRITICAL) thresholds.

We are going to create a file called linux-servers-base.cfg in the /etc/nagios/objects directory. This is going to be used to define our gateway host and define the service check we will use to check the load on the gateway server. The file will look like Listing 18-21.

Listing 18-21. linux-servers-base.cfg

define host{
        name            basic-ssh-checks
        use               linux-server
        hostgroups   linux-group
        register        0
        }

define host{
        host_name       gateway
        alias                 gateway.example.com
        use                  basic-ssh-checks
        }

define service{
         use                              local-service
         hostgroup_name         linux-group
         service_description     Current Load
         check_command         check_load_ssh!5.0,4.0,3.0!10.0,6.0,4.0
        }

define hostgroup{
        hostgroup_name  linux-group
        }

In Listing 18-21 we have defined one host template, one host, one service, and one hostgroup. The host template is basically used to attach any hosts that use this template to the linux-group hostgroup. We are using the hostgroup here to link the host template and the service together. Every host that uses the template basic-ssh-checks will get the Current Load service because they share the same hostgroup.

Taking a look at our service definition closer, we can see that we have declared it like this:

define service{
         use                              local-service
         hostgroup_name         linux-group
         service_description     Current Load
         check_command         check_load_ssh!5.0,4.0,3.0!10.0,6.0,4.0
        }

Our service, called Current Load, executes the check_load_ssh command and passes two arguments, which specify the average load over the 1-, 5-, and 15-minute intervals required to trigger a WARNING or CRITICAL status.

Finally in Listing 18-21 we have the host definition for the gateway server. This uses the template basic-ssh-checks, which calls in the Current Load service check via the hostgroup. It also will have all the notification and check period definitions provided by the linux-server template and the generic-host templates as we do not override anything here or in the templates.

This is a simple example of how to perform remote monitoring; it is not ideal (the key-based SSH connection could be a security vulnerability), but it is the simplest method. For more complicated environments, using the NRPE server and corresponding command is usually a better approach. NCAP and NRDP are also available and require a combination of different plug-ins with a mix of languages (which makes them awkward to install and configure).

Logging and Debugging

You can view the logs for Nagios in the /var/log/nagios/nagios.log file. You may also have these notifications being replicated to the syslog (rsyslog) daemon.

If you do not want this functionality, you can set your use_syslog directive to the following:

use_syslog=0

When you need to debug Nagios , you will need to edit the /etc/nagios/nagios.cfg file and edit the following:

debug_level=0

Do you remember from LDAP logging when we had additive logging directives that give different log levels by adding binary numbers together? Nagios is the same. Here is the list of possible logging levels:

#          -1 = Everything
#          0 = Nothing
#          1 = Functions
#          2 = Configuration
#          4 = Process information
#          8 = Scheduled events
#          16 = Host/service checks
#          32 = Notifications
#          64 = Event broker
#          128 = External commands
#          256 = Commands
#          512 = Scheduled downtime
#          1024 = Comments
#          2048 = Macros

So if we wanted to see host and service checks in our logs and the external commands, we would change debug=0 to debug=144, which is 16 (host/service checks) + 128 (external commands).

You will need to restart Nagios for this to take effect, and the debug logs will appear in a /var/log/nagios/debug.log file.

Nagios Plug-ons

You can choose from a large collection of plug-ins to create services and commands to check them. You can see a partial list of the available plug-ins in Table 18-7.

This is a small selection of the available plug-ins, but it should give you a good basis for creating appropriate checks for your environment. Most Nagios plug-ins are simple and self-explanatory. Almost all of them provide the --help option to display their function and options.

Other plug-ins outside of the Nagios plug-in pack are also available. For example, you can find a huge collection of such plug-ins at https://exchange.nagios.org/directory/Plugins .

You can develop your own plug-ins , if required. Some simple guidelines and examples for such development are located at https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/pluginapi.html .

Console Authentication

To protect against people making malicious use of your Nagios console, the web server has some basic authentication. Nagios uses Apache’s basic HTTP authentication to protect the console. When you open the console, you will be prompted for a username and password, as you can see in Figure 18-9.

Figure 18-9. Nagios authentication

Apache basic authentication is configured by specifying a file holding usernames and passwords to the web server. This file is then queried by web sites that are secured by basic authentication. The default username and password are nagiosadmin.

Let’s look inside our Nagios Apache configuration for the location of our password file. On CentOS, inside our /etc/httpd/conf.d/nagios.conf file, we find the following:

AuthName "Nagios Access"
AuthType Basic
AuthUserFile /etc/nagios/passwd
<IfModule mod_authz_core.c>
      # Apache 2.4
     <RequireAll>
        Require all granted
        Require valid-user
    </RequireAll>
</IfModule>

On Ubuntu, in the /etc/apache2/conf-available/nagios.conf file, we have this:

AuthName "Nagios Access"
AuthType Basic
AuthUserFile /etc/nagios/htpasswd.users
Require valid-user

The AuthName and AuthType directives enable basic authentication. The AuthUserFile specifies the location of the password file, /etc/nagios/passwd on CentOS and /etc/nagios/htpasswd.users on Ubuntu. This is the file we need to add our users to. The Require valid-user directive indicates that only users specified in this file can log into the console.

Once you’ve found where your authentication file needs to be, you need to create the file to hold your users and passwords. You do this using a command called htpasswd.

Listing 18-22 shows this command in action.

Listing 18-22. Using the htpasswd Command

$ sudo htpasswd -c /etc/nagios/passwd jsmith
New password:
Re-type new password:
Adding password for user jsmith

The htpasswd command has two variables: the location of the file that holds our usernames and passwords and the username of the user. We also use a command-line switch, -c. The -c switch is used when you first create a password file. After that, you can drop this switch and merely specify the file and the user to be added.

In Listing 18-22, we added a new password to the file with the -c option called /etc/nagios/passwd. We specify the user we’re adding, jsmith, and are prompted for a password and then a verification of the entered password. Both passwords must match. If they do, the command will be successful, and the user will be added to the specified password file.

We can then use this username and password to sign in to the Nagios console and display the console screen, as you can see in Figure 18-10.

Figure 18-10. The Nagios console

Once we have users and passwords created, we can sign on to the console, but we also need to configure what access each user has. This is done by editing a configuration file called cgi.cfg, located in the /etc/nagios directory.

Nagios has two types of users on the console: authenticated users and authenticated contacts. Both types of users need a username and password to sign on to the console. Authenticated users have their access specified in the cgi.cfg configuration file. Authenticated contacts are authenticated users, with the username of each matching the name of a Nagios contact.

Hence, if the username jsmith, for example, matches the name of a contact, which is the value of the contact_name directive, this authenticated user becomes an authenticated contact.

So, what’s the difference? Authenticated users are granted some generic rights to view the web console. Authenticated contacts are granted further rights to view and manipulate the hosts and services for which they are contacts.

Let’s look at the cgi.cfg file. The first directive in the cgi.cfg file is called use_authentication. It controls whether authentication is enabled for the Nagios web console and whether Nagios will use the authentication credentials provided from the web server. The directive looks like this:

use_authentication=1

A setting of 1, which is the default, enables authentication, and a setting of 0 disables it. Authorization for particular functions on the console is provided by a series of directives in the cgi.cfg file that take lists of users, separated by commas, as options. Here’s an example:

authorized_for_all_services=jsmith,nagiosadmin

The authorized_for_all_services directive controls who can view services on the console, and we’ve specified that the users jsmith and nagiosadmin have this access.

Table 18-8 contains the full list of possible authorization directives and describes each one.

The first directive in Table 18-8, authorized_for_system_information, provides access to view information about the Nagios process and the server, such as when the process started and what settings are set on the server.

The second directive, authorized_for_configuration_information, provides authorization to view all configuration information and the object definitions for your monitoring environment. This includes the configuration of your hosts, services, contacts, and commands, as well as all other object types.

The third directive, authorized_for_system_commands, controls who has access to start, stop, or restart the Nagios process from the web console.

The next two directives, authorized_for_all_services and authorized_for_all_hosts, control which users can view all service and host information on the web console. Remember, authenticated contacts can view the information about the hosts and services for which they are contacts.

The last two directives in Table 18-8, authorized_for_all_service_commands and authorized_for_all_host_commands, allow you to specify users who are authorized to issue external commands to services and hosts, respectively. This allows you to perform actions such as disabling active checks of the host or service or enabling or disabling notifications for the host or service.

If you want to specify that all users have access to a particular function, use the * symbol.

authorized_for_all_services=*

This directive setting would provide all authenticated users with access to view information about all services defined on the Nagios server. The * symbol will work for all authorization directives.

As we mentioned earlier, in addition to any authorization granted to them, users who are also contacts have access to the hosts and services for which they are contacts. For services, this access includes

• Viewing of service status

• Viewing of service configuration

• Ability to view service history and notifications

• Ability to issue commands to the service (start and stop checking, for example)

For hosts, this access includes

• Viewing of host status

• Viewing of host configuration

• Ability to view host history and notifications

• Ability to issue commands to the host (start and stop checking, for example)

Authenticated contacts that have access to a particular host because they are a contact for that host also have the same access to all the services on that host just as if they were a contact for those services. For example, if you are an authenticated contact for the server1 host, you are able to view the status, configuration, service history, and notifications as well as issue commands to all the services defined on that host.

Console Functions

The Nagios console provides not only an interface through which to view your hosts and services but also the ability to control how they are monitored. We’re going to walk you through a few of the screens available in the console, but we recommend you take a look at the views, graphs, lists, and controls available on the console.

For a summary of the status of your environment, the best view is the Tactical Monitoring Overview screen, which you can see in Figure 18-11.

Figure 18-11. The Tactical Monitoring Overview screen

This screen displays the current status of the Nagios server and a summary of your host and service statuses. It is reached by clicking the Tactical Monitoring Overview link in the left menu.

To see your hosts and services in more detail, the Host Detail and Service Detail links in the left menu display a full list of the hosts and services being monitored. You can see the Service Detail screen in Figure 18-12.

Figure 18-12. The Hosts Detail screen

You can see here that we have two hosts listed: the localhost or the Nagios server and the gateway server we configured earlier. You can see that we can ping both hosts and that they are in the status of UP. On the right side you can see information about the ping response; also, you can see the last time they were checked and how long they have been in that current state for.

In Figure 18-13 we show the Service Detail screen.

Figure 18-13. The Service Detail screen

Here in Figure 18-13 we can see the status of the services that we are checking on our hosts. You can see the check for Current Load is working on our gateway server. The HTTP check is in a WARNING state because the httpd service is responding but the result is not 200 OK but rather 403. We would have to change the check to handle the authentication the Nagios web service requires before we will get a 200 OK and for it to be green.

On both these screens, you can click the hosts and services to drill down into their status and configuration.

The last console screen we’re going to show you in Figure 18-14 is the Process Info screen, which displays the status of the Nagios process and its configuration. You can also configure a variety of Nagios options, such as enabling and disabling checking and notifications, through this screen.

Figure 18-14. The Nagios Process Information screen

You can do other things through this console, such as host and service reports, check the event log, and see and schedule downtime for hosts and services.