In this recipe, we'll learn how to specify an alternative method for a contact to receive notifications about a service. A very typical method for a contact to receive notifications is by being sent an e-mail to their contact address; e-mail messages could be sent to an inbox or a paging device.
However, notifications are just text; we can arrange to deal with them via any command we wish, in much the same way we can configure host or service checks. In this recipe, we'll set up a new contact called motd, which, when it receives notifications, will write them into the server's /etc/motd to be displayed on login.
You should have a Nagios Core 4.0 or newer server, with at least one host or service configured already. You should understand how notifications are generated and their default behavior in being sent to the contacts and contact_groups for hosts or services.
We'll use the example of a host called troy.example.net, configured to send notifications to an ops contact group. We'll add a new contact within this group called motd.
We can arrange our custom notification method as follows:
- Ensure that your
motd(5)file can be written to by Nagios Core and that it's a static file that your system will not overwrite on boot. If you run your Nagios Core server as the recommended default user and group ofnagios, you could do something like this:# chgrp nagios /etc/motd # chmod g+w /etc/motd
- It would be a good idea to test that the user can write to the file using
su(8)orsudo(8):# sudo -s -u nagios $ printf 'test ' >>/etc/motd
- Change to the objects configuration directory for Nagios Core. The default is
/usr/local/nagios/etc/objects. If you've put the definition for your host in a different file, move to that directory instead.# cd /usr/local/nagios/etc/objects - Edit the file containing the definitions for your notification commands. By default, this is in the file
commands.cfg. Append the following definitions to the file:define command { command_name notify-host-motd command_line /usr/bin/printf "%s " "$SHORTDATETIME$: $HOSTALIAS$ is $HOSTSTATE$" >>/etc/motd } define command { command_name notify-service-motd command_line /usr/bin/printf "%s " "$SHORTDATETIME$: $SERVICEDESC$ on $HOSTALIAS$ is $SERVICESTATE$" >>/etc/motd } - Edit the file containing the definitions for your contacts. By default, this is in the file
contacts.cfg. Append the following definitions to the file:define contact { use generic-contact contact_name motd contact_groups ops alias MOTD host_notification_commands notify-host-motd service_notification_commands notify-service-motd } - Edit the file containing the definitions for your host and/or services to ensure that they specify
opsin theircontact_groupsdirectives:define host { host_name troy.example.net ... contact_groups ops } define service { host_name troy.example.net ... contact_groups ops } - Validate the configuration and restart the Nagios Core server:
# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg # /etc/init.d/nagios reload
With this done, the next notifications that go out for the host and services should not only be e-mailed to any contacts in the ops group, but should also be written to the MOTD to be presented on login:
06-30-2012 17:24:05: troy is DOWN 06-30-2012 17:24:05: HTTP on troy is CRITICAL 06-30-2012 17:25:07: troy is DOWN 06-30-2012 17:25:07: HTTP on troy is CRITICAL
The first part of the configuration that we added is defining new notification commands. These are actually command definitions, just the same as those used with the check_command definitions for hosts and services; the difference is that they're used by contacts to send notifications to people (or in this case, write them to files) in the appropriate way.
If you're using the default Nagios Core configuration, the commands already defined in commands.cfg should include notify-host-by-email and notify-service-by-email. I've truncated the complete command line in the following code as it's very long:
define command {
command_name notify-host-by-email
command_line /usr/bin/printf "%b" "***** Nagios *****
Notification Type: $NOTIFICATIONTYPE$
Host: $HOSTNAME$
State: $HOSTSTATE$
Address: $HOSTADDRESS$
Info: $HOSTOUTPUT$
Date/Time: $LONGDATETIME$
" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
}
define command {
command_name notify-service-by-email
command_line /usr/bin/printf "%b" "***** Nagios *****
Notification Type: $NOTIFICATIONTYPE$
Service: $SERVICEDESC$
Host: $HOSTALIAS$
Address: $HOSTADDRESS$
State: $SERVICESTATE$
Date/Time: $LONGDATETIME$
Additional Info:
$SERVICEOUTPUT$
" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}These commands use the system printf implementation, along with many Nagios Core macros, to build an appropriate notification string, which is e-mailed to the appropriate contact using a shell pipe into the system mailer.
If we inspect the templates.cfg file, where the generic-contact template is defined, we can see these methods are referred to in two directives:
define contact {
...
service_notification_commands notify-service-by-email
host_notification_commands notify-host-by-email
}This configuration thus defines a particular notification command to be used for hosts that derive from the generic-host template.
Our own command definition does something similar in building a string with a call to printf; but instead of writing the output into a pipe to the system mailer, it appends it to the /etc/motd file.
The command_line we specified for the motd contact to process host notifications is as follows:
/usr/bin/printf "%s " "$SHORTDATETIME$: $HOSTALIAS$ is $HOSTSTATE$" >>/etc/motd
The first thing Nagios Core does when a notification event for this contact is successfully fired is substitute values for its macros, $SHORTDATETIME$, $HOSTALIAS$, and $HOSTSTATE$:
/usr/bin/printf "%s " "06-30-2012 17:24:05: troy is DOWN" >>/etc/motd
The resulting string is then executed as a command by the running Nagios Core user, by default nagios, and provided that the user has the appropriate permissions, it's appended to the end of the MOTD.
Adding messages to the MOTD may not be useful for all network administrators, particularly for larger or more sensitive networks for which hundreds of notifications could be generated daily. The main thing to take away from this is that Nagios Core can be configured to process notification events in pretty much any way, provided the following conditions are met:
- We can define a command line that will consistently do what is needed with the results of the macro substitution Nagios Core will perform for us
- The
nagiosuser, or whichever user the Nagios Core server is running, has all the relevant permissions it needs to run the command
E-mail notifications, particularly to a mobile paging device, are a very sensible basis for notifications for most administrators. However, if some other means of notification is needed, such as inserting into a database or piping into a Perl script, all that is possible as well.
Note that it's safest to specify the full paths to any binaries or files we use in these commands, hence /usr/bin/printf rather than merely printf. This is to ensure that the program we actually intend to run is run and avoid having to mess about with $PATH specifications for the nagios user.