NRPE is a client-server solution for running check commands on remote computers. It is designed explicitly to allow the central Nagios server to trigger checks on other machines in a secure manner.
NRPE offers a very good security mechanism along with encryption mechanisms. It is possible to specify a list of machines that can run checks via NRPE and which plugins can be run along with aliases that should be used by the central Nagios server.
The main difference is that the communication overhead is much smaller than for running checks over the SSH protocol. This means that both the central Nagios server and the remote machine need less CPU time to perform a check. This is mainly important for Nagios servers that deal with a lot of checks that are performed remotely on machines. If the SSH overhead compared to NRPE is 1 second per each test that has to be run, then when performing 20,000 checks it maps to 5.5 hours that would be spent on SSH overhead.
NRPE also allows running only specific commands to perform specific checks—so it is not possible to use the NRPE protocol to run arbitrary commands. While the same is possible with SSH by configuring it properly, NRPE provides this by default.
NRPE uses the TCP protocol with SSL encryption on top of it. Enabling encryption is optional, but it is highly recommended. By default, NRPE communicates on port 5666. The connection is always made from the machine running the Nagios daemon to the remote machine. If your company has firewalls set up for local connectivity, make sure that you allow communications from port 5666 of your Nagios server.
The following is an illustration of how such a check is performed:
Nagios determines that an active check should be performed. It runs the check_nrpe plugin that connects to the remote host's NRPE daemon. After the NRPE daemon accepts this as a valid host to send commands to, check_nrpe sends the command to be run along with any parameter to the remote machine.
Next, the NRPE daemon translates these into the actual system command to be run. In case the specified command is not allowed, the NRPE daemon will reject this request. Otherwise it will run the command and pass the results back to check_nrpe on the machine hosting the Nagios daemon. This information is then passed back to the Nagios daemon and stored in the data files and/or databases.
The NRPE package consists of two parts: the NRPE daemon and the NRPE check command. The first one needs to be running on all remote machines that are to be monitored using this method. The NRPE check command (check_nrpe) is a Nagios plugin to perform active checks and needs to be installed on the machine on which the Nagios daemon is running.
NRPE is a core add-on for Nagios and it is maintained by the Nagios development team. NRPE can be downloaded as both source code and binary packages with multiple Linux distributions.
The NRPE source package can be downloaded from the Nagios download page (https://www.nagios.org/downloads/nagios-core-addons/). The actual downloads are hosted on SourceForge and the NRPE sources can be retrieved from https://sourceforge.net/projects/nagios/files/nrpe-2.x/nrpe-2.15/—version 2.15 is the latest version at the time of writing.
For Ubuntu Linux, the package names are nagios-nrpe-server and nagios-nrpe-plugin for the daemon and client, respectively. For Ubuntu, the command to install the client and the server is as follows:
apt-get install nagios-nrpe-server nagios-nrpe-plugin
For RHEL, CentOS, and Fedora systems that have yum installed, the package names are nagios-nrpe and nagios-plugins-nrpe for the daemon and the client, respectively. The command to install both client and server is as follows:
yum install nagios-nrpe nagios-plugins-nrpe
Microsoft Windows version of the NRPE daemon can be found in the NRPE_NT project on SourceForge (http://sourceforge.net/projects/nrpent/). It offers the same functionality as its Unix version and is configured in the same way.
The Nagios plugins do not provide the Windows version, so you will need to compile Nagios plugins using the Cygwin package (visit http://www.cygwin.com/). You can also provide only your own check commands and set up NRPE_NT to use those. In the case of Microsoft Windows, it is important to remember that your plugins need to be command-line tools and cannot be created as GUI-based tools.
Monitoring Microsoft Windows-based machines and using the NRPE protocol for performing the checks is described in more detail in Chapter 12, Advanced Monitoring.
If you are using NRPE from prebuilt packages, you can skip this section and resume with the NRPE configuration information. Compiling NRPE requires a standard compiler, linker, and similar tools to be present on your system. It also needs the OpenSSL package along with the actual openssl command line, which is used to generate the Diffie-Hellman key for each instance.
On an Ubuntu Linux system, installing the prerequisite packages can be done by performing the following command:
apt-get install gcc make binutils cpp pkg-config libc6-dev
libssl-dev openssl
For other Linux distributions and operating systems, the commands and package names may vary, but should be very similar.
More information on what packages should be installed on other operating systems and how to do this can be found in Chapter 2, Installing Nagios 4.
Now that our packages are set up, the next step is to run the configure script that will set up the NRPE parameters and create the Diffie-Hellman key.
For standard paths and users that were used in Chapter 2, Installing Nagios 4, the command is as follows:
sh configure --sysconfdir=/etc/nagios --libexecdir=/opt/nagios/plugins --prefix=/opt/nagios --localstatedir=/var/nagios --with-nrpe-user=nagios --with-nrpe-group=nagios --with-nagios-user=nagios --with-nagios-group=nagios --enable-ssl
If the configuration step has failed, one common reason is that the SSL library could not be found even if it was installed properly in the system. This is due to a configuration script sometimes not locating the libraries, especially on 64bit Linux systems.
What you can do is add the --with-ssl-lib option and point to the proper directory. First we'll need to find libssl.so —such as follows:
# find /usr -name libssl.so /usr/lib/x86_64-linux-gnu/libssl.so
This means the directory is /usr/lib/x86_64-linux-gnu and --with-ssl-lib should be set to that—for example:
sh configure --sysconfdir=/etc/nagios --libexecdir=/opt/nagios/plugins --prefix=/opt/nagios --localstatedir=/var/nagios --with-nrpe-user=nagios --with-nrpe-group=nagios --with-nagios-user=nagios --with-nagios-group=nagios --with-ssl-lib=/usr/lib/x86_64-linux-gnu --enable-ssl
If running the configure script still fails, it is probably because one or more of the required packages are missing. If this happens, verify whether all the packages mentioned earlier in the chapter have been installed, and then try again. Also, if you know that the package is properly installed, it may require additional options to be passed. In such cases, it is recommended to check for the exact error code on the Internet.
The next step is to actually build the NRPE client and daemon. To do this, run the following command:
make all
This command will build both the binaries and then create the sample configuration files for the NRPE daemon.
It is a very common problem that the build fails, claiming that the get_dh512 function could not be found. The problem is not obvious. In this case, please make sure that the openssl command is installed, the directory where it is located is added to the PATH environment variable, and then run all of the steps again—starting with the configure script.
The problem is that the configure script tries to generate a Diffie-Hellman key if a problem exists during this step. Then the script itself does not fail to complete, but the build process eventually fails. Please make sure that somewhere at the end of the output from the configure script, a text similar to the one that follows is printed out:
*** Generating DH Parameters for SSL/TLS *** Generating DH parameters, 512 bit long safe prime, generator 2 This is going to take a long time +..............+...........+........++*+*++*++*++*++*
If the openssl command is not present, the following error will show up instead:
*** Generating DH Parameters for SSL/TLS *** configure: line 6703: /usr/bin/openssl: No such file or directory
If the compilation process fails for any other reason, it is most probably due to the missing libraries or header files. In this case, installing the packages mentioned earlier will help.
Assuming that the build succeeded, the next step is to install either the NRPE client or the daemon. On the machine that is running the Nagios daemon, we need to install the client check_nrpe command. To do this, type the following command:
make install-plugin
This command will copy the check_nrpe command to the /opt/nagios/plugins directory. NRPE does not require any configuration file for the NRPE client, and hence, no additional file needs to be copied. For all of the remaining machines, please run the following command to install the NRPE daemon:
make install-daemon
This command will copy the nrpe binary to the /opt/nagios/bin directory.
Because the NRPE daemon requires configuration, it is recommended that you copy the sample-config/nrpe.cfg file as /etc/nagios/nrpe.cfg.
Our NRPE daemon is now built and ready to be deployed on remote machines. We need to configure it and set up the system so that it accepts connections from other computers. The following steps should be applied to all machines where NRPE should be running.
First we need to create a user and a group named nagios that NRPE daemon will be running as:
groupadd nagios useradd -g nagios -d /opt/nagios nagios
We also need to create a home directory for the user, and it is a good idea to lock out access for that user if no checks are to be performed over SSH. To do this, run the following commands:
mkdir /opt/nagios chown nagios:nagios /opt/nagios passwd -l nagios
This was needed as the machine is not the same as where the Nagios server itself is running.
There are many ways of setting this up—NRPE can work either as a standalone process that handles incoming connections using init.d or system approach. It can also be run as part of the inetd setup (http://en.wikipedia.org/wiki/inetd) or the xinetd (https://en.wikipedia.org/wiki/Xinetd) setup. In all cases, a configuration file is needed. This file specifies the commands to be used and the additional options to run the NRPE daemon standalone.
To run NRPE as a standalone process, which is the preferred way in most cases, all that is needed is to create a script called /etc/init.d/nrpe containing the following:
#!/bin/sh
#
### BEGIN INIT INFO
# Provides: nrpe
# Required-Start: $local_fs $syslog $network
# Required-Stop: $local_fs $syslog $network
# Short-Description: Starts and stops the NRPE server
# Description: Starts and stops the NRPE server
### END INIT INFO
is_running() {
pgrep -u nagios nrpe >/dev/null
}
case "$1" in
start)
echo -n "Starting NRPE: "
if ! is_running ; then
/opt/nagios/bin/nrpe -d -c /etc/nagios/nrpe.cfg
fi
echo "done."
;;
stop)
echo -n "Stopping NRPE: "
if is_running ; then
pkill -u nagios nrpe
sleep 1
pkill -u nagios -9 nrpe
fi
echo "done."
;;
restart)
$0 stop
sleep 2
$0 start
;;
status)
if is_running ; then
echo "NRPE is running"
exit 0
else
echo "NRPE is NOT running"
exit 1
fi
;;
esac
The script has to be set to be executable—which we can do by running:
# chmod 0755 /etc/init.d/nrpe
After that we need to register nrpe as a system service—which can be done using the following command on Ubuntu:
# update-rc.d nrpe
And for CentOS, RHEL, and many other distributions the command is:
# chkconfig --level 345 nrpe on
In order for the script to work properly, we need to create a configuration file for NRPE.
The NRPE configuration file is similar to the main Nagios configuration file—all parameters are written in the form of <name>=<value>. If you have compiled NRPE from the source, then a default configuration can be found in the sample-config/nrpe.cfg file.
A sample NRPE configuration script that will work for both standalone installations as well as under inetd is as follows:
log_facility=daemon pid_file=/var/run/nrpe.pid server_port=5666 nrpe_user=nagios nrpe_group=nagios allowed_hosts=192.168.0.1 command_timeout=60 connection_timeout=300 debug=0
The first series of parameters includes information related to logging. NRPE uses standard Unix logging mechanisms. The log_facility parameter specifies the syslog facility name to be used for logging. The default value is daemon, but it can be set to any of the predefined syslog facility names.
A standalone NRPE daemon also allows the setting up of the IP address and the port to listen to as well as the user and group names to be used. In order to specify that NRPE should listen only on a specific IP address, you need to use the server_address parameter. If this parameter is omitted, then the NRPE will listen on all the network interfaces. The server_port parameter is used to specify the port number on which NRPE should listen.
If NRPE should accept connections only from a predefined list of machines, you need to specify the allowed_hosts parameter, which will contain a list of all the IP addresses of these machines, separated by commas. This should be set to the IP address of Nagios instance(s) that will be performing checks on this machine.
NRPE usually runs as a separate user for security reasons. The options to specify the user and group names that should be used by NRPE are nrpe_user and nrpe_group respectively.
We can also specify the file to which NRPE should write the PID of the daemon process; this can be useful in startup scripts or service monitoring tools that can read this file to determine if NRPE is running. The option name is pid_file.
We can also tell NRPE how long a command can run. The first option is command_timeout, and it tells NRPE how many seconds a command can run before it should be stopped. If a command is running for more than the specified number of seconds, it is terminated and a CRITICAL status is sent back to the NRPE client.
The connection_timeout option specifies the time in seconds after which a connection should be closed if no data has been received. This does not change the way the command times out, but it only specifies how much time NRPE should wait for a command to be sent.
NRPE also offers a debug option that can specify whether it should record a large amount of information in the system log. A value of 1 enables verbose logging and 0 disables it. This should be disabled in production, but can be useful during initial runs in case you run into a problem.
We also need to configure the commands that can be run by the allowed machines.
The NRPE commands define aliases for the actual commands that will be executed. All commands have a unique name and the actual command line to be run. Usually command names are the plugin names or the plugin names with some description appended.
For example, the check_disk command that checks the /home directory could be called check_disk_home.
The commands are also defined in the nrpe.cfg file and each check is defined as command[<command_name>]=<command_to_execute>. The command_name has to be unique. The same set of commands can be run by all hosts specified in the allowed_hosts parameter.
An example command definition to use check_disk and to verify the space on the root partition is as follows:
command[check_disk_sys]=/opt/nagios/plugins/check_disk -w 20% -c 10% -p /
It is also worth noting that there is no possibility of defining which hosts can run which commands, for example it is not possible to only allow certain hosts to run certain checks. All defined commands can be run by any host that is allowed to run commands on this instance.
It would be a good idea to create a template configuration that will contain the typical checks and the hosts that should be allowed to run the checks. These can be modified later for individual hosts, but using a template makes it easier to deploy the checks for a large number of boxes. A typical set of commands would be as follows:
command[check_rootdisk]=/opt/nagios/plugins/check_disk -w 20%
-c 10% -p /
command[check_swap]=/opt/nagios/plugins/check_swap -w 40% -c 20%
command[check_sensors]=/opt/nagios/plugins/check_sensors
command[check_users]=/opt/nagios/plugins/check_users -w 10 -c 20
command[check_load]=/opt/nagios/plugins/check_load
-w 10,8,5 -c 20,18,15
command[check_zombies]=/opt/nagios/plugins/check_procs -w 5 -c 10
-s Z
command[check_all_procs]=/opt/nagios/plugins/check_procs -w 150 -c 200In case you need to troubleshoot why a check is failing, it would be a good idea to set the debug parameter to 1 in nrpe.cfg. If NRPE is running in standalone mode, it will need to be restarted for the changes to take effect. An example log from a connection is as follows:
Apr 21 20:07:29 ubuntu2 nrpe[5569]: Handling the connection... Apr 21 20:07:29 ubuntu2 nrpe[5569]: Host is asking for command 'check_root_disk' to be run... Apr 21 20:07:29 ubuntu2 nrpe[5569]: Running command: /opt/nagios/plugins/check_disk -w 20% -c 10% -p / Apr 21 20:07:29 ubuntu2 nrpe[5569]: Command completed with return code 0 and output: DISK OK - free space: / 7211 MB (90% inode=96%);| /=759MB;6717;7557;0;8397 Apr 21 20:07:29 ubuntu2 nrpe[5569]: Return Code: 0, Output: DISK OK - free space: / 7211 MB (90% inode=96%);| /=759MB;6717;7557;0;8397
Another requirement for using NRPE is that the commands need to be specified using the full path to the plugin, and no macro substitution can take place. Not being able to use any macro definitions requires more attention when writing macros. It also requires that any change to the command is edited in the NRPE configuration on the remote machine, and not in the Nagios configurations on the central server. This introduces a very strict security model, but makes NRPE a bit harder to maintain.
In some cases, it is better to be able to pass arguments to NRPE from the Nagios server and have NRPE put these into the command definition. Even though this functionality is disabled for security reasons, it is possible to enable it. How NRPE can be set up to accept parameters from the Nagios server is described in the Using command arguments with NRPE section in this chapter.
The next step is to set up Nagios to use NRPE for performing checks via a remote machine. Using NRPE to perform checks requires the creation of one or more commands that will use the check_nrpe plugin to send actual check requests to a remote machine.
The syntax of the plugin is as follows:
check_nrpe -H <host> [-n] [-u] [-p <port>] [-t <timeout>] [-c <command>] [-a <arglist...>]
The following table describes all of the options accepted by the plugin. The items required are marked in bold:
|
Option |
Description |
|
|
This provides the hostname or IP address of the machine to connect; this option must be specified |
|
|
This is the name of the command that should be executed; the command needs to be defined in the |
|
|
This disables SSL for communication |
|
|
This connects to the specified port; defaults to 5666 |
|
|
This is the number of seconds after which a connection will be terminated; defaults to |
|
|
If a timeout occurs, this will return an |
The only two required attributes are -H and -c, which specify the host and the command alias to run on that machine, respectively.
The next thing we should do is make sure that the NRPE server on the remote machine is working correctly. Assuming that check_swap is a valid command defined in NRPE on a remote machine, we can now try to connect from the Nagios server. The first thing that's worth checking is whether calling check_nrpe directly, works:
$ /opt/nagios/plugins/check_nrpe -H 192.168.2.52 -c check_swap SWAP OK - 100% free (431 MB out of 431 MB) |swap=431MB;86;43;0;431
In our example, 192.168.2.52 is the IP address of the remote computer. As the connection was successful, NRPE passed the actual plugin output to the standard output. After a successful check, we can now define a command in the Nagios configuration that will perform a check over NRPE:
define command
{
command_name check_swap_nrpe
command_line $USER1$/check_nrpe -H "$HOSTADDRESS$"
-c "check_swap"
}
We can then use the check_swap_nrpe command in a service definition. NRPE has a much lower overhead as compared to SSH. So, in some cases, it would be a good idea to use NRPE even for performing local checks.
In case we are defining a service for a group of hosts, we can use the same trick as those for checks over SSH to perform checks on a local machine by using the plugins directly and checking all of the remaining machines using NRPE. This will reduce the overhead related to monitoring the local machine and remove the need to install NRPE on the local host.
The following is a sample configuration that defines a check for swap usage locally for the computer on which it is defined, and over NRPE for all the remaining machines:
define service
{
use generic-service
host_name localhost
service_description SWAP
check_command check_swap
normal_check_interval 15
}
define service
{
use generic-service
host_name !localhost
hostgroup_name linux-servers
service_description SWAP
check_command check_swap_nrpe
normal_check_interval 30
}
By default, NRPE is configured to run only the predefined commands and it is not possible to pass any arguments to the commands that will be run. In some cases, this is hard to manage as changes to the command configurations need to be done at the remote machine level and not at the central Nagios server level—for example, with a large number of partitions mounted on various servers.
In such cases it might be worth investigating an option included in NRPE to pass arguments to commands. This option is disabled by default as it is considered to be a large security concern. This is because it is possible to send malicious arguments to a check command and make it perform actions other than the ones it should be doing.
If you trust all requests to NRPE to be coming from your Nagios instance it is possible to enable this functionality within the NRPE daemon. This allows easier management of NRPE and the Nagios configuration.
The first thing that needs to be done is the rebuilding of the NRPE daemon with this option enabled. To do this, run the configure script again with the --enable-command-args flag added.
For example, the configure script invocation could be as follows:
sh configure --sysconfdir=/etc/nagios --libexecdir=/opt/nagios/plugins --prefix=/opt/nagios --localstatedir=/var/nagios --with-nrpe-user=nagios --with-nrpe-group=nagios --with-nagios-user=nagios --with-n agios-group=nagios --enable-command-args --enable-ssl
Please note that if the --with-ssl-lib option had to be specified to build NRPE, it also has to be added to the example invocation above.
It is also necessary to rebuild the NRPE daemon and reinstall the binary by using the following commands:
make all make install-plugin make install-daemon
If you are running NRPE as a standalone daemon, then you need to restart the daemon after overwriting the binary. Only the daemon on the remote machine needs to be reconfigured and recompiled. It is not necessary to rebuild the NRPE client as it always supports the passing of arguments to the NRPE daemon.
The next step is to add the dont_blame_nrpe option to the nrpe.cfg file and set it to 1. This option, despite its strange name, enables the functionality to use arguments in the command definitions. When NRPE is compiled with this option and the option is enabled in the NRPE configuration, passing arguments to commands is possible.
Now it is possible to use the $ARGn$ macros in the NRPE configuration, similar to how they are defined in Nagios. This works in the same way as Nagios, where $ARG1$ indicates the first argument, $ARG2$ the second one, and so on for up to 16 arguments. For example, a check command that checks the disk space on any partition looks like the following:
command[check_disk]=/opt/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
This requires that the warning and critical levels are passed during the check. The actual path to the mount point, which is specified as a third parameter, is essential. Arguments are passed to check_nrpe by specifying the -a flag and passing all required arguments after it, with each argument as a separate parameter. An example invocation of the check command as a standalone command would be as follows:
$ /opt/nagios/plugins/check_nrpe -H 10.0.0.1 -c check_disk -a 10% 5% /usr DISK OK - free space: /usr 7209 MB (90% inode=96%)
After making sure that the check works, we can now define a command and the corresponding service definition. The command will pass the arguments specified in the actual service definition:
define command
{
command_name check_disk_nrpe
command_line $USER1$/check_disk -H "$HOSTADDRESS$"
-c "check_disk" -a $ARG1$ $ARG2$ $ARG3$
}
And, the actual service definition is as follows:
define service
{
use generic-service
host_name !localhost
hostgroup_name linux-servers
service_description Disk space on /usr
check_command check_disk_nrpe!10%!5%!/usr
}
This way you can define multiple partition checks without any modifications of the configuration file on the remote machines. Of course arguments can also be used for various plugins - for example, to be able to configure the load, user, and process thresholds directly in Nagios configuration rather than in each machine's NRPE settings.
Passing arguments to NRPE is a very useful feature. However, it comes at the price of a lower security level. When enabling it, you should keep in mind that malicious users are able to send parameters to NRPE, and so they may be able to inject shell commands and run arbitrary commands under the nagios account.
When using the functionality, it is recommended you have a strict source IP address policy in both the firewalls and the remote machine to limit the security issues related to the passing of arguments down to the actual check commands.
Our NRPE configuration should now be complete and working as expected. In case NRPE-based checks are not working properly, there are some steps that you can take to determine the root cause of the problem.
The first thing that should be checked is whether the Nagios server can connect to the NRPE process on the remote machine. We can check if NRPE accepts connections by using check_tcp from the Nagios plugins.
By default, NRPE uses port 5666. Assuming that we want to test connectivity of NRPE on 192.168.2.1, the following example shows how to check it:
$ /opt/nagios/plugins/check_tcp -H 192.168.2.1 -p 5666 TCP OK - 0.009 second response time on port 5666|time=0.008794s;;;0.000000;10.000000
If NRPE is not set up on the remote host, the plugin will return Connection refused. If the connection could not be established, the result will be No route to host. In these cases, you need to make sure that the NRPE server is working and the traffic that the TCP port NRPE is listening on is not blocked by the firewalls.
If the plugin did report a connection could be established, the next step is to try to run an invalid command. We can then check the output from the plugin and see what NRPE has reported.
The following is an example that assumes that dummycommand is not defined in the NRPE configuration on the remote machine:
$ /opt/nagios/plugins/check_nrpe -H 192.168.2.1 -c dummycommand NRPE: Command 'dummycommand' not defined
If you received a CHECK_NRPE: Error - Could not complete SSL handshake error or something similar, it means that NRPE is not configured to accept connections from your machine—either via the allowed_hosts option in the NRPE configuration or in the inetd configuration. In order to check this, search the system logs on the machine where NRPE is running for nrpe text.
For example, on most systems, to check if the NRPE is configured we need to execute the following command:
# grep nrpe /var/log/syslog /var/log/messages (...) ubuntu1 nrpe[3023]: Host 192.168.2.13 is not allowed to talk to us!
This indicates that your Nagios server is not added to the list of allowed hosts in the NRPE configuration. Add it in the allowed_hosts option and restart the NRPE process.
Another error message that could be returned by the check_nrpe command is CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. This message usually means that you have passed arguments or invalid characters in the command name and the NRPE server refused the request because of these.
Looking at the remote server's logs will usually provide more detailed information:
# grep nrpe /var/log/syslog /var/log/messages (...) ubuntu1 nrpe[3023]: Error: Request contained command arguments! ubuntu1 nrpe[3023]: Client request was invalid, bailing out...
In this situation, you need to make sure that you enable arguments or change the Nagios configuration to not to use arguments over NRPE.
Another possibility is that the check returns CHECK_NRPE: Socket timeout after 10 seconds or a similar message. In this case, the check command has not been completed within the configured time. You may need to increase command_timeout in the NRPE configuration.