As mentioned in earlier chapters, Icinga maintains abstraction over the internal implementation of check plugins and how the status checks are performed. It only knows what it has to monitor and what commands it has to use to get the status of the services that are being monitored. It does not have details about any internal mechanism for performing status checks; it relies on external programs and commands to do the dirty work.

These external programs are called plugins. A plugin can be any executable (a compiled program or a script), which can be run from the command line. Icinga will execute the plugin configured for each service check and determine the status of the service from the exit status of the plugin. Icinga is not aware of the internal logic or implementation of the plugin itself. Once Icinga has the status of the service check, it calls notification commands or event handlers based on its configuration.

Note that the plugins can also be used to perform host checks. Each host object definition also has a check_command directive, which refers to the Icinga command object that is used to determine the status of the host. This check command can refer to a plugin that implements the checking of the host status and reports the status back to Icinga. So depending on whether the plugin is used for a host status check or service status check, it sets the relevant macros to values as reported by the plugins. This will be explained shortly.

Icinga's default installation comes with a number of plugins, and a few others are provided by the nagios-plugins package. Other plugins for performing more specific checks are available as separate packages, for example, nagios-plugins-dhcp. Most standard check plugins provide a --help command-line option to see the usage and options provided by the plugin.

Writing custom plugins

It is also possible to write custom check plugins, which can be scripts (bash, python, and so on) or compiled programs (C, Java, and so on). These plugins need to comply with a minimum two things:

• The exit codes, 0=OK, 1=WARNING, 2=CRITICAL, and 3=UNKNOWN, which are stored in the $HOSTSTATEID$ and $HOSTSTATE$ macros for host checks, and the $SERVICESTATEID$ and $SERVICESTATE$ macros for service checks
• At least one line of output written to STDOUT, which is stored in the $HOSTOUTPUT$ macro for host checks and the $SERVICEOUTPUT$ macro for service checks

Apart from these, the plugin can output the following:

• Multiline output written to STDOUT; Icinga will accept and store more lines of output in the $LONGHOSTOUTPUT$ or $LONGSERVICEOUTPUT$ macro.
• Output written to STDOUT is stored as performance data for other applications, which need to be processed. It is stored in the $HOSTPERFDATA$ or $SERVICEPERFDATA$ macro. (The performance data is used for various graphing and reporting purposes among other possible use cases.)

  The format of the output that will be parsed into these macros is as follows:

  

  Plugin output interpretation (The fonts of the lines are changed for representational purpose only)

The line seen in the bold style is stored in the $HOSTOUTPUT$ or $SERVICEOUTPUT$ macro and the rest of the parts are optional. The lines seen in the italic style are stored in the $LONGHOSTOUTPUT$ or $LONGSERVICEOUTPUT$ macro. The lines in normal text format are stored in the $HOSTPERFSDATA$ or $SERVICEPERFDATA$ macro. The lines of output and performance data output are separated by pipe (|) symbol.

For example, if we have a check script that writes the following output to STDOUT:

Sample output of the check_disk check plugin

The bold part of the output will be stored in the $SERVICEOUTPUT$ macro and the part seen in normal font will be stored in the $SERVICEPERFDATA$ macro. Since there are no long text lines between first line of output and the pipe symbol, the $LONGSERVICEOUTPUT$ macro for the long output will have an empty value.

Another example of a check script output is as follows:

Full sample output of the check_disk check plugin

Icinga will only read the first 8 KB of output that a plugin writes to STDOUT. This is to prevent overflow of data reported to Icinga.

Note

All plugins that the Icinga server executes are executed as the operating system user that is configured in the main configuration file icinga.cfg by the value of the icinga_user directive, which is icinga by default. We have to make sure to have proper permissions in place (make it executable for the icinga user, give sudo-permissions permissions if required, and so on).