Service objects offer some directives to control or specify whether, which, and how the notifications should be sent out.

Notification options can also be customized inside contact definitions. Let's look at various available directives:

Escalation is basically if person1 receives an alert and the problem continues to be there for x number of minutes then a notification should be sent (escalated) to person2.

Escalation is another aspect of notification configuration and can be done using the hostescalation and serviceescalation Icinga objects. These objects are used to define escalation paths to various people. This path can go on as long as we have configured it.

The escalations work when there is notification_interval defined in the service definition, which will re-notify contacts after the defined interval. Escalation logic kicks in when the serviceescalation object is defined. In the object definition, we define the contacts that should be notified at the nth renotification.

For example, we have three contacts defined: onduty, techlead, and manager. We want the onduty contact to be notified immediately, techlead if the check does not recover within 15 minutes, then notify the techlead if it does not recover within the next 15 minutes (30 minutes from the start), and finally to notify the manager contact if it does not recover within the next 15 minutes (45 minutes from start). For such a scenario, we need to set the notification_interval directive to 15 minutes (assuming the length of a time unit is one minute in the main configuration) in a service object definition, so that Icinga will retrigger the notification every 15 minutes until the check recovers. Now, we also need to define two serviceescalation objects as follows (we will use the localhost host and the HTTP service as examples; this can be used in the same way for remote hosts too):

define serviceescalation {
    host_name               localhost
    service_description     HTTP
    first_notification      2
    last_notification       3
    contacts                techlead
}

define serviceescalation {
    host_name               localhost
    service_description     HTTP
    first_notification      4
    last_notification       0
    contacts                manager
}

The preceding escalation definitions have the first_notification and last_notification directives, which determine the number of notifications for which the given escalation is valid. The first_notification directive specifies n for nth notification, with which this escalation becomes valid. The last_notification directive specifies n for nth notification, with which the escalation becomes ineffective. Escalation being valid implies that the contacts specified in the escalation will be notified. When the service goes CRITICAL, the onduty contact is immediately notified as the first notification.

For the first escalation definition, escalation becomes valid with the second notification (first_notification is 2) for the service and will become ineffective after the third notification (last_notification is 3) has been sent out. So, the second and the third notifications will be sent to the techlead contact after 15 and 30 minutes each.

For the second escalation definition, escalation becomes valid with the fourth notification (first_notification is 4) being sent out for the service and will never become ineffective (last_notification is 0). So the fourth notification (after 45 minutes) will go to the manager contact and will be notified every 15 minutes until the service check recovers.

The following table puts together a timeline for this example:

A similar configuration can be made for host escalations too:

define hostescalation {
    host_name               localhost
    first_notification      2
    last_notification       3
    contacts                techlead
}

define hostescalation {
    host_name               localhost
    first_notification      4
    last_notification       0
    contacts                manager
}

Let's take a quick look at various other directives offered by escalation objects to control notifications:

This covers most of the configuration options that Icinga provides to customize the behavior of notifications.