Silencing

Silencing is a common concept in monitoring/alerting systems; it is how you can avoid alert notifications from going out in a time-capped way. It is often used to disable notifications during maintenance windows, or to temporarily suppress alerts of lower importance during incidents. Alertmanager takes this concept and supercharges it by taking advantage of the fact that alerts coming in usually have one or more differentiating labels: ones from the originating alerting rule expression, the alertname, the alert's label fields, from alert_relabel_configs, as well as the ones from the Prometheus external_labels. This means that any one of these labels (or a combination of them) can be used to temporarily disable notifications through either direct matching or through regular expression matching:

Figure 11.4: Creating a silence matching alertname=NodeExporterDown
You should be careful with regex matching as you can accidentally silence more than you expect. The Alertmanager web UI can help prevent this as it shows a preview of which firing alerts will be suppressed when creating new silences.

Silences are defined at runtime. They can be set using the Alertmanager web interface, amtool (the Alertmanager command-line interface, which is going to be presented shortly), or directly through the API. They can be set while an alert is firing, such as during an incident, or in advance so that planned maintenance doesn't spam the people doing on-call. It is not supposed to be a permanent solution for a firing alert, only a temporary measure; this is why creating a silence requires that you set an expiration date for it, and why the web UI only recognizes durations up to days.

Since the silencing step comes after inhibition, if you silence an alert that is triggering inhibition rules, it will continue to inhibit other alerts.

If the alert didn't match any of the silences, it will go through to the next step in the notification pipeline, which is the routing phase.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.226.28.197