Dispatching alert groups to the notification pipeline

Whenever an alerting rule is triggered, Prometheus will send an alert in the form of a JSON payload to the Alertmanager API, and it will keep sending updates at each evaluation interval of that rule or every minute (configurable through the --rules.alert.resend-delay flag), whichever is longer. When alerts are received by Alertmanager, they go through the dispatching step, where they will be grouped using one or more of the alert labels, such as alertname. We're going to discuss more about this in the Alertmanager configuration section, later in this chapter. This allows you to sort alerts into categories, which can reduce the number of notifications that are sent as multiple alerts in the same category and are grouped together in a single notification, which will then trigger the notification pipeline:

Figure 11.3: Alertmanager interface grouping alerts by alertname

When running multiple Prometheus instances with the same configuration (a common setup when pursuing high availability/redundancy), alerting rules for the same condition won't necessarily trigger at the exact same time. Alertmanager accounts for this situation by having a configurable time interval. It will wait before doing anything else so that similar alerts can be grouped together and thus avoid sending multiple notifications for a single type of problem.

This grouping is done in parallel across all the user-specified criteria. Each group will then trigger the notification pipeline, which we'll cover next.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.139.97.157