We covered how alerting rules work in Prometheus in Chapter 9, Defining Alerting and Recording Rules, but those, by themselves, aren't all that useful. As we mentioned previously, Prometheus delegates notification handling and routing to external systems through a Webhook-style HTTP interface. This is where Alertmanager comes in.

Alertmanager is responsible for accepting the alerts generated from Prometheus alerting rules and converting them into notifications. The latter can take any form, such as email messages, chat messages, pages, or even Webhooks that will then trigger custom actions, such as logging alerts to a data store or creating/updating tickets. Alertmanager is also the only component in the official stack that distributes its state across instances so that it can keep track of things such as which alerts were already sent and which ones are silenced.