Alerting is a critical component in any monitoring stack. In the Prometheus ecosystem, alerts and their subsequent notifications are decoupled. Alertmanager is the component that handles these alerts. In this chapter, we'll be focusing on converting alerts into useful notifications using Alertmanager. From reliability to customization, we'll delve into the inner workings of the Alertmanager service, providing the required knowledge to configure, troubleshoot, and customize all the options available. We'll make sure that concepts such as alert routing, silencing, and inhibition are clear so that you can decide how to implement them in your own stack.

Since Alertmanager is a critical component, high availability will also be explored, and we will also explain the relationship between Prometheus and Alertmanager. We will customize notifications and learn how to build and use reusable templates so that notifications are appropriate and carry accurate information when they reach their destination. We will finish this chapter by learning how to monitor the monitoring system and, more importantly, learning how to be alerted when the system is partially or completely down.

We will explore the following topics in this chapter:

• Test environment for this chapter.
• Alertmanager fundamentals.
• Alertmanager configuration.
• Common Alertmanager notification integrations.
• Customizing your alert notifications.
• Who watches the Watchmen?