We can configure alerts in Grafana on dashboards. Grafana provides a query-based alerting system:
- Run Grafana in your local browser using the http://localhost:3000/?orgId=1 URL to access the Grafana UI.
- Click on Alerting, then Notification channels, and then New channels:
- Add the data for the new channel. We can select from a wide range of notifications in Grafana, including email, Slack, Microsoft teams, Webhook, or PagerDuty:
The New Notification Channel form will contain the following tags:
-
- Name: This is the name of the notification channel. This value must be unique.
- Description: Here, we add additional information that we might want to include for documentation purposes.
- Type: Select Email, and enter the email address of the recipient. We can enter multiple email addresses, separated by commas.
- After we create the notification channel, let's define our metrics. To do this, click on the Dashboard, go to New dashboard, and then click on Graph. Under Panel Title, click on Edit:
- We then need to define our metrics. We can use any data source that is supported by Grafana. In our example, as we have already configured Prometheus, we are going to use this as the data source. After we click on Edit, Grafana will ask you to select a Data Source and Metrics. In this case, we have selected storage_operation_duration_seconds_sum as the metric. As you can see, it has created a time-series graph:
- Click on the General tab and define a title for your alert and description:
- Click on the Alert tab to set the definition of the alert. We can set the condition and the threshold value. We also need to define the frequency of the alerts as follows:
- After completing the alert definition, save the alert and execute the Test Rule. Then, verify whether it is working. For email alerts, we need to configure SMTP:
