In the alert rule definition, there were two optional sections: labels and annotations. Labels define the identity of an alert and they can change according to the evaluation cycle they're in; if they do this, it will alter the alert identity. To demonstrate this point, we'll introduce the ALERTS metric, which tracks all active alerts and their labels. As we can see in the following figure, we have a label called alertstate, which tracks the alert state and transitions from pending to firing:

Something to keep in mind is the issue of using a sample value in a label. Although it is technically possible, it's also a very bad idea. Doing so will change the alert identity every time the value changes, and, as such, will always be restarting the defined for countdown, resulting in an alert that will never enter the firing state.

On the other hand, annotations do not belong to the identity of an alert, and, as such, are not stored in the ALERTS metric. These are useful to enrich the alert with more context and information. Annotations are also templated using the Go templating language, as we saw in our example. By using the {{ .Labels.instance }} template syntax, we are accessing the available alert labels, selecting the instance label, and using its value in the annotation description field. The value of the firing sample is available by using {{ .Value }}, if required.

The following snippet shows the alert rule in our example:

   annotations:
     description: "Node exporter {{ .Labels.instance }} is down."
     link: "https://example.com"

This will produce the following rendered result when firing:

       "annotations": {
           "description": "Node exporter prometheus:9100 is down.",
           "link": "https://example.com"
       },