As the number of services grows, it becomes harder and harder to tie everything together – both in terms of services being correctly configured to contact each other, as well as operators having visibility of how the system is behaving. A common solution to these problems is to implement a service discovery system that acts as registry and that can then be consulted by software clients, as well as the monitoring system.

Prometheus integrates seamlessly with a few mainstream service discovery systems and currently supports Consul, Nerve, and ServerSets. Integrating directly with discovery services allows Prometheus to always have an up-to-date view of what is running and where, allowing service instances to be monitored automatically as soon as they are created, up until they are destroyed.

Consul is by far the most popular, as it provides a full set of features to implement service discovery and powerful yet simple-to-use command-line tools and APIs, and is easy to scale. Let's use the following for our example:

scrape_configs:
  - job_name: 'consul_sd'
    consul_sd_configs:
      - server: http://consul.prom.inet:8500
        datacenter: dc1
    relabel_configs:
      - source_labels: [__meta_consul_service]
        target_label: job
      - source_labels: [job, __address__]
        regex: "consul;([^:]+):.+"
        target_label: __address__
        replacement: ${1}:9107

The preceding example translates into the following screenshot, where we can see not only the generated labels, but also the definitions of the targets:

The preceding example shows a working configuration for gathering data from all the available services registered within a Consul server, using relabel_configs to rewrite the target's job label to be the service name instead of the job_name. This means that every application instance registered in Consul would be automatically picked up as a scrape target and correctly assigned the proper job name. Additionally, the last relabel rule changes the target port to 9107 when the service is named Consul, thus changing the target from Consul itself to an exporter for it.