Static

You have already seen static configuration in Chapter 2, where targets are provided directly in the prometheus.yml. It is useful if you have a small and simple setup that rarely changes. This might be your home network, a scrape config that is only for a local Pushgateway, or even Prometheus scraping itself as in Example 8-1.

scrape_configs:
 - job_name: prometheus
   static_configs:
    - targets:
      - localhost:9090

If you are using a configuration management tool such as Ansible, you could have its templating system write out a list of all the machines it knows about to have their Node exporters scraped, such as in Example 8-2.

scrape_configs:
 - job_name: node
   static_configs:
    - targets:
{% for host in groups["all"] %}
      - {{ host }}:9100
{% endfor %}

In addition to providing a list of targets, a static config can also provide labels for those targets in the labels field. If you find yourself needing this, then file SD, covered in “File”, tends to be a better approach.

The plural in static_configs indicates that it is a list, and you can specify multiple static configs in one scrape config, as shown in Example 8-3. While there is not much point to doing this for static configs, it can be useful with other service discovery mechanisms if you want to talk to multiple data sources. You can even mix and match service discovery mechanisms within a scrape config, though that is unlikely to result in a particularly understandable configuration.

scrape_configs:
 - job_name: node
   static_configs:
    - targets:
       - host1:9100
    - targets:
       - host2:9100

The same applies to scrape_configs, a list of scrape configs in which you can specify as many as you like. The only restriction is that the job_name must be unique.

File

File service discovery, usually referred to as file SD, does not use the network. Instead, it reads monitoring targets from files you provide on the local filesystem. This allows you to integrate with service discovery systems Prometheus doesn’t support out of the box, or when Prometheus can’t quite do the things you need with the metadata available.

You can provide files in either JSON or YAML formats. The file extension must be .json for JSON, and either .yml or .yaml for YAML. You can see a JSON example in Example 8-4, which you would put in a file called filesd.json. You can have as many or as few targets as you like in a single file.

[
  {
    "targets": [ "host1:9100", "host2:9100" ],
    "labels": {
      "team": "infra",
      "job": "node"
    }
  },
  {
    "targets": [ "host1:9090" ],
    "labels": {
      "team": "monitoring",
      "job": "prometheus"
    }
  }
]

Configuration in Prometheus uses file_sd_configs in your scrape config as shown in Example 8-5. Each file SD config takes a list of filepaths, and you can use globs in the filename.³ Paths are relative to Prometheus’s working directory, which is to say the directory you start Prometheus in.

scrape_configs:
 - job_name: file
   file_sd_configs:
    - files:
       - '*.json'

Usually you would not provide metadata for use with relabelling when using file SD, but rather the ultimate target labels you would like to have.

If you visit http://localhost:9090/service-discovery in your browser⁴ and click on show more, you will see Figure 8-1, with both job and team labels from filesd.json.⁵ As these are made up targets, the scrapes will fail, unless you actually happen to have a host1 and host2 on your network.

Figure 8-1. Service discovery status page showing three discovered targets from file SD

Providing the targets with a file means it could come from templating in a configuration management system, a daemon that writes it out regularly, or even from a web service via a cronjob using wget. Changes are picked up automatically using inotify, so it would be wise to ensure file changes are made atomically using rename, similarly to how you did in “Textfile Collector”.

Consul

Consul service discovery is a service discovery mechanism that uses the network, as almost all mechanisms do. If you do not already have a service discovery system within your organisation, Consul is one of the easier ones to get up and running. Consul has an agent that runs on each of your machines, and these gossip amongst themselves. Applications talk only to the local agent on a machine. Some number of agents are also servers, providing persistence and consistency.

To try it out, you can set up a development Consul agent by following Example 8-6. If you wish to use Consul in production, you should follow the official Getting Started guide.

hostname $ wget https://releases.hashicorp.com/consul/1.0.2/
    consul_1.0.2_linux_amd64.zip
hostname $ unzip consul_1.0.2_linux_amd64.zip
hostname $ ./consul agent -dev

The Consul UI should now be available in your browser on http://localhost:8500/. Consul has a notion of services, and in the development setup has a single service, which is Consul itself. Next, run a Prometheus with the configuration in Example 8-7.

scrape_configs:
 - job_name: consul
   consul_sd_configs:
    - server: 'localhost:8500'

Go to http://localhost:9090/service-discovery in your browser and you will see Figure 8-2, showing that the Consul service discovery has discovered a single target with some metadata, which became a target with instance and job labels. If you had more agents and services, they would also show up here.

Figure 8-2. Service discovery status page showing one discovered target, its metadata, and target labels from Consul

Consul does not expose a /metrics, so the scrapes from your Prometheus will fail. But it does still provide enough to find all your machines running a Consul agent, and thus should be running a Node exporter that you can scrape. I will look at how in “Relabelling”.

EC2

Amazon’s Elastic Compute Cloud, more commonly known as EC2, is a popular provider of virtual machines. It is one of several cloud providers that Prometheus allows you to use out of the box for service discovery.

To use it you must provide Prometheus with credentials to use the EC2 API. One way you can do this is by setting up an IAM user with the AmazonEC2ReadOnlyAccess policy⁶ and providing the access key and secret key in the configuration file, as shown in Example 8-8.

scrape_configs:
 - job_name: ec2
   ec2_sd_configs:
    - region: <region>
      access_key: <access key>
      secret_key: <secret key>

If you aren’t already running some, start at least one EC2 instance in the EC2 region you have configured Prometheus to look at. If you go to http://localhost:9090/service-discovery in your browser, you can see the discovered targets and the metadata extracted from EC2. __meta_ec2_tag_Name="My Display Name", for example, is the Name tag on this instance, which is the name you will see in the EC2 Console (Figure 8-3).

You may notice that the instance label is using the private IP. This is a sensible default as it is presumed that Prometheus will be running beside what it is monitoring. Not all EC2 instances have public IPs, and there are network charges for talking to an EC2 instance’s public IP.

Figure 8-3. Service discovery status page showing one discovered target, its metadata, and target labels from EC2

You will find that service discovery for other cloud providers is broadly similar, but the configuration required and metadata returned vary.

Choosing What to Scrape

The first thing you will want to configure is which targets you actually want to scrape. If you are part of one team running one service, you don’t want your Prometheus to be scraping every target in the same EC2 region.

Continuing on from Example 8-5, what if you just wanted to monitor the infrastructure team’s machines? You can do this with the keep relabel action, as shown in Example 8-9. The regex is applied to the values of the labels listed in source_labels (joined by a semicolon), and if the regex matches, the target is kept. As there is only one action here, this results in all targets with team="infra" being kept.

But for a target with a team="monitoring" label, the regex will not match, and the target will be dropped.

scrape_configs:
 - job_name: file
   file_sd_configs:
    - files:
       - '*.json'
   relabel_configs:
    - source_labels: [team]
      regex: infra
      action: keep

You can have multiple relabel actions in a relabel_configs; all of them will be processed in order unless either a keep or drop action drops the target. For example, Example 8-10 will drop all targets, as a label cannot have both infra and monitoring as a value.

scrape_configs:
 - job_name: file
   file_sd_configs:
    - files:
       - '*.json'
   relabel_configs:
    - source_labels: [team]
      regex: infra
      action: keep
    - source_labels: [team]
      regex: monitoring
      action: keep

To allow multiple values for a label you would use | (the pipe symbol) for the alternation operator, which is a fancy way of saying one or the other. Example 8-11 shows the right way to keep only targets for either the infrastructure or monitoring teams.

scrape_configs:
 - job_name: file
   file_sd_configs:
    - files:
       - '*.json'
   relabel_configs:
    - source_labels: [team]
      regex: infra|monitoring
      action: keep

In addition to the keep action that drops targets that do not match, you can also use the drop action to drop targets that do match. You can also provide multiple labels in source_labels; their values will be joined with a semicolon.⁷ If you don’t want to scrape the Prometheus jobs of the monitoring team, you can combine these as in Example 8-12.

scrape_configs:
 - job_name: file
   file_sd_configs:
    - files:
       - '*.json'
   relabel_configs:
    - source_labels: [job, team]
      regex: prometheus;monitoring
      action: drop

How you use relabelling is up to you. You should define some conventions. For example, EC2 instances should have a team tag with the name of the team that owns it, or all production services should have a production tag in Consul. Without conventions every new service will require special handling for monitoring, which is probably not the best use of your time.

If your service discovery mechanism includes health checking of some form, do not use this to drop unhealthy instances. Even when an instance is reporting as unhealthy it could be producing useful metrics, particularly around startup and shutdown.

Target Labels

Target labels are labels that are added to the labels of every time series returned from a scrape. They are the identity of your targets,⁸ and accordingly they should not generally vary over time as might be the case with version numbers or machine owners.

Every time your target labels change the labels of the scraped time series, their identities also change. This will cause discontinuities in your graphs, and can cause issues with rules and alerts.

So what does make a good target label? You have already seen job and instance, target labels all targets have. It is also common to add target labels for the broader scope of the application, such as whether it is in development or production, their region, datacenter, and which team manages them. Labels for structure within your application can also make sense, for example, if there is sharding.

Target labels ultimately allow you to select, group, and aggregate targets in PromQL. For example, you might want alerts for development to be handled differently to production, to know which shard of your application is the most loaded, or which team is using the most CPU time.

But target labels come with a cost. While it is quite cheap to add one more label in terms of resources, the real cost comes when you are writing PromQL. Every additional label is one more you need to keep in mind for every single PromQL expression you write. For example, if you were to add a host label which was unique per target, that would violate the expectation that only instance is unique per target, which could break all of your aggregation that used without(instance). This is discussed further in Chapter 14.

As a rule of thumb your target labels should be a hierarchy, with each one adding additional distinctiveness. For example, you might have a hierarchy where regions contain datacenters that contain environments that contain services that contain jobs that contain instances. This isn’t a hard and fast rule; you might plan ahead a little and have a datacenter label even if you only have one datacenter today.⁹

For labels the application knows about but don’t make sense to have as target labels, such as version numbers, you can expose them using info metrics as discussed in “Info”.

If you find that you want every target in a Prometheus to share some labels such as region, you should instead use external_labels for them as discussed in “External Labels”.

Replace

So how do you use relabelling to specify your target labels? The answer is the replace action. The replace action allows you to copy labels around, while also applying regular expressions.

Continuing on from Example 8-5, let’s say that the monitoring team was renamed to the monitor team and you can’t change the file SD input yet so you want to use relabelling instead. Example 8-13 looks for a team label that matches the regular expression monitoring (which is to say, the exact string monitoring), and if it finds it, puts the replacement value monitor in the team label.

Example 8-13. Using a replace relabel action to replace team="monitoring” with team="monitor”

scrape_configs:
 - job_name: file
   file_sd_configs:
    - files:
       - '*.json'
   relabel_configs:
    - source_labels: [team]
      regex: monitoring
      replacement: monitor
      target_label: team
      action: replace

That’s fairly simple, but in practice having to specify replacement label values one by one would be a lot of work for you. Let’s say it turns out that the problem was the ing in monitoring, and you wanted relabelling to strip any trailing “ings” in team names. Example 8-14 does this by applying the regular expression (.*)ing, which matches all strings that end with ing and puts the start of the label value in the first capture group. The replacement value consists of that first capture group, which will be placed in the team label.

Example 8-14. Using a replace relabel action to remove a trailing ing from the team label

scrape_configs:
 - job_name: file
   file_sd_configs:
    - files:
       - '*.json'
   relabel_configs:
    - source_labels: [team]
      regex: '(.*)ing'
      replacement: '${1}'
      target_label: team
      action: replace

If one of your targets does not have a label value that matches, such as team="infra", then the replace action has no effect on that target, as you can see in Figure 8-4.

Figure 8-4. The ing is removed from monitoring, while the infra targets are unaffected

A label with an empty value is the same as not having that label, so if you wanted to you could remove the team label using Example 8-15.

Example 8-15. Using a replace relabel action to remove the team label

scrape_configs:
 - job_name: file
   file_sd_configs:
    - files:
      - '*.json'
   relabel_configs:
    - source_labels: []
      regex: '(.*)'
      replacement: '${1}'
      target_label: team
      action: replace

Note

All labels beginning with __ are discarded at the end of relabelling for target labels, so you don’t need to do this yourself.

Since performing a regular expression against the whole string, capturing it, and using it as the replacement is common, these are all defaults. Thus you can omit them,¹⁰ and Example 8-16 will have the same effect as Example 8-15.

Example 8-16. Using the defaults to remove the team label succinctly

scrape_configs:
 - job_name: file
   file_sd_configs:
    - files:
       - '*.json'
   relabel_configs:
    - source_labels: []
      target_label: team

Now that you have more of a sense of how the replace action works, let’s look at a more realistic example. Example 8-7 produced a target with port 80, but it’d be useful if you could change that to port 9100 where the Node exporter is running. In Example 8-17 I take the address from Consul and append :9100 to it, placing it in the __address__ label.

Example 8-17. Using the IP from Consul with port 9100 for the Node exporter

scrape_configs:
 - job_name: node
   consul_sd_configs:
    - server: 'localhost:8500'
   relabel_configs:
    - source_labels: [__meta_consul_address]
      regex: '(.*)'
      replacement: '${1}:9100'
      target_label: __address__

Tip

If relabelling produces two identical targets from one of your scrape configs, they will be deduplicated automatically. So if you have many Consul services running on each machine, only one target per machine would result from Example 8-17.

scrape_configs:
 - job_name: consul
   consul_sd_configs:
    - server: 'localhost:8500'
   relabel_configs:
    - source_labels: [__meta_consul_address]
      regex: '(.*)'
      replacement: '${1}:9100'
      target_label: __address__
    - source_labels: [__meta_consul_node]
      regex: '(.*)'
      replacement: '${1}:9100'
      target_label: instance

scrape_configs:
 - job_name: ec2
   ec2_sd_configs:
    - region: <region>
      access_key: <access key>
      secret_key: <secret key>
   relabel_configs:
    - source_labels: [__meta_ec2_tag_service]
      target_label: job
    - regex: __meta_ec2_public_tag_monitor_(.*)
      replacement: '${1}'
      action: labelmap

scrape_configs:
 - job_name: node
   consul_sd_configs:
    - server: 'localhost:8500'
   relabel_configs:
    - source_labels: [__meta_consul_tags]
      regex:  '.*,prod,.*'
      action: keep

scrape_configs:
 - job_name: node
   consul_sd_configs:
    - server: 'localhost:8500'
   relabel_configs:
    - source_labels: [__meta_consul_tags]
      regex:  '.*,(prod|staging|dev),.*'
      target_label: env

metric_relabel_configs

In addition to relabelling being used for its original purpose of mapping service discovery metadata to target labels, relabelling has also been applied to other areas of Prometheus. One of those is metric relabelling: relabelling applied to the time series scraped from a target.

The keep, drop, replace, and labelmap actions you have already seen can all be used in metric_relabel_configs as there are no restrictions on which relabel actions can be used where.¹⁴

There are two cases where you might use metric relabelling: when dropping expensive metrics and when fixing bad metrics. While it is better to fix such problems at the source, it is always good to know that you have tactical options while the fix is in progress.

Metric relabelling gives you access to the time series after it is scraped but before it is written to storage. The keep and drop actions can be applied to the __name__ label (discussed in “Reserved Labels and __name__”) to select which time series you actually want to ingest. If, for example, you discovered that the http_request_size_bytes¹⁵ metric of Prometheus had excessive cardinality and was causing performance issues, you could drop it as shown in Example 8-23. It is still being transferred over the network and parsed, but this approach can still offer you some breathing room.

scrape_configs:
 - job_name: prometheus
   static_configs:
    - targets:
       - localhost:9090
   metric_relabel_configs:
    - source_labels: [__name__]
      regex: http_request_size_bytes
      action: drop

The labels are also available, as mentioned in “Cumulative Histograms”, you can also drop certain buckets (but not +Inf) of histograms and you will still be able to calculate quantiles. Example 8-24 shows this with the prometheus_tsdb_​compaction_duration_seconds histogram in Prometheus.

scrape_configs:
 - job_name: prometheus
   static_configs:
    - targets:
       - localhost:9090
   metric_relabel_configs:
    - source_labels: [__name__, le]
      regex: 'prometheus_tsdb_compaction_duration_seconds_bucket;(4|32|256)'
      action: drop

You could also use metric_relabel_configs to rename metrics, rename labels, or even extract labels from metric names.

scrape_configs:
 - job_name: misbehaving
   static_configs:
    - targets:
       - localhost:1234
   metric_relabel_configs:
    - regex: 'node_.*'
      action: labeldrop

Label Clashes and honor_labels

While labeldrop can be used when an exporter incorrectly presumes it knows what labels you want, there is a small set of exporters where the exporter does know the labels you want. For example, metrics in the Pushgateway should not have an instance label, as was mentioned in “Pushgateway”, so you need some way of not having the Pushgateway’s instance target label apply.

But first let’s look at what happens when there is a target label with the same name as an instrumentation label from a scrape. To avoid misbehaving applications interfering with your target label setup, it is the target label that wins. If you had a clash on the job label, for example, the instrumentation label would be renamed to exported_job.

If instead you want the instrumentation label to win and override the target label, you can set honor_labels: true in your scrape config. This is the one place in Prometheus where an empty label is not the same thing as a missing label. If a scraped metric explicitly has an instance="" label, and honor_labels: true is configured, the resultant time series will have no instance label. This technique is used by the Pushgateway.

Aside from the Pushgateway, honor_labels can also come up when ingesting metrics from other monitoring systems if you do not follow the recommendation in Chapter 11 to run one exporter per application instance.

Now that you understand service discovery, you’re ready to look at monitoring containers and how service discovery can be used with Kubernetes.