blackbox_exporter is one of the most peculiar of all the currently available exporters in the Prometheus ecosystem. Its usage pattern is ingenious and usually, newcomers are puzzled by it. We'll be going to dive into this exporter with the hope of making its use as straightforward as possible.

The blackbox_exporter service exposes two main endpoints:

• /metrics: Where its own metrics are exposed
• /probe: It is the query endpoint that enables blackbox probes, returning their results in Prometheus exposition format

Besides the two previous endpoints, the / of the service also provides valuable information, including logs for the probes performed. This endpoint is available in the static infrastructure test environment at http://192.168.42.11:9115.

The blackbox exporter supports probing endpoints through a wide variety of protocols natively, such as TCP, ICMP, DNS, HTTP (versions 1 and 2), as well as TLS on most probes. Additionally, it also supports scripting text-based protocols such as IRC, IMAP, or SMTP by connecting through TCP and configuring what messages should be sent and what responses are expected; even plain HTTP would be possible to script but, as HTTP probing is such a common use case, it's already built in.

Having said that, this exporter doesn't cover all the blackbox-style monitoring needs. For those cases, writing your own exporter might be needed. As an example, you can't use blackbox_exporter to test a Kafka topic end to end, so you might need to look for an exporter able to produce a message to Kafka and then consume it back:

The /probe endpoint, when hit with an HTTP GET request with the parameters module and target, it executes the specified prober module against the defined target, and the result is then exposed as Prometheus metrics:

For example, a request such as http://192.168.42.11:9115/probe?module=http_2xx&target=example.com will return something like the following snippet (a couple of metrics were discarded for briefness):

# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.454460181
# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 0
# HELP probe_http_status_code Response HTTP status code
# TYPE probe_http_status_code gauge
probe_http_status_code 200
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 1

A quirk to be aware of when using blackbox_exporter is that the up metric does not reflect the status of the probe, but merely that Prometheus can reach the exporter. As can be seen in the previous metrics output, there is a probe_success metric that represents the status of the probe itself. This means that it is common for the up metric to appear healthy, but the probe might be failing, which is a common source for confusion.