Prometheus logging is very terse when compared with most current software. This is a very conscious effort from the part of the Prometheus maintainers, as extraneous logging can lead to performance issues. Additionally, supporting different log streams (such as application logs, access logs, and slow query logs) without just writing it all to standard output - and thus spamming the application log with other types of logs - would force Prometheus to explicitly support writing to files, which is undesirable in cloud-native environments. Having said that, you can configure Prometheus to increase application log verbosity by setting the --log.level flag. As an example, failed scrapes are considered normal operating behavior and, as such, do not show up in the logs; however, they can be recorded by increasing the log verbosity to the debug log level.

The Prometheus instance in the test environment for this chapter is already configured with the log level set to debug. You can confirm this by running the following:

vagrant@prometheus:~$ sudo systemctl cat prometheus.service

The relevant section should have the following flags set:

ExecStart=/usr/bin/prometheus 
    --log.level=debug 
    --config.file=/etc/prometheus/prometheus.yml 
    --storage.tsdb.path=/var/lib/prometheus/data 
    --web.console.templates=/usr/share/prometheus/consoles 
    --web.console.libraries=/usr/share/prometheus/console_libraries

So, now we can see what happens when a scrape fails. To make this happen, we can stop the node_exporter service on the test environment and have a look at the Prometheus logs:

vagrant@prometheus:~$ sudo service node-exporter stop
vagrant@prometheus:~$ sudo journalctl -fu prometheus | grep debug
Feb 23 15:28:14 prometheus prometheus[1438]: level=debug ts=2019-02-23T15:28:14.44856006Z caller=scrape.go:825 component="scrape manager" scrape_pool=node target=http://prometheus:9100/metrics msg="Scrape failed" err="Get http://prometheus:9100/metrics: dial tcp 192.168.42.10:9100: connect: connection refused"
Feb 23 15:28:29 prometheus prometheus[1438]: level=debug ts=2019-02-23T15:28:29.448826505Z caller=scrape.go:825 component="scrape manager" scrape_pool=node target=http://prometheus:9100/metrics msg="Scrape failed" err="Get http://prometheus:9100/metrics: dial tcp 192.168.42.10:9100: connect: connection refused"