Despite the intense debate regarding push versus pull and the deliberate decision of using pull in the Prometheus server design, there are some legitimate situations where push is more appropriate.

One of those situations is batch jobs, though, for this statement to truly make sense, we need to clearly define what is considered a batch job. In this scope, a service-level batch job is a processing workload not tied to a particular instance, executed infrequently or on a schedule, and as such is not always running. This kind of job makes it very hard to generate successful scrapes if instrumented, which, as discussed previously in Chapter 5, Running a Prometheus Server, results in metric staleness, even if running for long enough to be scraped occasionally.

There are alternatives to relying on pushing metrics; for example, by using the textfile collector from node_exporter as described previously. Nevertheless, this option does not come without downsides. If the workload is not specific to a particular instance, you'll end up with multiple time series plus the cleanup logic of the textfile collector files, unless the lifetime of the metric matches the lifetime of the instance, which can then work out well in practice.

As a last resort, you have Pushgateway, which we'll be covering next.