Nginx base distribution contains a simple module that provides access to several rather basic but useful counters inside Nginx that are very important to monitor on a regular basis. The module is named ngx_http_stub_status, and we will describe how to use it in monitoring.

This module is not compiled by default. To check if your Nginx distribution is compiled with the stub_status module, you can use this command:

$ nginx -V 2>&1 | fgrep -c http_stub_status_module
1
$

If you see 1, then this module is compiled and linked to your Nginx binary. Otherwise, you need to compile it using --with-http_stub_status_module parameter to configure script that is invoked during Nginx compilation.

Once you have this module available, you can use its directives (actually, there is only one) in nginx.conf. This is an example of stub-status sharing:

location /stub-status {
    stub_status;
}

This module belongs to the family of the so-called content-generating modules, which are able to directly process incoming HTTP requests and generate responses. The other main family of modules that can provide HTTP responses to clients are the upstream modules, which are more complex content generators consulting external resources for the actual data. While we spent a significant amount of time explaining upstreams in previous chapters, content generators were not given so much attention due to their relative simplicity. Some other interesting content-generating modules that are included in Nginx distribution are ngx_http_empty_gif and ngx_http_autoindex.

The earlier configuration will create a primitive web page as a response to the GET request with the URI of /stub-status. You can use any other URI as well. The page will contain several counters. Let's see an example and learn what those numbers mean. A fresh web server after a restart will generate this page:

This is not even a web page per se, but just a plain text HTTP response clearly intended to be processed by scripts and not by people:

$ curl -si http://localhost/stub-status | fgrep Content-Type
Content-Type: text/plain
$

Let's dive deeper into the values that you see in this output:

Most of these counters are also available as Nginx variables and can be used in the log_format configuration. The module exports these variables:

The global counts of connections and requests can be restored from the logs without any additional variables.

The only directive from this module is stub_status, and its usage pattern is obvious from the earlier example. You can specify it in one of the scopes; most probably, you will choose a location. Older versions of Nginx may require you to specify a parameter like ok or 1 for this directive.

While the simplest status content generator may only contain one line with this directive, it is highly recommended to make some additions.

This is the working practical example of stub_status configuration:

location /stub-status {
stub_status;  
    access_log   off;
    allow   $monitoring_ip;
    deny    all;
}

The access_log off directive will switch off logging for this location. You may want to comment out it while you debug but in the end only your monitoring system will make requests to this path from predefined IP addresses and at predefined intervals. The logging of this regular GET request-response pair will not be very useful while littering the logs with very redundant information.

The third and the fourth lines are about access control. While stub_status does not share any particularly confidential information, it may still be valuable to either competitors or malicious actors who plan to attack your infrastructure. A golden rule of not sharing anything by default works here.

To fully understand how we could use the data, let's write a simple manual alerting script using cron. As an example, we will specify the task as follows: once a minute check the requests rate and alert the administrator if the rate per minute exceeds a certain predefined threshold.

The code for the check is given later. It is a simple Perl script that should be called from a crontab with a time specification of */1 * * * *, which means running once every minute. It uses the PushBullet service to send alerts. PushBullet allows mobile apps and browser extensions to receive those notifications and has a simple HTTP API:

#! /usr/bin/perl
use strict;
use warnings;

use autodie;
use LWP::Simple;
use WWW::PushBullet;

my $stub_status_url = 'http://localhost/stub-status';
my $threshold = 2000; # requests per minute

my $data_file = '/var/run/req-rate';
my $pb = WWW::PushBullet->new( {
    apikey => '...PushBullet API key...',
});

BEGIN {
    # For LWP inside PushBullet
    $ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS} = 'Net::SSL';
}

open my $fh, '+<', $data_file;
my ($prev, $prev_ts) = split ' ', scalar <$fh>;

my $reqs;
if (get($stub_status_url) =~ /(d+)s+Reading: /s) {
    $reqs = $1;
}

my $now = time();

if ($prev && $reqs) {
    my $rate = ($reqs - $prev) / ($now - $prev_ts) * 60;
    if ($rate > $threshold) {
        $pb->push_note({ title => 'req rate alert', body => "Requests per minute: $rate" });
    }
}

seek $fh, 0, 0;
truncate $fh, 0;

say $fh "$reqs $now";

The script parses the output of the ngx_http_stub_status module with a simple regular expression and compares the global requests counter with the previous value, which it saves in a file. If the increase in the counter divided by the time passed exceeds a constant, it sends an alert. This is the model employed by all monitoring and alerting systems out there.

The next step for us is using the data from this module as a sensor in several monitoring systems that are available on the market. Most of the systems allow comprehensive graphs of all available counters and also custom alerts on events like a value exceeding a threshold of some kind.

Many hosted monitoring solutions, such as DataDog, Scout, New Relic or ServerDensity, support collecting data from the Nginx http_stub_status module as well.

Here is how ServerDensity displays data received from Nginx:

The earlier-mentioned graph will only contain two entities: current client connections and requests per second. You notice that those two come very close to each other, which means that there were no or little keep-alive connections that managed to pass over more than one request.