Although monitoring itself is a very broad term and as a job may include a lot of activities, there are basically two sides to what a web service monitoring constitutes. Graphing values from sensors is important as a tool to make strategic decisions based on visually apparent trends and also a post factum investigation and sometimes even as a forensics tool. The other side is alerting, which allows administrators to react to incidents as quickly as possible, preventing business consequences. All major monitoring solutions include alerting subsystems and even the toy monitoring script that we built earlier in this chapter is basically a simple alerter.
The http_stub_status
module provides scarce information, but it can still be used to quickly react to incidents. These are some good values that can be used to detect unusual conditions requiring immediate attention:
You can use any number of complex alert conditions using these three and also invent more metrics relevant to your particular business. The method to create such alerts is specific to the monitoring system you decided to choose. These are the example alerts we use inside ServerDensity:
munin.conf
. This file contains a list of all hosts that are under monitoring by this very master. Adding a threshold for a metric available on a host is as simple as this:[www1.example.com] nginx_status.total.warning 0:10 nginx_status.total.critical 0:20
The values are specified as ranges; that is, in the earlier case, the warning would be generated only when the nginx.status.total
metrics (which is the total number of current connections) will leave the range from 0 to 10.
Munin is able to send notifications about the alerts via e-mail, via syslog, via piping to external programs, or via Nagios, which is another popular monitoring system with a more sophisticated alerting subsystem. Configuration of the notification settings is easy enough. See the online Munin guide at http://guide.munin-monitoring.org/en/latest/tutorial/alert.html.
3.144.31.163