We have been discussing various performance tuning tips that would allow us to create a more scalable broker setup. However, in order to be able to observe how our setup behaves in various scenarios, it is not sufficient to do only partial performance measurements using PerfTest, a custom performance tool, or even a third-party performance-testing solution. In a production environment, we would typically want to have a real-time monitoring solution that would allow us to observe how our broker behaves at any point in time enabling us to take measures as fast as possible when something goes wrong with our RabbitMQ instances.
The RabbitMQ management plugin provides you with a good real-time overview of the resource utilization of the instances of a cluster and the message rates per queue or exchange. However, we may want to have a central monitoring infrastructure that monitors all the parts of our infrastructure, including the message broker. Moreover, we may want to make use of advanced features provided by a typical monitoring solution such as the ability to receive notifications (e-mail, SMS, and so on) when something wrong happens with the broker such as a failed RabbitMQ instance or an exceeded memory / CPU / free disk threshold. For this reason, we can leverage a monitoring solution to do the job.
We will briefly discuss the capabilities provided by the management plugin, and then we will see how to monitor RabbitMQ using Nagios, Monit, or Munin assuming that we are running our RabbitMQ instances in a Linux environment.
When you navigate to the Overview tab of the RabbitMQ management web interface and click on a node, you can observe the resource consumption by this node in real time under the Statistics section:
You can observe the number of file descriptors or socket descriptors that are used, Erlang processes, and memory currently used by the message broker along with the current free disk space. On the same page, you can observe more information about the distribution of memory among the different components of the message broker under the Memory details section by clicking on the Update button first in order to take a memory snapshot:
You can also check the message rates from Queues and Exchanges by clicking on a particular queue or exchange.
Nagios is an open source system monitoring application that provides a number of plugins to extend its capabilities along with various types of integrations with different network protocols and applications. In order to install Nagios in Ubuntu, you can use the following command:
sudo apt-get update sudo apt-get install nagios3 nagios-nrpe-plugin
When prompted during the installation, specify a proper password for the Nagios administrative panel. To check whether the Nagios service is running, execute the following command:
sudo service nagios3 status
You should now be able to log in to the Nagios administrative interface from http://localhost/nagios3 and provide the nagiosadmin user along with the password that you specified during the installation. The next thing to do is to install some Nagios health checks (or write your own if the installed ones are not proper):
git clone https://github.com/jamesc/nagios-plugins-rabbitmq sudo chown -R nagios:nagios nagios-plugins-rabbitmq/ mv nagios-plugins-rabbitmq /usr/lib/nagios/plugins/ sudo apt-get install libnagios-plugin-perl sudo apt-get install libnagios-object-perl apt-get install perl-Nagios-Plugin apt-get install libreadonly-xs-perl sudp perl -MCPAN -e 'install Bundle::LWP' perl -MCPAN -e 'install Monitoring::Plugin' sudo cp -R /usr/share/perl/5.14.2/CPAN/LWP/ /etc/perl/ sudo cpan install JSON
In short, the process described in the preceding commands is as follows:
- We download the sources of the health checks from the nagios-plugins-rabbitmq GitHub repository. You can see the available checks (provided as Perl scripts) under the
nagios-plugins-rabbitmq/scriptsdirectory; they use the RabbitMQ management REST API. - We change the permissions of the sources and move them to the Nagios plugins directory.
- We install the Monitoring:Plugin Perl plugin along with the additional dependencies that is needed in order to write plugins for Nagios under Perl; this is required as the RabbitMQ health checks that we downloaded are provided as Perl scripts that depend on this library.
To verify that you can run a check, you can execute the following:
cd nagios-plugins-rabbitmq/scripts ./check_rabbitmq_server
If you are prompted to provide a hostname, then check whether your compiles are fine. You need to define a particular command using this script in the commands.cfg configuration file of Nagios:
sudo vim /etc/nagios3/commands.cfg
define command {
command_name check_rabbitmq_server
command_line /usr/lib/nagios/plugins/nagios-plugins-rabbitmq/scripts/check_rabbitmq_server -H localhost --port=15672 -u guest -p guest
}You can now restart the service with the following:
sudo service nagios3 restart
When you navigate to the Configuration menu under the System section, select Commands from the dropdown and click on view; you should see the you check-rabbitmq-server command in the list. In a similar way, you can define the other already provided RabbitMQ checks if you need them to monitor.
You can create a service definition that uses the command and allows you to specify which groups you would like to notify, for example, in case the RabbitMQ server goes down. You can do this with the other RabbitMQ health checks as well. You can also write your own health checks for RabbitMQ, for example, using Java and RabbitMQ management REST API or the rabbitmqctl utility.
Monit is a Unix utility to monitor processes. You can also use it to monitor the RabbitMQ instance process in a pretty straightforward manner. Monit requires a pid file that stores the process ID for the currently running process. In the earlier versions of the rabbitmq-server script init script under /etc/init.d, you had to add the creation and deletion of this pid file manually upon the service startup/shutdown. However, the later versions of RabbitMQ store a pid file for the RabbitMQ Erlang process under the /var/run/rabbitmq/pid directory.
In order to install Monit, execute the following command:
sudo apt-get install monit
You can then add the following configuration to the /etc/monit/monitrc file in order to monitor the RabbitMQ process from the localhost:
set httpd port 2812 and use address localhost allow localhost allow @monit allow @users readonly CHECK PROCESS rabbitmq-server WITH PIDFILE /var/run/rabbitmq/pid GROUP rabbitmq START PROGRAM "/usr/sbin/service rabbitmq-server start" STOP PROGRAM "/usr/sbin/service rabbitmq-server stop" IF DOES NOT EXIST FOR 3 CYCLES THEN RESTART IF FAILED PORT 5672 4 TIMES WITHIN 6 CYCLES THEN RESTART
You can start monit in the background with the following command:
sudo service monit start sudo monit
You can then check the status of the monited processes (including the Erlang process of RabbitMQ) using the following command:
sudo monit status
You should see an output similar to the following:
Process 'rabbitmq-server' status Running monitoring status Monitored pid 1046 parent pid 1039 uptime 6d 12h 6m children 2 memory kilobytes 13680 memory kilobytes total 14484 memory percent 0.6% memory percent total 0.7% cpu percent 0.0% cpu percent total 0.0% port response time 0.000s to localhost:5672 [DEFAULT via TCP] data collected Mon, 31 Aug 2015 01:47:59
You can use Munin as a nice alternative to Nagios for the monitoring. The following command installs Munin in Ubuntu (note that the Apache HTTP server must also be installed):
sudo apt-get install apache2 sudo apt-get install munin
You must then edit the Munin configuration:
vim /etc/munin/munin.conf
Uncomment the following and change the value of the
htmldir attribute to /var/www/munin:
dbdir /var/lib/munin htmldir /var /www/munin logdir /var/log/munin rundir /var/run/munin tmpldir /etc/munin/templates
Add the following to the Munin configuration file in order to enable monitoring on the localhost:
[MuninMonitor]
address 127.0.0.1
use_node_name yesOpen the Munin Apache configuration and change the alias to allow external connections:
sudo vim /etc/munin/apache.conf
Alias /munin /var/www/munin
<Directory /var/www/munin>
Order allow,deny
#Allow from localhost 127.0.0.0/8 ::1
Allow from all
Options NoneCreate the /var/www/munin directory, change permissions to the munin user and group, and finally restart the apache2 and munin-node services:
sudo mkdir /var/www/munin sudo chown munin:munin /var/www/munin sudo service munin-node restart sudo service apache2 restart
If you navigate to http://localhost/munin/, you should be able to see the Munin administrative interface:
Now, we need to install the Munin RabbitMQ set of plugins. To do so, execute the following commands in order to download the Munin plugins directly to the Munin plugins directory:
cd /etc/munin/plugins/ sudo git clone https://github.com/ask/rabbitmq-munin sudo cp rabbitmq-munin/* .
Add the following configuration to the /etc/munin/plugin-conf.d/munin-node file:
sudo vim /etc/munin/plugin-conf.d/munin-node [rabbitmq_connections] user root [rabbitmq_consumers] user root [rabbitmq_messages] user root [rabbitmq_messages_unacknowledged] user root [rabbitmq_messages_uncommitted] user root [rabbitmq_queue_memory] user root
Finally, restart the munin-node service and check whether you have the munin plugins available from the administrative interface:
sudo service munin-node restart
