You can confirm whether or not your RabbitMQ installation is operating as a cluster by running the cluster status command:

rabbitmqctl cluster_status
Cluster status of node 'rabbit@tc-ost1' ...
[{nodes,[{disc,['rabbit@tc-ost1']}]},
 {running_nodes,['rabbit@tc-ost1']},
 {partitions,[]}]
...done.

In the preceding output, note that in the running_nodes section, there is only one node listed. This is an indication that we are not running clustered RabbitMQ. For details on how to cluster RabbitMQ, refer to the latest Rabbit documentation at https://www.rabbitmq.com/clustering.html.

In addition to clustering Rabbit, you should also consider using mirrored queues. OpenStack leverages RabbitMQ queues, and by default, queues in Rabbit are located on a single node even if you are running a Rabbit cluster. Since the queue remains on one node by default, if you lose that node, you will lose the messages in that queue. The answer to this problem is to leverage mirrored queues. When enabled, these queues will be mirrored across multiple nodes. If the master node goes down for some reason, one of the node mirrors can take over for that queue.

To set up mirrored queues, make sure you are running RabbitMQ in a cluster mode. Once your cluster is set up, you need to set the high-availability policy for all non autogenerated queues. The following command will accomplish this:

rabbitmqctl set_policy ha-all '^(?!amq.).*' '{"ha-mode": "all"}'

Once the policy has been set, you will need to configure each OpenStack service to leverage mirrored, high-availability queues. Locate the Rabbit section in the configuration file for each service and include the following values:

rabbit_hosts=rabbit1:5672,rabbit2:5672
rabbit_durable_queues=true
rabbit_ha_queues=true

More information about mirrored queues can be found in the Rabbit HA guide located at https://www.rabbitmq.com/ha.html.