Within a Cluster, each Queue will be mirrored. A mirrored queue consists of one Master and one or more slaves. Messages published to the queue are replicated to all slaves. Consumers are connected to the master regardless of which node they connect to, with slaves dropping messages that have been acknowledged by the master. Queue-mirroring therefore enhances availability. Note that in a mirrored-queue scenario, all the queues do all the work and the work is NOT DISTRIBUTED across nodes.

In the Event-Data Pipeline, we will have mirrored queues. In order to not load the system with too much work, each queue should be mirrored exactly n times, where n is defined as an exact number. This can be changed later on based on the feedback on the Event-Data Pipeline's usage and stability.

In RabbitMQ, we will make use of the ha-sync-batch-size Policy configuration to synchronize messages between nodes in batches to improve performance.

It is important to note that currently there is no eager synchronization of mirrored queues in RabbitMQ and everything happens asynchronously.

The Flow when queues are declared as HA is depicted here. Note that in Data Bus, since the mirror count will be one, the queue will be mirrored on exactly one node instead of all the nodes:

We need to choose a value for ha-sync-batch-size. For this, the following needs to be considered:

• The average queue message size.
• Setting a batch size that is too high could result in the nodes in the cluster missing the Tick Messages from other nodes.