Another important aspect of a Distributed Data Intensive System is reliable messaging. By reliable messaging, we mean that if a message publisher publishes a message, then we provide him with a guarantee that the message will not be lost during transit. It does not mean that the message is successfully consumed by the consumer. What it means is that the messaging system will not lose the message for an agreed SLA time duration. If the consumer is not able to consume the message within the SLA, it is up to the messaging system to decide what to do with the message. The message can be permanently deleted, put in a Dead Letter queue, or persisted outside of the messaging system.

Reliable delivery requirements also vary from system to system. There are usually three types of Reliability guarantees that an application would require, they are as follows:

• At-least-once delivery: This requirement states that a message published by a publisher should be delivered at least once to the consumer listening on the other side of the pipe. This is usually tied to a time unit. If the underlying messaging system does not respond with a positive Acknowledgement within a specified time, then the publisher assumes that the message is lost in transit and re-sends the message. Thus, a message may arrive at the consuming side twice. This pattern is the most common pattern used within the Distributed messaging system. One of the key things to properly implement such a pattern is that the consuming side should be idempotent to the messages. Thus, if they receive the same message multiple times, they should always handle it in exactly the same manner.
• At-most-once delivery: With this requirement, the publisher of the message publishes only once and does not wait for an acknowledgement from the underlying messaging systems. It is the messaging system's responsibility to deliver the message properly to the consumer. Usually such a pattern is not desirable as this means loss of data.
• Exactly-once delivery: With this pattern, the underlying messaging system goes the extra length to make sure that a message is delivered exactly once, neither more nor less, to the consumer. Usually this sort of guarantee requires a dual-handshake process and is the most expensive to implement. If you require exactly-once delivery, you will be better off calling the consumer directly over an HTTP protocol using the request-response mechanism.

All these patterns will be discussed in more detail in the coming chapters, with details on implementation of each pattern.

The reason we should always consider reliable messaging as part of the distributed system is because an architecture that is based on the notion of decoupling the resource-producer from the resource-consumer scales almost instantly and can be designed to be fast, durable, distributed, fault-tolerant, and reliable. Distributed messaging is based on the same concept of decoupling the message-producer from the message-consumer.

A distributed messaging system consists of the following:

• Producers that generate messages and publish data to the underlying broker.
• Broker is an indirection layer that mediates the communication between the producer and consumer. The Broker can perform tasks such as message validation, message transformation, and routing. The Broker also is responsible for nonfunctional requirements such as message reliability and persistence. The Broker consists of one or more topics. A topic is a category or feed name to which messages are published.
• Consumers are entities that are responsible for message-consumption. Consumers read from a topic within a broker.
• In a distributed architecture, all the entities (Producer, Consumer, and Broker) can be launched in multiple instances that can be monitored and controlled using a coordinator, such as Zookeeper:

It is advisable to have an implementation that supports some sort of message-ordering (partitioned or global) in a distributed broker scenario.

Event-sourcing, and thus the ability to play back state from the past, is an important pattern that a reliable distributed-messaging implementation should support.