Understanding Your Messaging Requirements

Understanding best practices for designing your infrastructure and applications depends on the constraints imposed by your functional and nonfunctional requirements. To simplify the problem, let’s start by asking some questions:

Where does data come from, and where is it going?

All data platforms must have data flowing into the system from a source (ingest) and flowing out to a sink (egress). How do you ingest data and make it available to the rest of your system?

How fast do you need to react to incoming data?

You want results as soon as possible, but if you clarify the latency requirements you actually need, then you can adjust your design and technology choices accordingly.

What are your message delivery semantics?

Can your system tolerate dropped or duplicate messages? Do you need each and every message exactly once? Be careful, as this can potentially have a big impact in the throughput of your system.

How is your data keyed?

How you key messages has a large impact on your technology choices. You use keys in distributed systems to figure out how to distribute (partition), the data. We’ll discuss how partitioning can affect our requirements and performance.

Let’s explore some of the architectural decisions based on your answers.

Data Ingestion

Data ingestion represents the source of all the messages coming into your system. Some examples of ingestion sources include the following:

• A user-facing RESTful API that sits at the periphery of our system, responding to HTTP requests originating from our end users

• The Change Data Capture (CDC) log of a database that records mutation operations (Create/Update/Delete)

• A filesystem directory from which files are read

The source of messages entering your system is not usually within your control. Therefore, we should persist messages as soon as possible. A robust and simple model is to persist all messages onto an append-only event log (aka event store or event journal).

The event log model provides maximum versatility to the rest of the platform. Immutable messages are appended to the log. This allows us to scale the writing of messages for a few reasons. We no longer need to use blocking operations to make a write, and we can easily partition our writes across many physical machines to increase write throughput. The event log becomes the source of truth for all other data models (a golden database).

To create derivative data models, we replay the event log and create an appropriate data model for our use case. If we want to perform fast analytical queries across multiple entities, we may choose to build an OLAP cube. If we want to know the latest value of an entity, we could update an in-memory database. Usually, the event log is processed immediately and continuously, but that does not prevent us from also replaying the log less frequently or on demand with very little impact to our read and write throughput to the event log itself.

What we’ve just described is the Event Sourcing design pattern, illustrated in Figure 3-1, which is part of a larger aggregate of design patterns called Command and Query Responsibility Segregation (CQRS) and commonly used in event-driven architectures.

Event logs are also central to the Kappa architecture. Kappa is an evolution of the Lambda architecture, but instead of managing a batch and fast layer, we implement only a fast layer that uses an event log to persist messages.

Figure 3-1. Event Sourcing: event messages append to an event log and are replayed to create different models

Apache Kafka is a Publish/Subscribe (or pub/sub) system based on the concept of a distributed log. Event messages are captured in the log in a way that ensures consumers can access them as soon as possible while also making them durable by persisting them to disk. The distributed log implementation enables Kafka to provide the guarantees of durability and resilience (by persisting to disk), fault tolerance (by replication), and the replay of messages.

Fast Data, Low Latency

How fast should Fast Data be? We will classify as Fast Data the platforms that can react to event messages in the millisecond-to-minutes range.

Apache Kafka is well suited for this range of latency. Kafka made a fundamental design choice to take advantage of low-level capabilities of the OS and hardware to be as low latency as possible. Messages produced onto a topic are immediately stored in the platform’s Page Cache, which is a special area of physical system memory used to optimize disk access. Once a message is in Page Cache, it is queued to be written to disk and made available to consumers at the same time. This allows messages passing through a Kafka broker to be made available nearly instantaneously to downstream consumers because they’re not copied again to another place in memory or buffer. This is known as zero-copy transfer within the Kafka broker.

Zero-copy does not preclude the possibility of streaming messages from an earlier offset that’s no longer in memory, but obviously this operation will incur a slight delay to initially seek the data on disk and make it available to a consumer by bringing it back into Page Cache. In general, the most significant source of latency when subscribing to a Kafka topic is usually the network connection between the client and broker.

Kafka is fast, but other factors can contribute to latency. An important aspect is the choice of delivery guarantees we require for our application. We discuss this in more detail in the next section.

Message Delivery Semantics

As we saw in Chapter 1, there are three types of message delivery semantics: at-most-once, at-least-once, and exactly-once. In the context of the message backbone, these semantics describe how messages are delivered to a destination when accounting for common failure use cases such as network partitions/failures, producer (source) failure, and consumer (sink, application processor) failure.

Some argue that exactly-once semantics are impossible. The crux of the argument is that such delivery semantics are impossible to guarantee at the protocol level, but we can fake it at higher levels. Kafka performs additional operations at the application processing layer that can fake exactly-once delivery guarantees. So instead of calling it exactly-once message delivery, let’s expand the definition to exactly-once processing at the Application layer.

A plausible alternative to exactly-once processing in its most basic form is at-least-once message delivery with effective idempotency guarantees on the sink. The following operations are required to make this work:

• Retry until acknowledgement from sink.

• Idempotent data sources on the receiving side. Persist received messages to an idempotent data store that will ensure no duplicates, or implement de-duplication logic at the application layer.

• Enforce that source messages are not processed more than once.

Distributing Messages

A topic represents a type of data. Partitioning messages is the key to supporting high-volume data streams. A partition is a subset of messages in a topic. Partitions are distributed across available Kafka brokers, as illustrated in Figure 3-2. How you decide which partition a message is stored in depends on your requirements.

Figure 3-2. Kafka topic partitioning strategy

If your intention is to simply capture discrete messages and order does not matter, then it may be acceptable to evenly distribute messages across partitions (round-robin), similar to the way an HTTP load balancer may work. This provides the best performance as messages are evenly distributed. A caveat to this approach is that we sacrifice message order: one message may arrive before another, but because they’re in two different partitions being read at different rates, an older message might be read first.

Usually, we decide on a partition strategy to control the way messages are distributed across partitions, which allows us to maintain order with respect to a particular key found in the message. Partitioning by key allows for horizontal scalability while maintaining guarantees about order.

The hard part is choosing what key to use. It may not be enough to simply pick a unique identifier, because if we receive an uneven distribution of messages based on a certain key, then some partitions are busier than others, potentially creating a bottleneck. This problem is known as a hot partition and generally involves tweaking your partitioning strategy after you start learning the trends of your messages.