The Event Log Is the Core Abstraction

Logs have been used for a long time as a mechanism for services to output information about what they are doing, including problems they encounter. Log entries usually include a timestamp, a notion of “urgency” (e.g., error, warning, or informational), information about the process and/or machine, and an ad hoc text message with more details. Well-structured log messages at appropriate execution points are proxies for significant events.

The metaphor of a log generalizes to a wide class of data streams, such as these examples:

Database CRUD transactions

Each insert, update, and delete that changes state is an event. Many databases use a WAL (write-ahead log) internally to append such events durably and quickly to a file before acknowledging the change to clients, after which in-memory data structures and other files with the actual records can be updated with less urgency. That way, if the database crashes after the WAL write completes, the WAL can be used to reconstruct and complete any in-flight transactions once the database is running again.

Telemetry from IoT devices

Almost all widely deployed devices, including cars, phones, network routers, computers, airplane engines, home automation devices, medical devices, kitchen appliances, etc., are now capable of sending telemetry back to the manufacturer for analysis. Some of these devices also use remote services to implement their functionality, like Apple’s Siri for voice recognition. Manufacturers use the telemetry to better understand how their products are used; to ensure compliance with licenses, laws, and regulations (e.g., obeying road speed limits); and to detect anomalous behavior that may indicate incipient failures, so that proactive action can prevent service disruption.

Clickstreams

How do users interact with a website? Are there sections that are confusing or slow? Is the process of purchasing goods and services as streamlined as possible? Which website version leads to more purchases, “A” or “B”? Logging user activity allows for clickstream analysis.

State transitions in a process

Automated processes, such as manufacturing, chemical processing, etc., are examples of systems that routinely transition from one state to another. Logs are a popular way to capture and propagate these state transitions so that downstream consumers can process them as they see fit.

Logs also enable two general architecture patterns: ES (event sourcing), and CQRS (command-query responsibility segregation).

The database WAL is an example of event sourcing. It is a record of all changes (events) that have occurred. The WAL can be replayed (“sourced”) to reconstruct the state of the database at any point in time, even though the only state visible to queries in most databases is the latest snapshot in time. Hence, an event source provides the ability to replay history and can be used to reconstruct a lost database or replicate one to additional copies.

This approach to replication supports CQRS. Having a separate data store for writes (“commands”) vs. reads (“queries”) enables each one to be tuned and scaled independently, according to its unique characteristics. For example, I might have few high-volume writers, but a large number of occasional readers. Also, if the write database goes down, reading can continue, at least for a while. Similarly, if reading becomes unavailable, writes can continue. The trade-off is accepting eventually consistency, as the read data stores will lag the write data stores.¹

Hence, an architecture with event logs at the core is a flexible architecture for a wide spectrum of applications.

Message Queues Are the Core Integration Tool

Message queues are first-in, first-out (FIFO) data structures, which is also the natural way to process logs. Message queues organize data into user-defined topics, where each topic has its own queue. This promotes scalability through parallelism, and it also allows producers (sometimes called writers) and consumers (readers) to focus on the topics of interest. Most implementations allow more than one producer to insert messages and more than one consumer to extract them.

Reading semantics vary with the message queue implementation. In most implementations, when a message is read, it is also deleted from the queue. The queue waits for acknowledgment from the consumer before deleting the message, but this means that policies and enforcement mechanisms are required to handle concurrency cases such as a second consumer polling the queue before the acknowledgment is received. Should the same message be given to the second consumer, effectively implementing at least once behavior (see “At Most Once. At Least Once. Exactly Once.”)? Or should the next message in the queue be returned instead, while waiting for the acknowledgment for the first message? What happens if the acknowledgment for the first message is never received? Presumably a timeout occurs and the first message is made available for a subsequent consumer. But what happens if the messages need to be processed in the same order in which they appear in the queue? In this case the consumers will need to coordinate to ensure proper ordering. Ugh…

On the other hand, having multiple readers is a way to improve performance through parallelism, but any one reader instance won’t see every message in the topic, so the reader must be stateless; it can’t know global state about the stream.

Kafka is unique in that messages are not deleted when read, so any number of readers can ingest all the messages in a topic. Instead, Kafka uses either a user-specified retention time (the time to live, or TTL, which defaults to seven days), a maximum number of bytes in the queue (the default is unbounded), or both to know when to delete the oldest messages.

Kafka can’t just return the head element for a topic’s queue, since it isn’t immediately deleted the first time it’s read. Instead, Kafka remembers the offset into the topic for each consumer and returns the next message on the next read.

Hence, a Kafka consumer could maintain stream state, since it will see all the messages in the topic. However, since any process might crash, it’s necessary to persist any important state changes. One way to do this is to write the current state to another Kafka topic!

Message queues provide many advantages. They decouple producers from consumers, making it easy for them to come and go. They support an arbitrary number of producers and consumers per topic, promoting scalability. They expose a narrow message queue abstraction, which not only makes them easy to use but also effectively hides the implementation so that many scalability and resiliency features can be implemented behind the scenes.

Why Kafka?

Kafka’s current popularity is because it is ideally suited as the backbone of fast data architectures. It combines the benefits of event logs as the fundamental abstraction for streaming with the benefits of message queues. The Kafka documentation describes it as “a distributed, partitioned, replicated commit log service.” Note the emphasis on logging, which is why Kafka doesn’t delete messages once they’ve been read. Instead, multiple consumers can see the whole log and process it as they see fit (or even reprocess the log when an analysis task fails). The quote also hints that Kafka topics are partitioned for greater scalability and the partitions can be replicated across the cluster for greater durability and resiliency.

Kafka has also benefited from years of production use and development at LinkedIn, where it started. A year ago, LinkedIn’s Kafka infrastructure surpassed 1.1 trillion messages a day, and it’s still growing.

With the Kafka backbone and persistence options like distributed filesystems and databases, the third key element is the processing engine. For the last several years, Kafka, Cassandra, and Spark Streaming have been a very popular combination for streaming implementations.⁴ However, our thinking about stream processing semantics is evolving, too, which has fueled the emergence of Spark competitors.