Kafka’s architecture in whole is quite simplistic in nature and has some very important components, which are the crux of the whole workings of it.

The following figure shows the workings of Kafka with all components labelled. The main components which constitute Kafka architecture are as follows:

• Message: Flows from producer all the way to consumer through topics existing in a broker.
• Producer: Producer, as the name suggests produces these stream messages and pumps into the topic.
• Topic: The category into which the producer pumps the message. The producer produces messages of a particular category which fall into this topic. Ideally, for each category of messages a new topic is created in Kafka.
• Broker: Kafka cluster comprising of multiple servers/nodes. Each node is a Kafka broker and contains multiple partitions holding multiple topics.
• Partition: Each of the Kafka topic, is partitioned and each partition contains ordered messages from the producer. Writes to each partition are sequential in nature and this aspect is one of the main reasons for Kafka’s high performance. When writing onto the topic partition, each message is given a sequence number referred to as offset and this uniquely identifies a message in a partition. Kafka, being distributed in nature, distributes these topic partitions across multiple brokers for fault tolerance and durability of messages.
• Consumer: Again as the name suggests, subscribes to a particular message category (a topic) and consumes messages in that topic. This consumer reads messages from the topic and is entrusted with maintaining the audit of which message has been read and actioned. Kafka does not remove these messages nor take an audit of messages read by various consumers connecting to the topic and reading messages.
• Consumer Group: A group of consumers logically grouped in Kafka is known as the consumer group.

So, the producers produce messages and dumps into various categorized topics, from there consumers subscribed to these topic reads the messages and do the necessary. The communication of various components is over the efficient and reliable TCP protocol. This is the overall architecture of Kafka.

We have dedicated the sections here to detailing each of these components:

Detailed Kafka documentation can be found on Kafka’s official site, available at https://kafka.apache.org/documentation/.