Topics, partitions and offsets

A topic is a named stream of events, such as orders or page_views. Topics are split into partitions, and partitions are what actually get stored, replicated and read.

Partitions are ordered logs

Each partition is an append only log. Every event written to it gets an increasing number called an offset: 0, 1, 2 and so on. Order is guaranteed within a partition, never across partitions.

orders partition 0: [0] [1] [2] [3] [4] ...
orders partition 1: [0] [1] [2] ...
orders partition 2: [0] [1] [2] [3] ...

Keys decide the partition

When a producer sends an event with a key, Kafka hashes the key to choose a partition. All events with the same key land in the same partition, in order. Use the entity you need ordered as the key: an order id, a customer id, an account number.

If you send events without a key, the producer spreads them across partitions for throughput, and you lose ordering between related events.

Consumer groups share the work

Consumers join a consumer group with a group id. Kafka assigns each partition to exactly one consumer in the group. With six partitions and three consumers, each consumer reads two partitions. Add a fourth consumer and the partitions are reassigned.

Two consequences follow:

1. A group can never have more active consumers than partitions. Extra consumers sit idle.
2. The partition count is your ceiling for parallelism, so choose it with growth in mind. You can add partitions later, but doing so changes which partition a key maps to.

Committed offsets

Each group stores, per partition, the offset it has processed up to. This is the committed offset. When a consumer restarts, it continues from there. The difference between the latest offset in a partition and the group's committed offset is called lag, and it is the most important number you will monitor.

Replication

Each partition has a leader and followers on other brokers. With a replication factor of 3, the data survives the loss of two brokers, provided enough replicas were in sync.