Why teams use an event log

Most systems start with direct calls. The checkout service calls the email service, the warehouse loader queries the production database at night, and the analytics team copies tables they need. This works until there are many producers and many consumers, and every new consumer needs a new integration.

Kafka changes the shape of the problem. Producers write events, small records describing something that happened, to a durable, ordered log. Consumers read that log at their own pace. The producer does not know or care who reads.

What makes Kafka different from a queue

In a classic message queue, a message is usually removed once a consumer acknowledges it. In Kafka, events stay in the log for a configured retention period, for example seven days, or forever with compaction. That has big consequences:

• Many independent readers. The fraud service, the warehouse loader and a search indexer can all read the same events.
• Replay. A new consumer can start from the beginning, and a buggy consumer can be fixed and rewound.
• Decoupling in time. If the warehouse loader is down for an hour, it catches up afterwards. Nothing is lost.

When Kafka is worth it

Kafka is a good fit when you have high event volumes, several consumers of the same data, a need for replay, or you want change data capture from databases. It is often overkill for a single team with one producer and one consumer and modest volume, where a managed queue or even a database table may be simpler.

What changed in Kafka 4

Kafka 4.0 removed ZooKeeper entirely. Cluster metadata is now managed by Kafka itself through the KRaft consensus protocol, run by controller nodes. For you this means fewer moving parts, faster startup and simpler local setups. Kafka 4 also made the new consumer group rebalance protocol generally available, which reduces the stop the world pauses older consumers suffered during rebalances.

In the next lesson we look at the data model in detail: topics, partitions and offsets.