Sara L.
Sample Up to date with Kafka 4
Most material I found still shows ZooKeeper. This one starts with KRaft and the official Docker image, which saved me time.
Apache Kafka 4 for data engineers: streams you can trust
Topics, partitions, producers and consumers in KRaft mode, delivery guarantees, schemas and Kafka Connect, with Python examples
Created by Priya Raghavan
- Updated September 2026
- English
- Intermediate
What you will learn
- Explain topics, partitions, offsets and consumer groups to a colleague
- Run a Kafka 4 cluster in KRaft mode locally with Docker
- Configure producers for durability with acks, idempotence and retries
- Write consumers that commit offsets correctly and survive restarts
- Manage schemas and compatibility with a schema registry
- Diagnose consumer lag and choose a sensible partition count
Course content
Kafka in plain terms
Reliable producers
- Video: acks, retries and idempotence in depth 12 min
- Video: Batching, compression and throughput 10 min
- Article: Choosing good keys 9 min
- Video: Transactions and the outbox pattern 11 min
Consumers that do not lose or repeat work
- Video: Offset commit strategies compared 11 min
- Article: Idempotent processing downstream 10 min
- Video: Rebalances and the new consumer protocol 10 min
- Video: Poison messages and dead letter topics 9 min
Schemas and data contracts
- Article: Why JSON without a schema hurts later 8 min
- Video: Avro and Protobuf with a schema registry 12 min
- Video: Compatibility rules and safe evolution 11 min
Kafka Connect and change data capture
- Article: Kafka Connect concepts 9 min
- Video: Change data capture from PostgreSQL with Debezium 13 min
- Video: Sinking events into object storage and a warehouse 11 min
Operating Kafka
- Article: Monitoring lag and broker health 10 min
- Video: Sizing partitions and retention 10 min
- Video: Security basics: TLS, SASL and ACLs 11 min
- Article: Managed Kafka or self hosted 9 min
- Quiz: Final check 6 min
Requirements
- Comfortable with Python and the command line
- Docker installed
- Basic understanding of what a database and a message are
Description
Kafka sits underneath a huge share of modern data platforms. It is also easy to use badly: lost messages, duplicate processing, consumers that fall hours behind. This course teaches how Kafka really works so you can make it reliable.
We use Apache Kafka 4, which runs only in KRaft mode. ZooKeeper is gone, so the setup is simpler than older tutorials suggest, and some old advice no longer applies.
What you will learn
- How topics, partitions, offsets and consumer groups fit together
- Producer settings that decide whether data can be lost or duplicated
- Consumer patterns for at least once and effectively once processing
- Schemas with Avro or Protobuf and a schema registry
- Kafka Connect for moving data into and out of Kafka without custom code
- Monitoring lag, sizing partitions and operating a small cluster
How the course works
Section one is written in full and gets Kafka running on your laptop with the official Apache Docker image. Later sections are videos built around one scenario: order events from a web application flowing to a warehouse and to a real time dashboard. Code examples use Python with the confluent-kafka client, and every concept maps directly to Java, Go or any other client library.
Who this course is for
- Data engineers adding streaming to batch pipelines
- Backend developers who publish or consume events
- Platform engineers preparing to operate Kafka
Instructor
Priya Raghavan
Streaming and data platform engineer. Kafka, Python pipelines and DuckDB explained plainly
- 4.4 instructor rating
- 21 reviews
- 7,170 students
- 3 courses
- Sample
I work on data platforms where events arrive all day and someone always wants yesterday's numbers fixed. Most of my time goes into Kafka topics, Python ingestion services and the small analytical tools that make debugging them faster.
I started teaching because new data engineers kept asking me the same questions: why is this consumer lagging, why did this file load twice, why is this query slow. My courses answer those questions with small, complete examples rather than slides.
Outside work I maintain a few open source notebooks for teaching SQL over Parquet files, and I like explaining file formats more than I probably should.
Student reviews
Vikram P.
Sample Finally clear on offsets and lag
The partition and offset lesson plus reading kafka-consumer-groups output made lag finally make sense to me.
Peter G.
Sample Idempotent consumers explained well
The idea of effectively once through idempotent writes rather than chasing exactly once everywhere was the key insight for our team.
Chiara B.
Sample Practical Debezium walkthrough
We needed CDC from PostgreSQL and the Debezium lesson got us running in an afternoon, including how deletes and tombstones work.
Jakub N.
Sample Strong producer section
The broker failure experiments showed exactly when acks=1 loses data. Schema section could use one more worked example.
Femi O.
Sample Good pace for engineers
Assumes you know Python, which suited me. The dead letter topic pattern is now in our pipeline.
Lin H.
Sample Useful but dense in places
Accurate and thorough, but the rebalance lesson moved quickly. I had to rewatch it twice.
Keep exploring
All Data Engineering courses
More in Data Engineering
$74.99
Enroll now