⚡ Streaming & Kafka · Chapter 2

Kafka hands-on: your first producer and consumer

Twenty minutes from nothing to watching your own events flow. Concepts stick once you have broken something yourself.

Study this lesson free in the app →

The basics

Kafka is a system that stores streams of events — "ride requested", "payment captured" — in order, on disk, so that many programs can read them independently, at their own pace. This topic is about running it on your own laptop and watching it behave.

The problem it solves: reading definitions of topics, partitions and consumer groups doesn't make them stick. Seeing a second consumer join and steal half the partitions, or a new consumer receive "nothing" because it started at the end, does. Twenty minutes with one Docker container replaces a week of diagrams.

The mental model: a shared notebook that only gets new lines appended. Producers write lines. Consumers read lines and keep a bookmark (an offset) of how far they've got. Nobody erases a line when they read it, so a second reader can start from page one.

Explained

You can read about topics and partitions for a week and still not feel it.

Run one broker, send three events, start two consumers, and watch one of them go quiet when the other joins. That is when consumer groups make sense.

Everything below runs on a laptop, costs nothing, and takes about twenty minutes.

1. One broker, one command

# docker-compose.yml — modern Kafka, no ZooKeeper needed
services:
  kafka:
    image: apache/kafka:3.8.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d
docker compose logs -f kafka     # wait for "Kafka Server started"

One broker is not production — no replication, no failover. It is exactly right for learning, because everything that happens is your doing.

2. Create a topic and look at it

# a shell inside the container
docker compose exec kafka bash

# create a topic with 3 partitions
/opt/kafka/bin/kafka-topics.sh --create \
  --topic ride_events --partitions 3 --replication-factor 1 \
  --bootstrap-server localhost:9092

# what did we get?
/opt/kafka/bin/kafka-topics.sh --describe \
  --topic ride_events --bootstrap-server localhost:9092

Topic: ride_events  PartitionCount: 3  ReplicationFactor: 1
  Partition: 0  Leader: 1  Replicas: 1  Isr: 1
  Partition: 1  Leader: 1  Replicas: 1  Isr: 1
  Partition: 2  Leader: 1  Replicas: 1  Isr: 1

Three partitions on one broker. Partitions are about parallelism; replicas are about survival. With one broker you get the first and none of the second.

3. Produce and consume from the shell

# terminal A — a producer that reads what you type
/opt/kafka/bin/kafka-console-producer.sh \
  --topic ride_events --bootstrap-server localhost:9092 \
  --property parse.key=true --property key.separator=:

# type these, one per line:
ride_1:{"event":"requested","fare":0}
ride_1:{"event":"completed","fare":250}
ride_2:{"event":"requested","fare":0}

# terminal B — read everything from the beginning
/opt/kafka/bin/kafka-console-consumer.sh \
  --topic ride_events --from-beginning \
  --bootstrap-server localhost:9092 \
  --property print.key=true --property print.partition=true

Look at the partition numbers. Both ride_1 events land on the same partition; ride_2 may land elsewhere. You just watched hash(key) % 3 decide routing — with your own eyes rather than from a diagram.

4. The same thing in Python

pip install confluent-kafka

# producer.py
import json
from confluent_kafka import Producer

p = Producer({"bootstrap.servers": "localhost:9092"})

def delivered(err, msg):
    if err:
        print(f"FAILED: {err}")            # never assume it arrived
    else:
        print(f"→ partition {msg.partition()} offset {msg.offset()}")

for i in range(10):
    event = {"event_id": f"evt_{i}", "ride_id": f"ride_{i % 3}", "fare": 100 + i}
    p.produce(
        "ride_events",
        key=event["ride_id"],               # the key decides the partition
        value=json.dumps(event),
        callback=delivered,
    )

p.flush()      # produce() is async — without this, the script exits first

# consumer.py
import json
from confluent_kafka import Consumer

c = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "billing",                 # the consumer GROUP
    "auto.offset.reset": "earliest",       # new group starts from the beginning
    "enable.auto.commit": False,           # we commit after the work succeeds
})
c.subscribe(["ride_events"])

try:
    while True:
        msg = c.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            print("error:", msg.error())
            continue

        event = json.loads(msg.value())
        print(f"p{msg.partition()} @{msg.offset()}  {event['event_id']}")

        c.commit(msg)                      # AFTER processing — never before
finally:
    c.close()                              # leaves the group cleanly

5. The experiment that teaches the most

Do this in order and watch what happens. It takes four minutes and replaces an hour of reading.

a. Run one consumer. It receives all three partitions. Note the p0 p1 p2 in the output.

b. Start a second consumer with the same group.id. Within a few seconds both log a rebalance. Now one has two partitions, the other has one. Produce more events — the load splits.

c. Start a fourth consumer. It sits idle, receiving nothing. There are only three partitions, and each belongs to exactly one member. Partition count is the ceiling on parallelism — now you've seen it rather than been told it.

d. Kill one consumer (Ctrl-C). Another rebalance; its partitions are picked up by the survivors within seconds. Nothing is lost.

e. Start a consumer with a different group.id. It receives everything from the beginning, while the first group carries on undisturbed. Two independent readers of the same log.

f. Stop everything, produce 5 more events, then restart the original group. It resumes from its committed offset — not from the beginning, not from now. That is the bookmark doing its job.

What just happened in step (e)

This is what makes a log different from a queue. A queue hands a message to one consumer and forgets it. A log keeps it, and hands out bookmarks.

6. Inspecting a running system

# who is in the group, and how far behind are they?
/opt/kafka/bin/kafka-consumer-groups.sh --describe --group billing \
  --bootstrap-server localhost:9092

TOPIC        PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID
ride_events  0          14              14              0    consumer-billing-1
ride_events  1          9               21              12   consumer-billing-2
ride_events  2          7               7               0    consumer-billing-1

LAG is the number to watch in production. It is how many messages exist that this group hasn't processed. Steady lag near zero is healthy. Lag climbing means the consumer is slower than the producer — the earliest warning you get that something downstream is failing.

# replay: move a group back to the start (stop the consumers first)
/opt/kafka/bin/kafka-consumer-groups.sh --reset-offsets --to-earliest \
  --group billing --topic ride_events --execute \
  --bootstrap-server localhost:9092

That last command is the superpower. Fix a bug, reset the offset, and reprocess history — provided your consumer is idempotent.

Words to know

In two minutes

• One apache/kafka container in KRaft mode (no ZooKeeper) is a complete Kafka for learning.
• Create a topic with 3 partitions, produce keyed events, and watch the same key always land on the same partition.
• produce() is asynchronous: use a delivery callback and flush() before exiting, or events silently never leave.
• A new consumer group starts at the latest offset by default — set auto.offset.reset=earliest to read history.
• Consumers in one group split partitions; extra consumers beyond the partition count sit idle.
• Commit offsets after processing, not on a timer, or a crash can skip messages.
• kafka-consumer-groups --describe shows per-partition lag — the first thing to check in any 'Kafka lost my data' report.

Test yourself

There are 9 questions, 3 puzzles and 7 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.