⚡ Streaming & Kafka · Chapter 5

Consumer groups, rebalancing and offsets

Three mechanisms decide whether your consumer loses data, duplicates it, or quietly stops: where the offset is committed, when the group rebalances, and how long a slow poll is tolerated.

Study this lesson free in the app →

The basics

A consumer group is a set of consumer processes sharing one group.id that split a topic's partitions between them, so each partition is read by exactly one member and the group's progress survives restarts.

The problem it solves: one consumer can't keep up with a busy topic, and if it crashes someone has to carry on from where it stopped — without skipping events or redoing hours of work. The group gives you both: parallelism (one member per partition) and continuity (committed offsets stored in Kafka).

The mental model: a warehouse night shift with three loading bays. Each bay has a clipboard recording the last crate handled — the committed offset. When a worker joins or leaves, everyone stops while the supervisor reassigns bays — a rebalance. Write on the clipboard before unloading a crate and a collapse means a crate is skipped; write after and the next worker may unload one twice. Kafka gives you the same choice.

Explained

A consumer group is a promise: every partition is processed by exactly one member, and progress survives restarts.

Keeping that promise costs three things you have to configure — commit timing, rebalance behaviour, and poll timeouts. Get them wrong and the failures are subtle: not a crash, but silent gaps or silent doubles.

The night shift handover

A warehouse runs three loading bays with a rotating crew.

The clipboard at each bay records the last crate handled. A new worker picks up the clipboard and continues — that's the committed offset.

The handover happens whenever someone joins or leaves. Everyone stops, the supervisor reassigns bays, everyone restarts. Brief, disruptive, necessary — that's a rebalance.

The rule about the clipboard is where it gets interesting. Write it down before you unload the crate, and if you collapse mid-lift the next worker skips it entirely — data loss. Write it down after, and if you collapse the next worker redoes that crate — a duplicate.

There is no third option. You are choosing which failure you'd rather have, and the industry chose duplicates — because duplicates can be absorbed by idempotency, and gaps cannot be recovered at all.

The lifecycle of a consumer

poll() does double duty: it fetches records and tells the coordinator you're alive. That's why slow processing between polls looks identical to a dead consumer.

The three timeouts that actually matter

• session.timeout.ms (default 45s) — how long the coordinator waits for a heartbeat before declaring you dead. Heartbeats run on a background thread, so this survives slow processing
• max.poll.interval.ms (default 5 min) — the maximum time allowed between calls to poll(). Exceed it and you are ejected from the group, even though heartbeats were fine
• max.poll.records (default 500) — how many records a single poll() returns

The classic production failure lives in the interaction of the last two. You fetch 500 records, each takes 2 seconds to process, so the next poll() is 16 minutes away. At 5 minutes the coordinator ejects you, rebalances your partitions to someone else — and when your original consumer finally finishes and tries to commit, the commit is rejected because it no longer owns those partitions.

The work was done. The offset was never recorded. Another consumer reprocesses all of it. From the outside this looks like a mysterious duplicate storm, and it repeats forever because each cycle takes just as long.

Fix: lower max.poll.records, raise max.poll.interval.ms, or move slow work off the poll loop.

Step through a rebalance

Calling close() on shutdown turns that 45-second gap into a sub-second one. It's one line, and it's the difference between graceful and grim during a deploy.

The duplicate storm

A payments consumer processes 500 records per poll. Each record calls an external API taking ~2 seconds.

Symptoms:

• consumer lag climbs steadily
• the log shows a rebalance roughly every 5 minutes
• the same payments are processed over and over
• the team has already tried adding more consumer instances — it got worse

A consumer written for production

from confluent_kafka import Consumer, KafkaError

c = Consumer({
    "bootstrap.servers": BROKERS,
    "group.id": "billing",
    "enable.auto.commit": False,        # we decide when progress is recorded
    "auto.offset.reset": "earliest",
    "max.poll.records": 50,            # sized to the work per record
    "max.poll.interval.ms": 600_000,   # generous, but bounded
})

def on_assign(consumer, partitions):
    log.info("assigned %s", [(p.topic, p.partition) for p in partitions])

def on_revoke(consumer, partitions):
    log.info("revoked %s — committing what we finished", len(partitions))
    try:
        consumer.commit(asynchronous=False)   # last chance before losing them
    except Exception as err:
        log.warning("commit on revoke failed: %s", err)

c.subscribe(["ride_events"], on_assign=on_assign, on_revoke=on_revoke)

try:
    while running:
        msg = c.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            if msg.error().code() != KafkaError._PARTITION_EOF:
                log.error("consume error: %s", msg.error())
            continue

        try:
            handle(msg)                       # MUST be idempotent
        except RetryableError:
            continue                          # don't commit; it'll be redelivered
        except Exception as err:
            dlq.send(msg, reason=repr(err))   # poison pill — park it, move on

        c.commit(msg, asynchronous=False)     # after the work, always
finally:
    c.close()                                 # leave the group immediately

Five production details in there: manual commit after work, a revoke hook that commits before losing partitions, a DLQ so one poison record can't stall a partition forever, retryable errors left uncommitted, and close() in a finally.

Rebalances are the hidden cost of scale

Every join, leave, crash, deploy or max.poll.interval breach pauses the entire group, not just the affected member. With the default eager protocol, everyone gives up all partitions and gets a new assignment — a stop-the-world pause of seconds.

At scale that adds up:

• Rolling deploy of 20 pods = 20 rebalances unless you use static membership
• group.instance.id (static membership) — a restarting pod reclaims its own partitions without triggering a full rebalance. The single highest-value setting for deploy-heavy environments
• Cooperative sticky assignor — incremental rebalancing; only the partitions that actually move are revoked, so unaffected members keep working
• More partitions = longer rebalances. Another reason not to over-provision partitions "just in case"

What to watch in production, in priority order:

1. Lag per partition — not aggregate. One lagging partition means a hot key; the average hides it
2. Rebalance rate — a group rebalancing constantly is a group not working
3. Commit failure rate — rejected commits mean ejection, and duplicates
4. Time in rebalancing state — if it's a meaningful share of uptime, look at timeouts and deploy strategy

Words to know

In two minutes

• A consumer group splits partitions so each is processed by exactly one member; committed offsets let any member resume.
• Commit after processing: a crash then causes duplicates (absorb them with idempotent writes) rather than gaps.
• poll() also proves liveness: if processing a batch takes longer than max.poll.interval.ms, the consumer is ejected and the batch is redone.
• Size the batch to the interval: fewer max.poll.records, a bounded higher interval, or parallel processing.
• Every join, leave, crash or deploy rebalances the whole group; static membership and cooperative-sticky assignment shrink the cost.
• Call close() on shutdown so failover is immediate instead of waiting out the session timeout.
• Send poison records to a dead-letter topic after N attempts; watch lag per partition, not the average.

Test yourself

There are 8 questions, 3 puzzles and 7 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.