⚡ Streaming & Kafka · Chapter 3

Events and the Log — the Kafka mental model

Kafka is not a queue. It is an append-only log that many readers can read at their own speed.

Study this lesson free in the app →

The basics

An event is a record of something that happened — "rider R42 requested a cab at 18:02 in Koramangala" — and a log is an append-only list of events kept in the order they arrived. Kafka is a distributed log.

The problem it solves: a cab app has one stream of ride events and many teams that need it — billing, fraud, surge pricing, the data lake. Wire each team directly to the app and you get N×M integrations, and one slow team backs up the app. A traditional queue doesn't fix it either: once one reader takes a message, it's gone for everyone else.

The mental model: a newspaper archive. New editions are added at the end; old ones are never edited. Any number of readers can visit, each with their own bookmark. A reader who was on leave for a week simply resumes from their bookmark — and a new reader can start from the first edition still on the shelf.

Explained

A queue loses a message once it is consumed.

A log keeps it, and every reader remembers their own position.

That one difference is why streaming platforms can replay yesterday, feed six teams from one topic, and survive a broken consumer.

The book series and the reading clubs

A topic is a book series — say, ride_events.

Each partition is one volume of the series. Volumes can be read in parallel, but within a volume the pages are strictly in order.

A producer writes new pages at the end. Nobody edits earlier pages.

A consumer group is a reading club. The club splits volumes among its members so it finishes faster — each volume read by exactly one member. A different club can read the whole series independently, at its own pace, without disturbing the first.

Every reader keeps a bookmark — the offset. Lose your bookmark and you either re-read (duplicates) or skip ahead (data loss).

The moving parts

• Event — an immutable fact that happened, with an id and a time
• Topic — a named stream of events
• Partition — an ordered, append-only sequence; the unit of parallelism and the unit of ordering
• Offset — the position of an event inside a partition
• Producer — writes; picks the partition via a key
• Consumer group — a set of consumers sharing the partitions of a topic; each partition goes to exactly one member
• Retention — how long events stay (time or size based), independent of whether they were read
• Broker — a server holding partitions; partitions are replicated across brokers for durability

Ordering guarantee: within a partition, never across a topic. Remember that sentence exactly.

One topic, two clubs

topic: ride_events
  P0 ──▶ e1 e2 e3 e4 ...
  P1 ──▶ e5 e6 e7 ...
  P2 ──▶ e8 e9 ...

group: billing        group: analytics
  c1 → P0, P1           c1 → P0
  c2 → P2               c2 → P1, P2
  (offset 402)          (offset 87 — lagging, and that's fine)

Add a third consumer to billing and Kafka rebalances partitions across the three. Add a fourth with only three partitions and one consumer sits idle — partitions cap your parallelism.

Why high-volume platforms put Kafka in the middle

Without a log, every producer must know every consumer: N×M integrations, and a slow consumer backs up the app.

With a log, producers write once and walk away. Consumers are decoupled in time — this buys you:

• Buffering — a traffic spike or a warehouse outage does not reach the app
• Replay — deploy a fixed consumer and re-read from an old offset
• Fan-out — fraud, pricing, ops dashboards and the lake all read the same events
• Back-pressure isolation — a slow consumer only grows its own lag

The number you watch in production is consumer lag: how far behind the newest offset a group is. Rising lag is the earliest warning that something downstream is dying.

Topic = series. Partition = volume. Offset = bookmark. Consumer group = reading club.

Order is guaranteed inside a volume — never across the series.

Gotchas that fail interviews

• Saying "Kafka guarantees ordering" — only per partition
• Adding consumers beyond the partition count and expecting more throughput
• Assuming a consumed message is deleted — retention decides that, not consumption
• Committing offsets before the work succeeds — a crash then silently loses events
• Ignoring rebalance storms — every join/leave pauses consumption briefly

The classic question

"How do you increase throughput of a Kafka consumer?"

Layered answer:

1. Add consumers up to the partition count
2. Beyond that, increase partitions (and accept the re-keying cost)
3. Batch and parallelise the downstream write — usually the real bottleneck is the sink, not Kafka
4. Check whether processing is synchronous per message when it could be async or bulk

Then add the honest caveat: "more partitions means more open file handles, more rebalance cost, and ordering only within each partition — so I'd confirm the ordering requirement before scaling that way."

Words to know

In two minutes

• An event is an immutable fact; a log appends events in order and never edits them.
• Kafka is a log, not a queue: reading doesn't delete — retention (time or size) does.
• Topics split into partitions; ordering is guaranteed within a partition, never across the topic.
• Each consumer group keeps its own offsets, so billing, fraud and the lake read the same events independently.
• Producers write once and are decoupled in time from consumers: buffering, replay and fan-out come free.
• Within a group, each partition goes to one consumer; partitions cap parallelism.
• Consumer lag is the production health signal; commit offsets after the work, not before.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.