⚡ Streaming & Kafka · Chapter 4

Choosing a Partition Key

The partition key decides two things at once: what stays ordered, and what gets overloaded.

Study this lesson free in the app →

The basics

A partition is one ordered slice of a Kafka topic, and the partition key is the field (like ride_id) whose hash decides which slice each event goes to.

The problem it solves: one machine can't handle every ride event in India at peak, so a topic is split into many partitions spread across brokers and read in parallel. But splitting has a price: Kafka only keeps events in order within one partition. The key is how you choose which events must stay together — and that same choice decides whether load spreads evenly or piles onto one machine.

The mental model: billing counters at a stadium. Send people to counters by ticket number and every counter is busy, but a family gets split up. Send them by surname initial and the "S" counter has a 40-minute queue while "Q" is empty. Pick the key that keeps together what must stay in order, and has enough distinct values to spread the crowd.

Explained

Pick ride_id and every event of one ride stays in order, spread evenly across the cluster.

Pick city and Bangalore melts one broker while Shillong idles.

Pick event_timestamp and everything arriving this second lands on the same partition — you have built a queue with extra steps.

Billing counters at a stadium

Twenty counters, one crowd.

Key by surname initial — the S counter has a 40-minute queue, the Q counter reads a newspaper. That is skew.

Key by ticket number — even spread, every counter busy. But a family arriving together gets split across counters, so nothing keeps them in sequence.

Key by arrival minute — everyone in the same minute goes to one counter. Perfectly ordered, perfectly useless.

The trade is always the same: ordering pulls related events together, throughput pushes them apart.

How the key becomes a partition

partition = hash(key) % partition_count

Consequences that follow directly from that formula:

• Same key → same partition → ordered relative to each other
• No key → round-robin → maximum spread, zero ordering
• Change the partition count → the mapping changes → old and new events for a key can sit in different partitions, so ordering breaks during the transition
• Key cardinality caps your spread. 5 distinct keys cannot fill 50 partitions

So: choose the key by asking "what must stay in order?" then verify "is it high-cardinality and evenly distributed?"

Good vs bad keys for RideFlow

stream                key            why
────────────────────────────────────────────────────────────
ride_events           ride_id        all events of one ride ordered;
                                     millions of ids = even spread

driver_location       driver_id      per-driver path stays sequential;
                                     high cardinality

payments              ride_id        charge/refund order preserved

✗ city                              10 keys, Bangalore = hotspot
✗ event_timestamp                   all of 'now' → one partition
✗ country                           1 key = 1 partition = no scale

When the natural key is skewed anyway

Sometimes reality is skewed — one merchant is 30% of transactions, one city is half the rides. Standard mitigations:

1. Salting — key becomes merchant_id:bucket where bucket is 0..N. Spreads the hot key across N partitions, at the cost of ordering within that merchant (you now need a merge/sort downstream)
2. Two-tier routing — hot keys get a dedicated topic with more partitions; everything else shares the default
3. Reduce the event, not the key — aggregate location pings client-side to 1/second instead of 1/200ms
4. Re-key at the boundary — ingest by ride_id, then re-partition by city only in the analytical layer, where skew costs a slow task rather than a dropped ping

Say salting in an interview and follow it immediately with "which breaks per-key ordering, so I'd only do it where ordering isn't required". The caveat is what makes it a senior answer.

Watch keys land in partitions

Same key, same partition — every time. That is the whole ordering guarantee: rider_7's events can never be split across partitions, so they stay in order. (Kafka uses murmur2, not a sum of characters, but the shape is the same: hash, then modulo the partition count.)

Now add partitions — and watch keys move

Same keys, 6 partitions instead of 4: most keys land somewhere new. A consumer that relied on per-key order across the change can now see events out of order. Decide the partition count up front.

Ordering pulls together. Throughput pushes apart. The key is where you settle the fight.

High cardinality, evenly spread, and matching the thing that must stay in order.

Words to know

In two minutes

• partition = hash(key) % partition_count: same key, same partition, kept in order.
• Choose the key from the ordering requirement first (ride_id for ride status changes), then check its distribution.
• Good keys are high-cardinality and even; city, status, country and timestamps make hotspots.
• Changing the partition count remaps keys, so ordering can break across the transition — size partitions up front.
• Consumer rebalancing spreads partitions, not load within a partition; skew stays skew.
• For a genuinely hot key: salt it, give it its own topic, shrink the events, or re-key downstream — and say what ordering you lose.
• The same trade-off rules Spark shuffles, warehouse clustering and DynamoDB keys.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.