🏗 Scale & System Design · Chapter 2

Idempotency — surviving the same event twice

Distributed systems retry. Retrying is not the bug — charging twice is.

Study this lesson free in the app →

The basics

Idempotency means an operation gives the same result whether it runs once or many times. Pressing a lift button five times still calls one lift.

The problem it solves: distributed systems retry. A Kafka consumer writes a charge, crashes before committing its offset, restarts and reads the same event again. A network timeout makes a client resend a request that actually succeeded. You cannot stop these retries — they are how systems avoid losing data. What you can do is make the retry change nothing, so the customer is charged ₹280 once, not twice.

The mental model: a hotel reception desk that asks "which booking?" before issuing a key card. With a booking number it can tell a retry from a new request. That number is the idempotency key, and it must come from whoever created the request, not be invented on arrival.

Explained

A payment consumer processes an event, writes the charge, then crashes before committing the offset.

On restart it reads the same event again. Two charges, one ride, one very angry customer and one very expensive support ticket.

You cannot prevent the retry. You design so the retry changes nothing.

The hotel key card

You ask reception for a key card. They make one.

You ask again — maybe the first request never reached them, maybe it did and the reply was lost. You genuinely do not know.

Non-idempotent reception hands you a second card and bills you twice.

Idempotent reception asks "which booking?", sees a card was already issued for booking 9001, and hands you the same one. Same result, no matter how many times you ask.

The booking number is the idempotency key. Without it, reception cannot tell a retry from a new request.

The three delivery semantics

• At-most-once — commit before processing. Never duplicates, can lose data. Acceptable for metrics; unacceptable for payments
• At-least-once — process, then commit. Never loses, can duplicate. The industry default
• Exactly-once — no duplicates, no loss. Achievable only within a coordinated boundary

So the real-world pattern is:

at-least-once delivery + idempotent processing = effectively-once

That phrase is the answer to almost every reliability question in an interview. Learn to say it and then explain the two halves.

Four ways to make a write idempotent

-- 1. Unique constraint + upsert (the workhorse)
CREATE UNIQUE INDEX ON payments (event_id);

INSERT INTO payments (event_id, ride_id, amount)
VALUES ('evt_901', 'r_9001', 280.50)
ON CONFLICT (event_id) DO NOTHING;

# 2. Dedup by key before the side effect
def handle(event, seen):                 # seen = Redis / state store
    if seen.exists(event["event_id"]):
        return                            # already done, silently succeed
    charge(event, idempotency_key=event["event_id"])
    seen.set(event["event_id"], ttl=7 * 86400)

-- 3. Idempotent batch: delete-then-insert the whole partition
DELETE FROM rides_daily WHERE ride_date = '2026-07-31';
INSERT INTO rides_daily SELECT ... WHERE ride_date = '2026-07-31';
-- re-running the job produces the same table, never doubled rows

4. Pass an idempotency key to the external API.
   Stripe, Razorpay, and every serious payment provider accept one
   for exactly this reason. Reuse the event_id.

Where the state lives is the hard part

Remembering "have I seen this id?" across billions of events is the real engineering.

• Unique constraint in the sink — simplest, correct, but costs a write and needs the sink to support it
• Bounded dedup window — keep ids for N days in Redis/RocksDB; assumes duplicates arrive close together, which is true for retries and false for backfills
• Bloom filter — tiny memory, may say "seen" when it wasn't. Never use alone where a false positive means a dropped payment
• Natural idempotency — design the write so repetition is harmless: SET status = 'completed' is idempotent; balance = balance + 100 is not

That last bullet is the highest-leverage idea in the topic. Prefer writes that are naturally idempotent over machinery that detects duplicates.

At-least-once delivery + idempotent processing = effectively once.

SET is idempotent. ADD is not.

The idempotency key is the question "which one did you mean?"

Gotchas

• Delete-then-insert per row to "refresh" — between the two statements a reader sees nothing, and a crash leaves the row missing. Use upsert or MERGE
• Deduping on a natural key that isn't unique — (ride_id, status) looks unique until a ride is cancelled twice legitimately
• A dedup window shorter than your retry window — a 1-hour Redis TTL cannot stop a 6-hour DLQ replay
• Idempotent consumer, non-idempotent side effect — you skip the DB write but still send the SMS. Guard the whole handler, not just the database call
• Generating the id inside the consumer — uuid4() on retry is a new id, so dedup can never work. The id must come from the producer, with the event

The RideFlow answer

"Kafka gives me at-least-once, so duplicates are a certainty rather than a risk. Every event carries a producer-generated eventid. The payment consumer writes into a table with a unique index on event_id and uses INSERT … ON CONFLICT DO NOTHING, and passes the same event_id as the idempotency key to the payment provider so its side is protected too. That makes reprocessing free — I can replay from any offset after a bug fix without reconciling anything by hand."_

Expect the follow-up: "and if the same ride is legitimately charged twice?" — that is why the key is event_id, not ride_id.

Words to know

In two minutes

• Retries are unavoidable in distributed systems; the goal is that a retry changes nothing.
• At-most-once loses data, at-least-once duplicates it, exactly-once holds only inside a coordinated boundary.
• The working formula: at-least-once delivery + idempotent processing = effectively once.
• Every event needs a producer-generated event_id; a uuid created by the consumer is new on every retry and defeats dedup.
• Make writes idempotent with a unique constraint plus upsert, a dedup store, partition overwrite for batches, or idempotency keys passed to payment APIs.
• Prefer naturally idempotent writes (SET a value) over increments (ADD to a value).
• Guard the whole handler, including side effects like SMS; size any dedup window to your longest replay horizon.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.