🐍 Backend with Python · Chapter 10

Background work: queues, workers, retries

Every job queue delivers some jobs twice, so the real question is whether paying a driver twice is possible in your code.

Study this lesson free in the app →

The basics

Background work is anything a request starts but shouldn't wait for: sending a receipt, resizing a photo, paying out drivers. The request drops a job on a queue and returns; a separate worker process picks it up.

The problem it solves: slow or flaky work (an SMTP server, a payment API, a 40-second report) inside a request makes users wait, ties up workers, and turns one partner's outage into your outage.

The mental model: an order spike in a restaurant kitchen. The waiter spikes the slip and goes back to the floor; a cook takes it when free. If the cook drops it half-done, the slip goes back on the spike, so some orders get cooked twice unless the slip number is checked.

Explained

The order spike in an Udupi kitchen

Lunch rush at a Matunga restaurant. The waiter writes slip #417, 2 masala dosa, spikes it on the kitchen rail and goes back to the tables. He doesn't stand there waiting for the dosa. That's the queue.

Any free cook pulls the next slip. That's a worker. The tawa isn't hot yet? He puts the slip back and tries again in a minute, then two, then four, and the cooks don't all rush the one tawa at the same second. That's backoff with jitter.

A cook makes the dosa, then gets called away before tearing the slip. Another cook sees #417 still on the rail and makes it again. That's at-least-once delivery. The fix isn't "cooks never get called away"; it's the pass-counter man who checks "#417 already went out" before plating. That's the idempotency key.

And the slip for "dosa with no batter"? After three tries it goes to the manager's tray instead of round and round the rail. That's the dead-letter queue.

The life of a job

The request's job ends at the 202 Accepted. Everything after it can fail, retry, and repeat without the user waiting.

Delivery guarantees, honestly

A worker does two things: run the handler and acknowledge the job. A crash can land between them, and the order you pick decides your guarantee.

| Order | If the worker dies in between | Guarantee | | --- | --- | --- | | ack, then run | job is gone, work never happened | at-most-once (lost jobs) | | run, then ack | job comes back, work happens again | at-least-once (duplicates) |

Nearly everyone picks at-least-once, because a lost payout is worse than a duplicate you can detect. SQS calls the gap the visibility timeout; Celery's acks_late=True acks after the task finishes.

"Exactly-once" doesn't exist end to end for side effects like sending an email or calling a bank. What you can build is effectively-once: at-least-once delivery plus a handler that notices it has already done this job.

Retry only what can succeed later: timeouts, connection errors, 429s and 503s. A 400 or an invalid email address fails the same way every time; send it to the DLQ on the first attempt.

A worker in 45 lines of stdlib

import queue, random, threading, time

jobs: queue.Queue = queue.Queue()
dead_letters: list[dict] = []
MAX_ATTEMPTS = 4
flaky = {"rcpt_2": 2, "rcpt_3": 99}       # fails this many times before working

def send_receipt(job: dict) -> None:
    if flaky.get(job["id"], 0) > 0:
        flaky[job["id"]] -= 1
        raise ConnectionError("SMTP timeout")
    print(f"  sent {job['id']} on attempt {job['attempt']}")

def backoff(attempt: int, base=0.05, cap=1.0) -> float:
    return random.uniform(0, min(cap, base * 2 ** attempt))   # full jitter

def worker() -> None:
    while True:
        job = jobs.get()
        if job is None:
            jobs.task_done()
            return
        try:
            send_receipt(job)
        except ConnectionError as e:
            if job["attempt"] >= MAX_ATTEMPTS:
                dead_letters.append({**job, "error": str(e)})
                print(f"  {job['id']} -> dead-letter after {job['attempt']} attempts")
            else:
                delay = backoff(job["attempt"])
                print(f"  {job['id']} failed (attempt {job['attempt']}), retry in {delay*1000:.0f} ms")
                threading.Timer(delay, jobs.put, [{**job, "attempt": job["attempt"] + 1}]).start()
        finally:
            jobs.task_done()

random.seed(7)
t = threading.Thread(target=worker)
t.start()
for rid in ("rcpt_1", "rcpt_2", "rcpt_3"):
    jobs.put({"id": rid, "attempt": 1})
time.sleep(2)
jobs.put(None)
t.join()
print("dead letters:", [d["id"] for d in dead_letters])

One run printed:

  sent rcpt_1 on attempt 1
  rcpt_2 failed (attempt 1), retry in 32 ms
  rcpt_3 failed (attempt 1), retry in 15 ms
  rcpt_3 failed (attempt 2), retry in 130 ms
  rcpt_2 failed (attempt 2), retry in 14 ms
  sent rcpt_2 on attempt 3
  rcpt_3 failed (attempt 3), retry in 214 ms
  rcpt_3 -> dead-letter after 4 attempts
dead letters: ['rcpt_3']

Retries re-enter the queue after a delay instead of sleeping in the worker, so the worker keeps serving other jobs. The exact interleaving changes from run to run because the retries are on timers. What this toy lacks is durability: kill the process and the queue is gone. That's what a real broker adds.

An idempotency key the database enforces

Checking if key in processed in Python and then inserting is a race: two workers can both pass the check. A unique constraint is atomic. Postgres uses the same ON CONFLICT … DO NOTHING syntax.

The ack that never came

ledger: list[tuple[str, int]] = []       # money actually moved
processed: set[str] = set()              # in real life: a UNIQUE column in Postgres

def pay_driver_naive(job):
    ledger.append((job["driver"], job["amount"]))

def pay_driver_idempotent(job):
    if job["key"] in processed:          # seen this exact job before: do nothing
        return
    ledger.append((job["driver"], job["amount"]))
    processed.add(job["key"])

job = {"key": "payout:2026-11-08:drv_17", "driver": "drv_17", "amount": 1840}

# at-least-once: the worker did the work, then died before acking,
# so the broker hands the same job out a second time
for handler in (pay_driver_naive, pay_driver_idempotent):
    ledger.clear()
    handler(job)        # first delivery, ack lost
    handler(job)        # redelivery
    print(f"{handler.__name__:<22} ledger={ledger}")

Backoff, herds, and real brokers

import random
random.seed(1)
base, cap = 0.5, 30.0
for attempt in range(1, 8):
    ceiling = min(cap, base * 2 ** attempt)
    print(f"attempt {attempt}: ceiling {ceiling:5.1f}s  slept {random.uniform(0, ceiling):5.2f}s")

attempt 1: ceiling   1.0s  slept  0.13s
attempt 2: ceiling   2.0s  slept  1.69s
attempt 3: ceiling   4.0s  slept  3.06s
attempt 4: ceiling   8.0s  slept  2.04s
attempt 5: ceiling  16.0s  slept  7.93s
attempt 6: ceiling  30.0s  slept 13.48s
attempt 7: ceiling  30.0s  slept 19.55s

Why jitter: 20,000 SMS jobs fail during a 60-second provider outage. With a fixed 1 s retry they hit the provider together every second; when it recovers, they all arrive in the same instant and knock it over again. Full jitter (uniform(0, ceiling)) spreads them across the window.

The tools: Celery (RabbitMQ or Redis broker; autoretry_for, retry_backoff, max_retries, acks_late). RQ (Redis, simple; Retry(max=3, interval=[10, 30, 60])). arq (Redis, asyncio-native). SQS (visibility timeout, and a redrive policy that moves a message to a DLQ after maxReceiveCount). Every one of them is at-least-once.

Where background jobs go wrong

• Non-idempotent handlers. Duplicates aren't a bug in the queue. Design for them.
• Enqueueing before the commit. The worker runs before the row exists, or the transaction rolls back and the job runs anyway. Enqueue after commit, or write the job into an outbox table in the same transaction and let a relay publish it.
• Retrying permanent errors. A 400 or an invalid address retried 10 times is 10 wasted calls and a late DLQ entry.
• No cap, no jitter. Unbounded retries hide outages; synchronised retries cause the next one.
• Fat job payloads. Send {"ride_id": 9001}, not the whole ride. The row may change before the job runs; read it fresh.
• A DLQ nobody watches. Alert on DLQ depth, and have a replay script ready.
• Visibility timeout shorter than the job. A 5-minute report with a 30 s lease gets picked up by a second worker while the first is still running.

Words to know

In two minutes

• Slow or flaky work moves out of the request: enqueue a small job, return 202, let a worker process it.
• Workers ack after the handler runs, so a crash in between redelivers the job: at-least-once, and duplicates are normal.
• Make handlers idempotent with a key derived from the job's meaning, enforced by a unique constraint in the same transaction as the effect.
• Retry only transient errors (timeouts, 429, 503) with capped exponential backoff and full jitter.
• Permanent errors and exhausted retries go to a dead-letter queue that someone alerts on and can replay.
• Enqueue after commit, or use an outbox table, so jobs never point at rows that don't exist.
• Celery, RQ, arq and SQS all give at-least-once; none of them make your handler safe for you.

Test yourself

There are 9 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.