🐍 Backend with Python · Chapter 14

Rate limiting, timeouts and backpressure

Every service has a maximum speed; the only choice is whether it says "no" early and politely, or says nothing at all while it falls over.

Study this lesson free in the app →

The basics

Three tools keep a service alive when demand outruns capacity. They all do the same thing: refuse or abandon work early, so the work you do accept finishes.

The problem it solves: without limits, one noisy client, one slow partner or one traffic spike fills every worker, connection and queue — and then every request fails, not just the excess.

The mental model: a service is a pipe with a fixed width. You can't widen it at 9 PM; you can only control what you let in, how long you wait, and how much you let pile up.

• Rate limiting — cap how fast each client may call you. A token bucket refills at a steady rate and allows short bursts. Over the limit → 429 Too Many Requests with Retry-After.
• Timeouts — cap how long you wait on anything outside your process: DB, partner API, queue. No timeout means a hung partner holds your worker forever.
• Backpressure — when you're full, push back on the sender: a bounded queue that blocks or rejects, a semaphore that caps concurrency. Load shedding is the polite form: a fast 503 now beats a 30 s timeout later.
• Retry budget — retries are extra load at the worst moment, so they need backoff with jitter and a cap.

Explained

The Tirupati darshan queue

The temple can let roughly a fixed number of pilgrims past the sanctum each hour. That's capacity, and no amount of shouting changes it.

• Rate limiting: each pilgrim gets a timed token for a slot. A tour operator who shows up with 400 people at once gets told "your slot is 4 PM" — that's 429 with Retry-After.
• Bounded queue: the waiting halls hold a set number of people. When they're full, new arrivals are told at the gate to come back tomorrow — not let in to stand in the sun for 30 hours. That's load shedding.
• Timeout: a priest won't wait forever for a pilgrim who has stepped away; the line moves on.
• Retry storm: imagine everyone turned away at 6 AM is told "try again at 7 AM" — at 7 the gate sees today's crowd plus yesterday's. Staggered return times are jitter.

The temple works because it says no at the gate, not inside the sanctum.

Which tool goes where

| Tool | Protects | Where it lives | On trigger | | --- | --- | --- | --- | | Rate limit | you, from one client | edge: gateway or middleware, keyed by API key / user / IP | 429 + Retry-After | | Timeout | you, from a slow dependency | every outbound call: DB, HTTP, Redis | fail fast, free the worker | | Concurrency limit | a dependency, from you | semaphore around the call | wait briefly, or shed | | Bounded queue | your memory and latency | between accept and process | block the producer, or 503 |

Algorithms:

• Token bucket — tokens refill at rate/s up to capacity; each request spends one. Allows bursts, O(1) state. The default.
• Sliding-window log — keep each request's timestamp; allow if fewer than N in the last W seconds. Exact, but one timestamp per request.
• Fixed window — count per calendar minute. Cheap, but allows 2× the limit across a boundary (N at 0:59, N at 1:00).

In-process limiters only see one pod. With 20 pods behind a load balancer, put the counter in Redis or at the gateway.

A token bucket, refill and spend

Run: 10 allowed, 2 rejected, then True False at t = 0.2, and after a 10 s idle exactly 10 allowed out of 20 — the cap stops idle time from banking unlimited burst.

429 with Retry-After, per API key

@app.middleware("http")
async def rate_limit(request: Request, call_next):
    key = request.headers.get("x-api-key", "anon")
    b = buckets.setdefault(key, TokenBucket(rate=2, capacity=3, clock=clock))
    if not b.allow():
        wait = math.ceil(b.retry_after())
        return JSONResponse({"detail": "rate limit exceeded"}, status_code=429,
                            headers={"Retry-After": str(wait)})
    return await call_next(request)

With TestClient and a frozen clock:

[200, 200, 200, 429, 429]                       # partner-a bursts 5
429 1 {'detail': 'rate limit exceeded'}         # Retry-After: 1 (seconds, whole number)
partner-b: 200                                  # separate bucket per key
after 1 s: 200                                  # 2 tokens refilled

The limiter runs before the handler, so a rejected call costs microseconds, not a DB query. Retry-After is in whole seconds — round up, or clients come back too early and get another 429.

The sliding window's edge

lim = SlidingWindowLog(limit=3, window=60, clock=clock)   # 3 OTPs per 60 s
# allow(): drop timestamps <= now - 60, then allow if fewer than 3 remain

for at in [0, 10, 59, 59.5, 60, 70, 71]:
    t[0] = at
    print(at, lim.allow("9876543210"))

Timeouts on everything that leaves the process

# top-level await: run in `python -m asyncio`, Jupyter, or inside an async def
# a local "partner" that accepts the connection and then never answers
async with httpx.AsyncClient(timeout=httpx.Timeout(5.0, read=0.3)) as client:
    await client.get(f"http://127.0.0.1:{port}/rate")
# → httpx: ReadTimeout after 0.4s

await asyncio.wait_for(asyncio.sleep(10), timeout=0.5)
# → wait_for: TimeoutError after 0.5s   (the inner coroutine is cancelled)

httpx.Timeout(5.0, read=0.3) sets connect / write / pool to 5 s and read to 0.3 s — four separate clocks. httpx's default is 5 s; timeout=None turns it off, which you should never do for a partner call.

asyncio.wait_for caps any awaitable — a DB call, a lock, a queue get. On Python 3.11+ it raises the built-in TimeoutError.

Pick the number from the budget, not from hope. If your endpoint promises p99 < 800 ms and calls a partner whose p99 is 250 ms, a 300 ms read timeout with one retry fits; a 30 s timeout means a hung partner holds your worker 37× longer than the whole request was allowed to take. Better still, pass a deadline down: each hop gets what's left, not a fresh 30 s.

Backpressure: bounded queues and semaphores

q = asyncio.Queue(maxsize=100)
for i in range(250):                     # a burst of 250 webhook events
    try:
        q.put_nowait(i)
    except asyncio.QueueFull:
        shed += 1                        # answer 503 now, not 30 s later
# accepted 100 shed 150 queue size 100

Or make the producer wait — await q.put(x) on a maxsize=2 queue paced a fast producer to a slow consumer:

['put 0', 'put 1', 'got 0', 'put 2', 'got 1', 'put 3', 'got 2', 'put 4', 'got 3', 'got 4']

Protect a dependency with a semaphore:

partner_limit = asyncio.Semaphore(10)    # partner allows 10 concurrent calls
async def call_partner(i):
    async with partner_limit:
        ...
# 100 calls, peak concurrency 10 took 0.21s

An unbounded queue is not a buffer, it's a delay line: when input outruns output it grows until every item in it is older than the client's timeout, and you spend 100% of your capacity on answers nobody is waiting for.

Thursday 14 March 2024, 14:00 — the retry storm

A payments partner slows to 4 s at 14:00. Our checkout calls an order service, which calls a wallet service, which calls the partner. Each layer retries 3 times, no backoff.

1 layers x 3 retries -> 4 calls at the bottom per user request
2 layers x 3 retries -> 16 calls at the bottom per user request
3 layers x 3 retries -> 64 calls at the bottom per user request

So at 300 checkouts/s the partner sees up to 19,200 calls/s. It recovers at 14:05, takes the flood, and falls over again at 14:06. Every retry fired after the same fixed 1 s, so the waves were synchronised.

The fixes we shipped:

1. Retry at one layer only — the one closest to the failing call.
2. Exponential backoff with full jitter: sleep = random() × min(cap, base × 2^attempt) → [0.013, 0.169, 0.306, 0.204, 0.793, 1.438] s — spread out, not in lockstep.
3. A retry budget: retries ≤ 10% of requests. With the partner fully down, 1,000 failing requests got 100 retries, not 3,000.
4. Only retry what's retryable — timeouts, 503, 429 (after Retry-After). Never a 400, and never a non-idempotent POST without an idempotency key (be-010).

Words to know

In two minutes

• Every service has a fixed capacity; overload control is deciding what to refuse, early.
• Rate limit per client at the edge; token bucket allows bursts with O(1) state; reject with 429 and Retry-After.
• Put a timeout on every outbound call (connect, read, pool); size it from the latency budget, ideally as a propagated deadline.
• Backpressure: bounded queues block or reject producers; a semaphore caps concurrency to each dependency.
• Load shedding returns a fast 503 instead of queueing work that will outlive the client's timeout.
• Retries multiply: 3 retries at 3 layers = 64× calls. Retry at one layer, with backoff + jitter, under a budget.
• In-process limiters see one pod; shared limits belong in Redis or the gateway.

Test yourself

There are 10 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.