🐍 Backend with Python · Chapter 11

Caching: where, how long, and how to invalidate

A cache is a copy of the truth that is allowed to be wrong for a while — your whole job is deciding how wrong, for how long, and who gets told when it changes.

Study this lesson free in the app →

The basics

A cache is a fast copy of an answer you already computed, kept so the next request skips the slow work — a database query, a partner API call, a heavy render.

The problem it solves: the same expensive question gets asked thousands of times a minute, and the answer barely changes.

The mental model: every cached value is a photocopy with an expiry stamp. The original (your database) is the truth. The copy is cheaper to read but can drift. You must decide three things, and each is a trade-off, not a default:

1. Where the copy lives — the browser, a CDN, your process's memory, or a shared Redis.
2. How long it may be stale — the TTL (time to live).
3. How it dies early when the truth changes — invalidation.

Explained

The specials board at a Udupi hotel

Lunch rush. Twenty waiters, one kitchen. If every waiter walked into the kitchen to ask "is the bisi bele bath still on?" the cook would never cook.

So the cook writes the day's specials on a chalkboard by the door. Waiters read the board — fast, no queue. That board is a cache.

• Stale data: the bisi bele bath ran out at 1:10, but the board still says it's on. A waiter takes an order the kitchen can't fill.
• TTL: the manager's rule — "redo the board every hour". Simple, but up to an hour wrong.
• Invalidation: the cook wipes that one line the moment the dish runs out. Accurate, but someone has to remember to do it.
• Stampede: the board is wiped at exactly 1:00 PM. All twenty waiters walk into the kitchen at once to ask what's on. Lunch stops.

Every caching decision in a backend is one of these four.

Where a cached answer can come from

The closer to the user, the faster the hit and the harder it is to take back. You can delete a Redis key in a millisecond; you cannot reach into a phone and delete what max-age=86400 told it to keep for a day.

A TTL cache, line by line

Ran with fake time: 1450, 1450 at +29.9 s, None at +30 s, and the dict is empty afterwards.

The price that would not change

from functools import lru_cache

PRICES = {"masala-dosa": 80}

@lru_cache(maxsize=2)
def price(item):
    return PRICES[item]

print(price("masala-dosa"), price("masala-dosa"))
PRICES["masala-dosa"] = 95
print(price("masala-dosa"))
print(price.cache_info())

HTTP caching: Cache-Control, ETag and 304

@app.get("/menu")
def menu(request: Request):
    body = json.dumps(MENU, sort_keys=True).encode()
    etag = '"' + hashlib.sha256(body).hexdigest()[:16] + '"'
    headers = {"ETag": etag, "Cache-Control": "public, max-age=60"}
    if request.headers.get("if-none-match") == etag:
        return Response(status_code=304, headers=headers)
    return Response(body, media_type="application/json", headers=headers)

@app.get("/me/wallet")
def wallet(response: Response):
    response.headers["Cache-Control"] = "no-store"
    return {"balance_paise": 125000}

With FastAPI's TestClient, this printed:

200 public, max-age=60 "f35f0525ab68d85f" 73   # first fetch: 73-byte body
304 0                                           # sent If-None-Match: empty body
200 True                                        # menu changed → new ETag, full body
no-store                                        # wallet: never stored anywhere

max-age=60 = reuse without asking for 60 s. ETag = after that, ask "has it changed?" and get a cheap 304. no-store for anything personal or money-shaped.

Cache-aside, write-through, and the two ways to invalidate

Cache-aside (lazy): read cache → on miss read DB → store. Writes go to the DB and then delete the key.

def update_city(key, city):
    db[key]["city"] = city
    cache.pop(key, None)        # delete, don't overwrite

Run: Pune Pune db reads: 1 → update → Kochi db reads: 2.

Why delete instead of set? Two writers race: A writes DB, B writes DB, B sets cache, A sets cache — the cache now holds A's older value until TTL. A delete can't go stale; worst case is one extra miss.

Write-through: every write updates DB and cache together. Reads are always warm, writes are slower, and you cache things nobody reads.

Invalidation strategies, cheapest first:

• TTL only — accept up to N seconds stale. Right for feeds, menus, exchange rates.
• Delete on write — exact for keys you can name.
• Versioned keys — menu:v42; bump the version and old keys simply age out. Great when one write affects many keys.
• Events — publish "user 42 changed" so every pod drops its in-process copy.

Diwali sale, 8:00:00 PM IST

The home-feed key had a 300 s TTL, set when the sale banner went live at 7:55. At 8:00:00 it expired. Traffic: 18,000 req/s. Rebuilding the feed: a 400 ms query.

In those 400 ms, ~7,200 requests missed and each fired the same query at a 50-connection Postgres pool. Queries queued, the rebuild took 9 s, and everything else on that DB timed out with it.

The same shape, reproduced with asyncio — 500 concurrent readers of one cold key:

async def get_locked(key):
    if key in cache:
        return cache[key]
    async with locks.setdefault(key, asyncio.Lock()):
        if key in cache:            # filled while we waited
            return cache[key]
        cache[key] = await load_from_db(key)
        return cache[key]

naive  db calls: 500
locked db calls: 1

Fixes, in order: one rebuilder per key (the lock above, per process; a Redis SET NX lock across pods), jittered TTLs (300 s ± 10% put at most 196 of 10,000 keys in any one second), and serve-stale-while-revalidating for hot keys.

Gotchas

• Caching personal data as public. A CDN will happily hand Asha's wallet to Ravi. Personal → private or no-store
• no-cache ≠ don't cache. It means store, but revalidate every time. no-store is the "never keep it" one
• In-process caches on 40 pods = 40 different truths. Fine for config, wrong for anything a user just edited
• lru_cache on a method keeps self alive forever and has no TTL. Use it for pure functions only
• Caching errors and None. A 5xx cached for 300 s is an outage you built yourself. Cache negatives briefly (5–30 s) on purpose, not by accident
• No cache metrics. Without hit ratio per key prefix you are tuning blind — next lesson

Words to know

In two minutes

• A cache is a copy of the truth with an expiry stamp; you trade correctness for speed on purpose.
• Decide three things: where the copy lives, how long it may be stale (TTL), and how it is invalidated early.
• Layers: browser → CDN → in-process → Redis → DB. Closer to the user is faster and harder to take back.
• Cache-aside is the default: read-through on miss; on write, update the DB then delete the key.
• HTTP gives you caching for free: Cache-Control max-age, ETag + If-None-Match → 304, and no-store for anything personal.
• Hot keys expiring cause stampedes; fix with a per-key lock or single-flight, jittered TTLs, and serve-stale.
• functools.lru_cache has no TTL and never sees its source change — pure functions only.
• DB load = traffic × miss ratio, so a small hit-ratio drop can double the database load.

Test yourself

There are 10 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.