🐍 Python for Data Engineers · Chapter 8

Files, APIs, retries and backoff

Every ingestion job is a negotiation with something unreliable. The code that handles the failure is bigger than the code that handles success.

Study this lesson free in the app →

The basics

Resilient ingestion is the code that pulls data from files and APIs you do not control and still produces a complete, correct result when they misbehave.

The problem it solves: partner APIs time out, rate-limit you (429), return 503 during their deploys, paginate across thousands of pages, and occasionally send HTML instead of JSON. A three-line requests.get works in a demo and fails at 2 AM — or worse, half-succeeds and publishes 62% of yesterday's payments as if it were all of them.

The mental model: every external call is a negotiation with something unreliable. Classify each failure as their fault (5xx, 429, timeouts — wait and retry) or your fault (4xx — stop and fail loudly). Keep a bookmark (checkpoint) so a crash on page 900 resumes at 900, and keep the raw response so a parser bug is a replay, not a re-pull.

Explained

The happy path is three lines:

r = requests.get(url)
for row in r.json()["data"]:
    sink.write(row)

The production version is forty. Not because the API is complicated, but because it times out, rate-limits you, paginates, returns a 502 twice a week, and once sends HTML where JSON should be.

Tuesday, 2:15 AM

Your job pulls yesterday's payments from a partner API. It has run fine for six months.

Tonight the partner is deploying. For ninety seconds the API returns 503. Your job — with no retry — fails, alerts on-call, and yesterday's revenue is missing from the morning dashboard.

With three retries and exponential backoff, the same outage is invisible: attempt 1 fails, wait 1s, fail, wait 2s, fail, wait 4s, succeed. Total delay: seven seconds. Nobody is woken up.

That is the entire business case for this lesson.

Anatomy of a resilient fetch

The step people skip is the last-but-one. Without a checkpoint, a failure on page 900 of 1000 means starting at page 1.

Retry with exponential backoff and jitter

import random, time
import requests

RETRYABLE = {429, 500, 502, 503, 504}

def fetch(url, params, attempts=5, timeout=30):
    for attempt in range(attempts):
        try:
            r = requests.get(url, params=params, timeout=timeout)

            if r.status_code in RETRYABLE:
                wait = float(r.headers.get("Retry-After", 0)) or backoff(attempt)
                time.sleep(wait)
                continue

            r.raise_for_status()          # 4xx = our bug, fail loudly
            return r.json()

        except (requests.Timeout, requests.ConnectionError):
            if attempt == attempts - 1:
                raise
            time.sleep(backoff(attempt))

    raise RuntimeError(f"{url} failed after {attempts} attempts")

def backoff(attempt):
    return min(2 ** attempt, 60) + random.uniform(0, 1)   # jitter

Four decisions worth defending in an interview: timeout always set, only retry what is retryable, honour Retry-After, add jitter.

Why jitter?

An API goes down for 30 seconds. 500 of your workers are mid-request. All of them retry with backoff of exactly 1s, 2s, 4s, 8s, 16s, 32s.

Pagination, three flavours

# 1. Cursor / token — the good kind, stable under inserts
def fetch_all(url):
    cursor = None
    while True:
        page = fetch(url, {"cursor": cursor, "limit": 1000})
        yield from page["data"]                 # stream, never accumulate
        cursor = page.get("next_cursor")
        if not cursor:
            return

# 2. Offset / page number — simple, but shifts if rows are inserted mid-run
#    page=1&size=1000, page=2&size=1000 …

# 3. Time window — the one to prefer for event data
#    ?updated_after=2026-07-31T00:00:00Z&updated_before=…
#    replayable, idempotent, and the window IS your checkpoint

Note yield from: the fetcher is a generator, so 4 million rows never sit in memory at once. Same lesson as before, applied to the network.

The rules that make ingestion boring

Boring is the goal. Six rules get you there:

1. Always set a timeout. Without one, requests waits forever and your job hangs instead of failing — far worse, because nothing alerts
2. Use a Session. Connection pooling makes a 1000-page pull several times faster
3. Land raw before transforming. Write the exact response to object storage; parse from there. When the parser has a bug you replay, you do not re-pull
4. Checkpoint your cursor. Store the last successful cursor/window, so a crash resumes rather than restarts
5. Make the whole pull idempotent. Re-running the same time window must produce the same rows — partition by that window and overwrite it
6. Log the request id. Every serious API returns one; it is the only thing that makes a support ticket answerable

Files: read big, write safe

import csv, gzip, json
from pathlib import Path

# stream a gzipped JSONL file — never decompress to disk first
with gzip.open("events.jsonl.gz", "rt") as fh:
    for line in fh:
        event = json.loads(line)

# csv.DictReader streams too, and gives you named fields
with open("rides.csv", newline="") as fh:
    for row in csv.DictReader(fh):
        ...

# atomic write — a crash must never leave a half-written file
# that a downstream job happily reads as complete
tmp = Path("out.parquet.tmp")
write(tmp)
tmp.rename("out.parquet")        # rename is atomic on the same filesystem

That last pattern matters more than it looks. Partial files are the classic silent data-quality bug: the job "succeeded", the file exists, and 40% of yesterday is missing.

Words to know

In two minutes

• The failure-handling code is bigger than the success code, and that is normal for ingestion.
• Always set a timeout; a hung job is worse than a crashed one because nothing alerts.
• Retry what is the server's fault (5xx, 429, timeouts) with exponential backoff plus jitter, honouring Retry-After; never retry 4xx.
• Prefer cursor or time-window pagination; offsets shift when rows are inserted mid-run. Stream pages with a generator, never accumulate.
• Land the raw response first, then parse — a parser bug becomes a replay instead of a re-pull from a rate-limited partner.
• Checkpoint the cursor or window so a crash resumes, and make re-running a window idempotent (overwrite that partition).
• Never swallow a failure into a partial success: re-raise after retries, and gate publishing on a row-count check against the last run.
• Write files atomically (temp file + rename on the same filesystem) so downstream never reads a half-written file.

Test yourself

There are 9 questions, 3 puzzles and 7 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.