🐍 Python for Data Engineers · Chapter 7

Generators and memory — 50M rows without dying

The difference between a pipeline that runs and one that gets OOM-killed is usually one character: [ versus (.

Study this lesson free in the app →

The basics

A generator is a function or expression that produces values one at a time, on demand, instead of building them all in memory first.

The problem it solves: pipelines read files and streams far bigger than RAM. f.readlines() on a 12 GB file of ride events on an 8 GB box gets the process killed. Iterating the same file line by line finishes in minutes, because only one line is ever held.

The mental model: a list is a bucket — you fill it completely, then carry it. A generator is a hose — at any instant only a cupful is inside, so the tank can be any size. The cost of the hose: water flows past once. You cannot rewind it or ask how much is in it.

Explained

Your job reads a 12 GB file of ride events on a box with 8 GB of RAM.

lines = f.readlines() — killed. for line in f: — finishes in four minutes.

Same file, same machine, same logic. The second one never holds more than one line at a time.

The bucket and the hose

You need to move water from a tank to a field.

The bucket approach: fill an enormous drum, carry it across, empty it. You need a drum as big as the tank, and a way to lift it. Fail either and nothing moves at all.

The hose approach: water flows continuously. At any instant only a cupful is inside the hose. The tank can be a thousand times bigger than anything you could carry, and it still works.

A list is the drum. A generator is the hose. Most beginners reach for the drum because you can see what's in it.

Where memory actually goes

Peak memory is decided at step 2 and never recovers. Everything after it inherits whatever shape you chose.

The four shapes

# 1. list comprehension — builds everything, then returns
fares = [parse(l).fare for l in lines]          # N objects in RAM

# 2. generator expression — produces on demand
fares = (parse(l).fare for l in lines)          # 1 object in RAM

# 3. generator function — same thing, with logic
def parse_events(path):
    with open(path) as fh:
        for line in fh:                          # the file itself is lazy
            event = parse(line)
            if event is not None:
                yield event                      # hands one out, then pauses

# 4. the consumer decides when it ends
total = sum(e.fare for e in parse_events(path) if e.is_billable)

Nothing in shape 3 runs until something iterates it. yield freezes the function mid-execution and resumes it on the next next().

Watch yield pause and resume

The pause-and-resume is the whole idea. A generator is a function with a bookmark.

Predict the output

events = (e for e in load_events())   # generator

print(sum(1 for _ in events))
print(sum(1 for _ in events))

Assume load_events() returns 1,000 events.

itertools: the batching toolkit

from itertools import islice, batched, chain

# batched (3.12+) — chunk a stream for bulk inserts
for chunk in batched(parse_events(path), 10_000):
    warehouse.insert_many(chunk)          # 10k rows in RAM, not 50M

# islice — take the first N without loading the rest (great for dev)
sample = list(islice(parse_events(path), 100))

# chain — treat many files as one stream
all_events = chain.from_iterable(parse_events(p) for p in paths)

# pre-3.12 batching, worth memorising
def batches(iterable, size):
    it = iter(iterable)
    while chunk := list(islice(it, size)):
        yield chunk

islice(stream, 100) is how you develop against a 12 GB file without waiting four minutes per run.

The batch size trade-off

Streaming one row at a time is memory-perfect and network-terrible — 50M individual inserts is 50M round trips.

So real pipelines stream and batch:

| Batch size | Memory | Round trips | Retry cost | | --- | --- | --- | --- | | 1 | tiny | catastrophic | trivial | | 10,000 | fine | good | lose 10k, redo them | | 5,000,000 | OOM risk | best | a failure wastes hours |

Pick the batch by what you can afford to redo, not by what fits. A 10k batch that fails costs seconds; a 5M batch that fails at 97% costs the whole run.

And the memory that actually kills jobs is rarely the batch — it is an accidental accumulator: a seen set that grows unbounded, a list you append to "just for logging", a dict cache nobody bounded.

Words to know

In two minutes

• A generator yields one item at a time; peak memory is one item, not the whole dataset.
• [...] builds a list, (...) builds a generator — often the one character between a job that runs and one that is OOM-killed.
• yield pauses the function with its local state intact; calling a generator function runs none of its body until the first next().
• A generator can be walked once. A second pass gives zero items and raises nothing — the classic silent bug.
• Stream and batch together: iterate lazily, write in chunks sized by what you can afford to redo (about 10k rows).
• Laziness is wasted if anything inside the loop grows without bound — the unbounded seen set is what really OOMs streaming jobs.
• sorted, list, len, json.dumps and tee with a lagging branch all quietly bring the whole stream back into memory.

Test yourself

There are 9 questions, 3 puzzles and 7 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.