🐍 Python for Data Engineers · Chapter 9

Concurrency: threads, processes, asyncio

Ask one question — is this job waiting, or is it computing? — and the right concurrency model falls out immediately.

Study this lesson free in the app →

The basics

Concurrency means making progress on several tasks at once — either by overlapping their waiting (threads, asyncio) or by running them on several CPU cores (processes).

The problem it solves: most data jobs spend their time waiting — for a partner API, Postgres, S3. A pull of 1,000 pages at 400 ms each is almost seven minutes of an idle CPU. Overlap the waits and it is under a minute. But throw threads at a job that is computing and it gets no faster, sometimes slower.

The mental model: a restaurant. Threads and async add waiters — one person serving many tables while the kitchen cooks. Processes add chefs — more hands doing the chopping. Ask one question first: is this job waiting, or computing?

Explained

Most data jobs are waiting: for an API, a database, a disk, S3. The CPU sits idle at 3% while the clock runs.

Waiting in parallel is nearly free. Computing in parallel is not — in Python it needs a different tool entirely, and picking the wrong one makes things slower.

One question decides it. Ask it out loud in the interview.

The restaurant

One waiter, one table at a time (sequential). Take the order, walk to the kitchen, stand there watching the food cook, carry it back, then greet the next table. Ninety percent of the shift is standing still.

One waiter, many tables (async / threads). Take an order, hand it to the kitchen, and while it cooks, go take three more orders. One person, no extra hands — just never standing idle. This is what you want when the kitchen is the slow part.

Many chefs (multiprocessing). If the bottleneck is actually the chopping, no amount of clever waiting helps. You need more people doing the work. More kitchens, more knives, more CPUs.

Threads and async add waiters. Multiprocessing adds chefs. Hiring the wrong one is why people say "I added threads and it got slower".

The decision, as a flowchart

Ninety percent of data-engineering concurrency is the second box. Most pipelines are I/O-bound and a thread pool is the whole answer.

The GIL, accurately

Python's Global Interpreter Lock allows only one thread to execute Python bytecode at a time in a single interpreter.

What that means in practice:

• CPU-bound work in threads gets no speedup. Four threads doing maths share one core's worth of bytecode execution — plus context-switching overhead, so it is often slower
• I/O-bound work in threads scales beautifully. The GIL is released during network and disk waits, so ten threads genuinely wait in parallel
• Processes sidestep it entirely — each has its own interpreter and its own GIL — at the cost of no shared memory and pickling everything you pass
• C extensions release it too — NumPy, Pandas and PyArrow do heavy work outside the lock, which is why vectorised code is fast despite the GIL

(Python 3.13 ships an experimental free-threaded build without the GIL. Mention it as a direction, not as something to depend on.)

Thread pool: the 80% answer

from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_city(city, day):
    return api.get(f"/rides?city={city}&date={day}")   # waits on the network

def pull_all(cities, day, workers=10):
    results, failures = [], []

    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(fetch_city, c, day): c for c in cities}

        for fut in as_completed(futures):
            city = futures[fut]
            try:
                results.append(fut.result())      # re-raises inside the worker
            except Exception as err:
                failures.append((city, repr(err)))   # one city failing ≠ job dead

    return results, failures

Two details that separate this from tutorial code: the {future: city} map so you know which item failed, and catching per-future so one bad city does not kill the run.

Step through 3 workers on 6 cities

Key insight: as_completed gives you results in finish order. If you need input order, use pool.map instead.

Predict which one wins

Two jobs, each on a 4-core machine:

Job A — download 200 files from S3 (each ~300 ms of network wait) Job B — parse and hash 200 already-downloaded files (each ~300 ms of CPU)

You run both with ThreadPoolExecutor(max_workers=8).

Processes and asyncio, briefly

# CPU-bound: one interpreter per core, no shared GIL
from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor() as pool:
    checksums = list(pool.map(hash_file, paths))
# cost: arguments and results are pickled between processes,
# so passing a 2 GB DataFrame per task will eat the gains

# I/O-bound, async-native libraries: thousands of concurrent waits, one thread
import asyncio, httpx

async def fetch_all(urls):
    async with httpx.AsyncClient(timeout=30) as client:
        sem = asyncio.Semaphore(20)              # your own rate limit

        async def one(u):
            async with sem:
                r = await client.get(u)
                return r.json()

        return await asyncio.gather(*(one(u) for u in urls),
                                    return_exceptions=True)

asyncio scales to thousands of concurrent requests where threads would collapse — but every library in the path must be async. One blocking call inside a coroutine freezes the entire event loop.

Words to know

In two minutes

• First diagnose: is the job waiting (I/O-bound) or computing (CPU-bound)? The answer picks the tool.
• The GIL stops two threads running Python bytecode at once, but it is released during I/O waits — so threads scale for waiting, not for computing.
• ThreadPoolExecutor with about 10 workers is the answer for most ingestion; map futures to inputs and catch errors per future.
• CPU-bound work needs ProcessPoolExecutor (pickling costs apply) or a library that works outside the GIL — NumPy, PyArrow, hashlib.
• asyncio handles thousands of concurrent waits on one thread, but a single blocking call freezes the whole event loop.
• Size workers by the real constraint — connection pool, rate limit — not by the core count; beyond that, workers just queue.
• Before concurrency, do less work and batch; concurrency adds failure modes, those two do not.

Test yourself

There are 9 questions, 3 puzzles and 7 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.