🐍 Python for Data Engineers · Chapter 6

Idiomatic Python for pipelines

The Python that passes code review is a small dialect. Learn the dozen idioms that make up 90% of pipeline code.

Study this lesson free in the app →

The basics

Idiomatic Python is the small set of built-in patterns experienced engineers reach for by reflex — comprehensions, enumerate, zip, dict.get, defaultdict, Counter, dataclasses, pathlib.

The problem it solves: the same transform can be 40 lines of counters, index arithmetic and nested ifs, or 12 lines that read like a sentence. The long version is not just ugly; it hides bugs (off-by-one, KeyError three stages later, a list used where a set was needed) and it fails code review.

The mental model: a cook's knife roll. Each idiom is a tool with one job. You do not invent a new way to group by key; you pick up defaultdict(list). Choosing the right container is half the work: dict for lookup, set for membership, list for order, Counter for frequency.

Explained

Two engineers write the same transform. One writes 40 lines of loops and index arithmetic. The other writes 12 lines that read like a sentence.

Both work. Only one gets merged without three rounds of comments — and only one is still readable when it breaks at 3 AM.

The kitchen, again

A cook who has worked one kitchen for ten years does not think about where the knives are. Their hands know.

Idioms are that. enumerate instead of a counter, a comprehension instead of an append loop, a dataclass instead of a bare dict — none of them are clever. They are the things you stop having to think about, so the thinking is left for the actual problem.

Loops → comprehensions

# the version everyone writes first
fares = []
for e in events:
    if e["event_type"] == "ride_completed":
        fares.append(e["fare"])

# the version that gets merged
fares = [e["fare"] for e in events if e["event_type"] == "ride_completed"]

# dict comprehension — index events by id
by_id = {e["event_id"]: e for e in events}

# set comprehension — distinct drivers, deduped for free
drivers = {e["driver_id"] for e in events}

Rule of thumb: one condition and one transform belongs in a comprehension. Two conditions and a nested loop belongs in a function with a name.

The six built-ins you will use daily

# enumerate — index without a counter variable
for line_no, line in enumerate(file, start=1):
    ...

# zip — walk two sequences together
for city, revenue in zip(cities, revenues):
    ...

# any / all — short-circuit validation
if any(e["fare"] < 0 for e in events):
    raise ValueError("negative fare in batch")

# sorted with a key — sort by a field, newest first
latest = sorted(events, key=lambda e: e["event_time"], reverse=True)

# max with a key and a default — safe on an empty batch
newest = max(events, key=lambda e: e["event_time"], default=None)

# sum with a generator — no intermediate list
total = sum(e["fare"] for e in events if e["event_type"] == "ride_completed")

Dicts: the four methods that remove most bugs

from collections import defaultdict, Counter

# .get with a default — never a KeyError
fare = event.get("fare", 0)

# defaultdict — group without checking whether the key exists
by_city = defaultdict(list)
for e in events:
    by_city[e["city"]].append(e)

# Counter — frequency in one line, with .most_common()
top_reasons = Counter(r.reason for r in rejections).most_common(5)

# dict unpacking — merge, right side wins
enriched = {**event, "ingested_at": now, "source": "kafka"}

defaultdict(list) and Counter between them replace an enormous amount of hand-written pipeline code. Counter(...).most_common(5) is also the fastest way to answer "what are the top 5 failure reasons?" in an interview.

Dataclasses: stop passing dicts around

from dataclasses import dataclass, field
from datetime import datetime
from decimal import Decimal

@dataclass(frozen=True, slots=True)
class RideEvent:
    event_id: str
    ride_id: str
    event_type: str
    event_time: datetime
    fare: Decimal | None = None
    tags: list[str] = field(default_factory=list)   # never a bare []

    @property
    def is_billable(self) -> bool:
        return self.event_type == "ride_completed" and self.fare is not None

Why it beats a dict:

• e.ride_i is an AttributeError now; e["ride_i"] is a KeyError three stages later
• frozen=True makes it hashable and immutable — an event is a fact, facts do not change
• slots=True cuts memory noticeably when you hold millions of them
• the class is the documentation of your schema

Type hints earn their keep at boundaries

Python does not enforce them at runtime — mypy or your editor does. That is still worth it, because the hint is a contract a reader can trust.

from collections.abc import Iterable, Iterator

def parse_events(lines: Iterable[str]) -> Iterator[RideEvent]:
    ...

def daily_revenue(events: Iterable[RideEvent]) -> dict[str, Decimal]:
    ...

Two rules that keep typing useful instead of exhausting:

1. Type the function signatures, not every local variable. The signature is the contract; locals are obvious from context
2. Accept the widest type, return the narrowest. Take Iterable, return list or Iterator — so callers can pass a generator, a list or a file

Paths, files and f-strings

from pathlib import Path

# pathlib beats os.path string-joining
raw = Path("data/raw") / "rides" / f"dt={run_date}" / "events.jsonl"
raw.parent.mkdir(parents=True, exist_ok=True)

for path in Path("data/raw").rglob("*.jsonl"):
    ...

# context managers — the file closes even if the loop raises
with raw.open() as fh:
    for line in fh:                       # streams, does not load the file
        ...

# f-strings, including the = debug form and number formatting
print(f"{len(events)=}")                 # len(events)=1043
print(f"loaded {count:,} rows in {secs:.2f}s")   # loaded 1,043,201 rows in 3.14s

Words to know

In two minutes

• Idiomatic Python is a small dialect: comprehensions, enumerate, zip, any/all, sorted/max with key, dict.get, defaultdict, Counter, dataclasses, pathlib, f-strings.
• One condition and one transform go in a comprehension; anything more deserves a named function.
• Pick the container by the question: dict for lookup, set for membership, list for order, Counter for frequency.
• x in list is O(n); inside a loop over a million items that is the classic accidental quadratic. Membership checks go against a set.
• Use frozen dataclasses at boundaries: typos fail immediately, the class documents the schema, slots save memory at volume.
• Type hints document contracts but are not enforced at runtime; type signatures, accept wide types, return narrow ones.
• Review-killer bugs: mutable default arguments, late-binding lambdas, mutating a list while iterating, bare except, float ==.

Test yourself

There are 9 questions, 3 puzzles and 6 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.