🎯 Interview & Career · Chapter 3

The Python round: what they're really testing

A data engineering Python round is rarely about clever algorithms. It's a messy file, a dictionary and a question: will your code survive the 10 millionth line, the malformed one, and the next engineer who reads it?

Study this lesson free in the app →

The basics

The Python round for data engineers usually gives you records — log lines, JSON events, a CSV — and asks you to clean, deduplicate, aggregate or join them, live, in a shared editor. Sometimes pandas or PySpark is allowed; often it's plain Python on purpose.

What they're really testing:

• Data structures — do you reach for a dict for lookups, a set for dedupe, a heap for top-k?
• Robustness — what happens with an empty file, a malformed line, a missing key?
• Scale awareness — does it work on 50 GB, or only on the 5-line sample?
• Readability — small named functions, clear variables, a test or two
• Communication — same as SQL: say what you're doing and why

The mental model: treat it as a mini pipeline — parse → validate → transform → aggregate → output — with each step a function you can test.

Words to know: O(n) / O(n log k), generator, defaultdict, heapq, idempotent, malformed record.

Explained

Interviewer: “Here's a file of ride events, one JSON per line. Return each driver's total completed fare. Events can be duplicated.”

The 5-line sample works with any approach. Then the follow-ups: “The file is 40 GB. … Line 3 million is truncated. … Now give me the top 5 drivers.”

Each follow-up tests one thing: streaming, robustness, data-structure choice. The first answer was just the warm-up.

Sorting the post office

A post office in Shillong gets 50,000 letters a day.

The new clerk tries to hold them all in his arms, then sort. (reading the whole file into memory) Then he checks each letter against a handwritten list of already-delivered ones, top to bottom. (in list, O(n) per letter) A torn envelope with no address stops him for an hour. (one malformed line crashes everything)

The experienced clerk takes letters one at a time from the bag, drops each into a labelled pigeonhole (dict), checks a stamped register for duplicates (set), and puts torn ones in a ‘problem’ tray to count later.

Same job. One clerk scales, one doesn't. That's the difference they're interviewing for.

Every DE Python answer has this shape

Say this shape out loud before coding: “I'll read, parse, validate, dedupe, aggregate, output — each as its own small function.” It shows decomposition in one sentence and gives you a map for the rest of the round.

A solution that survives the follow-ups

import json, heapq, logging
from collections import defaultdict

def read_events(path):
    """Stream events; skip and count malformed lines."""
    bad = 0
    with open(path) as f:
        for line in f:                       # one line in memory at a time
            try:
                yield json.loads(line)
            except json.JSONDecodeError:
                bad += 1
    logging.info("skipped %d malformed lines", bad)

def driver_totals(events):
    seen, totals = set(), defaultdict(float)
    for e in events:
        eid = e.get("event_id")
        if eid is None or eid in seen:       # O(1) membership
            continue
        seen.add(eid)
        if e.get("type") == "completed" and e.get("fare") is not None:
            totals[e["driver"]] += float(e["fare"])
    return totals

def top_drivers(totals, k=5):
    return heapq.nlargest(k, totals.items(), key=lambda kv: kv[1])  # O(n log k)

totals = driver_totals(read_events("rides.jsonl"))
print(top_drivers(totals))

Three functions, each one sentence to explain. Say out loud: “the seen set grows with distinct events — at billions I'd dedupe per day or use the warehouse.”

The scoring sheet, translated

• Problem solving — clarified input format, size, duplicates, what to do with bad rows?
• Data structures & complexity — right structure, complexity stated without being asked?
• Correctness & edge cases — empty input, missing keys, duplicates, None fares?
• Code quality — functions with one job, clear names, no 60-line blob, no global state.
• Testing — wrote or described tests; checked the sample by hand.
• Communication — explained trade-offs: “set gives O(1) dedupe but holds every id in memory.”

For data roles, add scale judgement: knowing when to say “at this size I wouldn't do it in Python — I'd load it and GROUP BY in the warehouse or use PySpark.” That sentence is often worth more than a micro-optimisation.

How candidates lose the Python round

• f.read() or readlines() on a huge file — loads everything into memory.
• if eid in seen_list — O(n) per check, O(n²) overall; 10 M events never finishes.
• Bare except: — hides real bugs (typos, KeyboardInterrupt). Catch the specific error.
• Silently dropping bad rows — skip them, but count and log them.
• Sorting everything for top-k — O(n log n) when O(n log k) is available; fine to mention both.
• Mutable default arguments — def f(acc=[]) shares one list across calls.
• Not running the sample by hand — the cheapest bug check there is.
• Fighting the interviewer's constraint — if they say “no pandas”, they want to see the structures.

The follow-up ladder

Interviewers rarely stop at the first solution. Expect this ladder and have a sentence ready for each rung:

1. “It's 40 GB.” → stream lines with a generator; memory now depends on distinct drivers, not file size.
2. “Some lines are broken.” → try/except JSONDecodeError, count, log, continue; fail the job if bad > 1%.
3. “Top 5 drivers.” → heapq.nlargest — O(n log k).
4. “Duplicates across days.” → the seen-set grows forever; dedupe per day partition, or push dedupe to the warehouse with a unique key.
5. “How would you test it?” → three tests: empty file, one malformed line, a duplicated completed event.
6. “Make it run daily in production.” → idempotent output per date, overwrite the partition, schedule in Airflow, alert on bad-line rate.

Stream it, structure it, survive bad lines, state the complexity, test the edges.

dict for lookup, set for dedupe, heap for top-k — and know when to say ‘this belongs in the warehouse’.

Words to know

In two minutes

• Most DE Python rounds are data manipulation: parse, clean, dedupe, aggregate, join — not puzzles.
• Pick the structure for the access pattern: dict for lookup, set for membership, heap for top-k, Counter for counts.
• Ask about input size early; for files bigger than memory, stream line by line with generators.
• Handle bad input deliberately: skip-and-count malformed records instead of crashing or silently ignoring.
• Structure the solution as small functions — parse, validate, aggregate — so each is testable and explainable.
• State time and space complexity unprompted; know why x in list inside a loop is O(n²).
• Write or describe two or three tests, including the empty and malformed cases.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.