🐍 Python for Data Engineers · Chapter 5

Errors, files and imports

Read the last line of the traceback first. Then read the line above it. That habit alone is worth a month of guessing.

Study this lesson free in the app →

The basics

An exception is Python's way of saying "I can't continue": it stops normal flow and, unless something catches it, prints a traceback and ends the program. Files are how data gets in and out; imports are how one file uses code from another.

The problem it solves: real data is dirty and real environments break. A missing key, a CSV that is actually JSON, a package installed in the wrong place. You need to read the error quickly, handle the cases you expect, and make sure the ones you don't expect are loud, not silent.

The mental model: a smoke alarm. The exception isn't the fire; it's what stops you sleeping through it. except: pass is taking the battery out. Catch the specific alarm you planned for, and if you catch it, fix it or record it.

Explained

Traceback (most recent call last):
  File "pipeline.py", line 42, in <module>
    total = fare_of(event)
  File "pipeline.py", line 18, in fare_of
    return float(event["fare"])
KeyError: 'fare'

Beginners see a wall of text and panic. There are only two facts here:

What went wrong — the last line: no key called fare. Where — the line just above it: pipeline.py, line 18.

Read bottom-up. The stack is printed oldest-first, but the useful part is at the end.

The smoke alarm

An exception is a smoke alarm. It's not the fire — it's the thing that stops you sleeping through the fire.

Catching everything and ignoring it — except: pass — is taking the battery out. The house still burns; you just find out later, from the insurance company.

Catching the specific alarm you planned for — "the toaster sets this off every morning, that one I handle" — is correct.

And the one rule that survives everything: if you catch it, either fix it or record it. Silence is the only genuinely wrong answer.

try / except, done properly

# ✗ the battery-removal version
try:
    fare = float(event["fare"])
except:
    fare = 0                # what failed? nobody will ever know

# ✓ specific, and it leaves evidence
try:
    fare = float(event["fare"])
except KeyError:
    log.warning("event %s has no fare", event["id"])
    return None
except ValueError:
    log.warning("event %s has unparseable fare %r", event["id"], event["fare"])
    return None

# the full shape
try:
    data = json.loads(line)
except json.JSONDecodeError as err:
    rejected.append((line, str(err)))    # keep the evidence
else:
    process(data)                        # runs only if no exception
finally:
    counter += 1                         # runs no matter what

The exceptions worth knowing by name: KeyError (missing dict key), ValueError (right type, wrong value), TypeError (wrong type entirely), FileNotFoundError, ZeroDivisionError, AttributeError (usually a typo, or a None you didn't expect).

Watch control jump

finally is how you guarantee cleanup — closing a connection, releasing a lock — no matter which way the function exits.

Files: always `with`

import json
from pathlib import Path

# `with` closes the file even if the loop raises
with open("rides.jsonl") as fh:
    for line in fh:                    # streams — one line at a time
        event = json.loads(line)

# writing
with open("out.jsonl", "w") as fh:    # "w" overwrites, "a" appends
    for e in events:
        fh.write(json.dumps(e) + "\n")

# whole-file JSON (only when it's small)
config = json.loads(Path("config.json").read_text())

# paths, portably
raw = Path("data") / "raw" / "rides.jsonl"
raw.parent.mkdir(parents=True, exist_ok=True)
raw.exists()

Two habits that prevent most file bugs: use with so files always close, and iterate the handle (for line in fh) rather than .read(), so a 12 GB file doesn't become a 12 GB variable.

Imports: where code comes from

import json                       # the whole module → json.loads()
from pathlib import Path          # one name → Path()
from datetime import datetime as dt   # renamed

from .validate import validate    # a module in the same package

Python finds modules by searching, in order: the current directory, then installed packages. Which explains the two errors everyone hits:

• ModuleNotFoundError: No module named 'pandas' — not installed in the environment you're running. Almost always: you installed it somewhere else, or forgot to activate the venv
• ImportError: cannot import name X — either a typo, or a circular import (a imports b, b imports a). The fix is usually to move the shared thing into a third module

And the trap that eats an afternoon: never name your file after a library. A local json.py or random.py shadows the real one, and the error message points anywhere but at your filename.

Why does this pipeline report success?

total = 0
for line in open("rides.jsonl"):
    try:
        event = json.loads(line)
        total += event["fare"]
    except:
        pass

print(f"total: {total}")

The file has 100,000 rides. It prints total: 0 and exits 0.

Gotchas

• except: pass — the single most destructive line in this lesson. It hides bugs and blocks Ctrl-C
• Catching Exception broadly at the top of a loop — fine only if you log the type and the row
• Not using with — a file left open leaks handles; on a long-running job it eventually fails to open anything
• .read() on a big file — the whole thing lands in memory
• Naming a file json.py / csv.py / random.py — shadows the real library, with a baffling error
• Relying on the current working directory — a script run from elsewhere can't find its data. Build paths from Path(__file__).parent
• Raising a bare Exception — say what kind, so callers can handle it selectively

Words to know

In two minutes

• Read a traceback bottom-up: the last line says what failed, the line above says where.
• Catch specific exceptions (KeyError, ValueError, JSONDecodeError), never a bare except: pass.
• If you catch an error, fix it or record it — keep the bad row and the reason.
• else runs only when no exception occurred; finally runs on every exit, even a return.
• Always open files with with, and iterate the handle line by line instead of .read().
• A job whose output is empty or mostly rejected must fail loudly, not exit 0.
• ModuleNotFoundError usually means the wrong environment; a local file named like a library shadows it.

Test yourself

There are 11 questions, 3 puzzles and 6 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.