🐍 Python for Data Engineers · Chapter 14

Packaging: venv, dependencies, Docker

"It works on my machine" is not a joke — it is a description of an unpinned dependency tree.

Study this lesson free in the app →

The basics

Packaging and Docker are how you turn "it works on my laptop" into a job that runs identically on a scheduler every night: pin exactly which software it uses, then ship that software sealed in a container image.

The problem it solves: a pipeline depends on a Python version, a dozen libraries and the libraries they depend on, system packages, timezone data. pip install pandas with no version resolves to whatever was published that morning. One night a rebuild pulls a new pyarrow, an argument disappears, and a job nobody touched crashes at 2 AM.

The mental model: a recipe, a lock file and a shipping container. The recipe (pyproject.toml) says what you want, loosely. The lock file records the exact versions you actually got, down to the last transitive dependency. The container image ships the whole kitchen — interpreter, libraries, OS — so dev, staging and prod run the same thing.

Explained

Your job ran perfectly for four months. Nobody touched the code. This morning it crashes on pd.read_parquet.

What changed? The build reinstalled pandas, which pulled a new minor version, which needed a newer pyarrow, which dropped the argument your code passes.

You did not change anything. The world under you did — because you never pinned it.

The recipe and the shipping container

A recipe says "flour, 200 g". Whose flour? Which mill? It usually works, until one day it doesn't and nobody can reproduce the failure.

A lock file says "flour, 200 g, brand X, batch 2026-07-14". Anyone, anywhere, gets the same result.

A container goes further: it ships the flour, the oven, the altitude and the tap water. Not just the ingredient list — the entire kitchen, sealed. The recipient does not need to own an oven or agree with you about which one is best.

That is why data jobs are shipped as containers: a pipeline depends on a Python version, a dozen libraries, a JVM for Spark, system codecs, timezone data and a locale. Listing those is hope. Shipping them is delivery.

From laptop to scheduled job

The step that fails silently is the third. Without a lock, every later step is running slightly different software from the one before it.

Declared vs locked — the distinction that matters

Declared dependencies are what you import, with the loosest constraint you can live with:

# pyproject.toml
[project]
name = "rideflow-pipeline"
requires-python = ">=3.12"
dependencies = [
  "pandas>=2.2,<3.0",
  "pyarrow>=16",
  "requests>=2.32",
]

Locked dependencies are the exact resolved tree — including things you never named:

pandas==2.2.3
pyarrow==17.0.0
numpy==2.1.1          # you never asked for this; pandas did
python-dateutil==2.9.0.post0
...

Commit both. The first says what you meant; the second guarantees what you got. Upgrading is then a deliberate act — regenerate the lock, run the tests, review the diff — rather than something that happens to you at 2 AM.

Modern toolchain: uv (fast, does venv + resolve + lock), or Poetry, or plain pip-tools. Any of them beats a hand-maintained requirements.txt.

A Dockerfile worth copying

# pin the digest, not just the tag — 'python:3.12-slim' moves
FROM python:3.12-slim@sha256:...

# 1. system deps first — changes rarely, so this layer stays cached
RUN apt-get update && apt-get install -y --no-install-recommends \
      build-essential \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# 2. dependencies BEFORE source — the cache trick that matters
COPY requirements.lock .
RUN pip install --no-cache-dir -r requirements.lock

# 3. source last — changes every commit, only this layer rebuilds
COPY src/ ./src/

# 4. don't run as root
RUN useradd --create-home runner
USER runner

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

ENTRYPOINT ["python", "-m", "src.pipeline"]

Step 2 is the one people get wrong. Copy the source before installing and every one-character code change reinstalls the entire dependency tree — turning a 10-second build into four minutes.

Find the three problems

FROM python:latest
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
ENV DB_PASSWORD=Sup3rSecret!
CMD python pipeline.py

It builds. It runs. It passed review.

Configuration: environment in, nothing hard-coded

import os
from dataclasses import dataclass

@dataclass(frozen=True)
class Config:
    db_url: str
    api_token: str
    batch_size: int = 10_000
    dry_run: bool = False

    @classmethod
    def from_env(cls) -> "Config":
        try:
            return cls(
                db_url=os.environ["DB_URL"],            # required: crash at startup
                api_token=os.environ["API_TOKEN"],
                batch_size=int(os.getenv("BATCH_SIZE", 10_000)),
                dry_run=os.getenv("DRY_RUN", "").lower() == "true",
            )
        except KeyError as missing:
            raise SystemExit(f"missing required env var: {missing}")

Two principles doing all the work:

• Fail at startup, not at 03:47. Reading required config immediately means a missing variable kills the job in the first second, not halfway through writing a partition
• Same image, every environment. Dev, staging and prod run the identical artifact with different environment variables. If the image differs per environment, you are testing something you will never deploy

What production adds

Beyond "it runs in a container", a scheduled job needs five things:

1. Image tags that mean something — rideflow:1.4.2 or the git SHA, never latest. When something breaks at 2 AM you must be able to say exactly what was running, and roll back to a specific thing
2. Structured logs to stdout — JSON lines with run_id, partition, rows, duration. Containers log to stdout; the platform collects it. Never write log files inside a container
3. Resource limits — memory and CPU declared. Without a memory limit, one runaway job takes down a shared node instead of just itself
4. A health signal — exit code 0 only on genuine success (remember the silent partial pull), plus emitted metrics for rows written and duration
5. Small images — slim bases, multi-stage builds, .dockerignore. A 3 GB image pulled onto 50 workers is minutes of pure network before any work starts

And the discipline that ties it together: the artifact you tested is the artifact you deploy. Build once, promote the same digest through environments.

Words to know

In two minutes

• Declare dependencies loosely in pyproject.toml and lock the full resolved tree exactly; commit both.
• Unpinned installs re-resolve on every build, so a job can break with zero code changes when someone else publishes a release.
• In a Dockerfile, order steps from least to most frequently changed: base, system packages, dependencies, then source — that is what keeps the cache useful.
• Pin the base image by version and digest; never latest.
• Secrets are injected at runtime from a secrets manager; a secret in ENV, ARG or a layer is permanent and means rotation.
• Read required config at startup so a missing variable fails in the first second, and run the same image in every environment.
• Tag images with the git SHA, build once and promote the same digest, log JSON to stdout, set memory limits, and don't run as root.

Test yourself

There are 9 questions, 3 puzzles and 7 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.