🔀 Pipelines & Orchestration · Chapter 2

Anatomy of a pipeline: extract, land, model, serve

Every data pipeline you will ever build is the same four boxes. Engineers who can name the box a bug lives in fix it in ten minutes; the rest re-run everything and hope.

Study this lesson free in the app →

The basics

A data pipeline moves data from where it's created to where it's used, changing its shape on the way. Almost every pipeline has four stages: extract (pull from the source), land (store a raw copy, untouched), model (clean and reshape into useful tables) and serve (hand it to dashboards, ML models or other apps).

The problem it solves: source systems — the ride-booking Postgres, the payment gateway's API, the GPS event stream — are built to run the business, not to answer questions. Querying them directly for analytics is slow, risky (you can slow down bookings) and inconsistent. A pipeline copies the data out once and reshapes it for questions.

The mental model: a restaurant kitchen. Ingredients are bought (extract), stored in the cold room as delivered (land), prepped and cooked (model), and plated for the table (serve). If a dish tastes wrong, you check each station in order — you don't rebuy the vegetables first.

Explained

Monday 9 am: Meghalaya Cabs' revenue dashboard shows ₹0 for Tura.

Is the source down? Did the extract skip Tura? Did the landing job write an empty file? Did a model filter it out? Is the dashboard cached?

Four stages, four places to look. Knowing the anatomy turns panic into a checklist.

The wedding caterer

A caterer for a 600-guest Khasi wedding doesn't cook at the market.

Extract: buy rice, pork, vegetables from three suppliers — on their schedule, without blocking their shop.

Land: everything goes into the store room exactly as delivered, labelled with the date and supplier. Nobody chops anything yet.

Model: wash, cut, cook — one recipe at a time, with a taster checking each pot.

Serve: plate it the way each table wants it: kids' table, elders' table, vegetarian table.

When the jadoh is too salty, you check the pot, not the market. And because the raw rice is still in the store room, you can cook it again.

Meghalaya Cabs, end to end

An orchestrator (Airflow, Dagster) runs the stages in order; data-quality checks sit between them.

Each stage, precisely

Extract — get data out without hurting the source. Full extract (small tables) or incremental (watermark or CDC). Read from a replica, page through APIs, respect rate limits.

Land — write the extract as received to cheap storage, partitioned by ingestion date, never modified. Schema drift and bad data are visible here, not hidden. This is your replay button.

Model — turn raw into trustworthy tables: cast types, dedupe, standardise ('SHG' and 'Shillong' → one city), join, apply business rules ("a completed ride has a fare > 0"). Layered as staging (1:1 with source, cleaned) → intermediate → marts (facts and dimensions). Usually SQL in the warehouse via dbt.

Serve — deliver in the shape each consumer needs: aggregated tables for BI, point lookups for apps, feature tables for ML, pushes back into operational tools (reverse ETL). Each has its own freshness SLA.

The skeleton in Python

def extract(day):
    # incremental, from a read replica — never the primary
    return replica.query(
        "SELECT * FROM rides WHERE updated_at >= %s AND updated_at < %s",
        [day, day + timedelta(days=1)])

def land(rows, day):
    # raw, untouched, partitioned by ingestion date
    path = f"s3://mc-lake/raw/rides/ingest_date={day}/part-0.json"
    s3.put(path, to_json_lines(rows))   # overwrite same path = idempotent
    return path

def model(day):
    # SQL in the warehouse: staging → marts (usually dbt)
    warehouse.run("dbt build --select +fact_rides")

def serve(day):
    refresh_dashboard("city_revenue")
    publish_features("driver_daily_stats", day)

Each stage takes the day it's processing as input and writes to a deterministic place — so any stage can be re-run alone.

Why the raw copy pays for itself

14 March. A model change maps the city code TUR to null instead of Tura. Nobody notices for 9 days.

Team A transformed in flight and stored only modelled tables. The source Postgres has since overwritten updated_at on many rides; the API only returns 7 days of history. Nine days of Tura revenue is gone for good — they reconstruct it from bank settlements over two weeks.

Team B landed raw first. They fix the mapping and re-run model and serve for 14–23 March from the raw zone. 40 minutes.

Landing raw isn't about storage. It's about being able to say "re-run from the 14th" when — not if — the logic is wrong.

At scale

Each stage scales differently. Extract is bound by the source (API rate limits, replica load) — parallelising it can take down production. Model is bound by warehouse compute. Serve is bound by consumer latency.

Contracts at each boundary. With 50 sources, the extract→land boundary needs schema checks and alerts on drift; the model→serve boundary needs tests so a broken mart never reaches a CFO.

Ownership. At scale, source teams own extract quality (data contracts), the platform team owns land and orchestration, analytics engineers own models, and each consumer owns its serve SLA.

Lineage. You need to answer "which dashboards use raw.rides?" before you change it — tools like dbt docs, OpenLineage or a catalog make that a query instead of a guess.

Gotchas

• Skipping the landing zone — transforming in flight leaves nothing to replay when logic is wrong
• Extracting from the primary DB — a heavy analytics query slows bookings; use a replica or CDC
• Modifying raw data — then it isn't raw; fixes belong in staging
• Using wall-clock time in any stage — breaks re-runs; pass the processing date in
• One giant script doing all four stages — a failure in serve forces a re-extract that hammers the source again
• Serving straight from staging — consumers then depend on source quirks; serve from marts

Words to know

In two minutes

• Every pipeline is extract → land → model → serve, with orchestration and quality checks around it.
• Extract gently: replicas, incremental pulls or CDC, API paging and rate limits.
• Land the data untouched, append-only, partitioned by ingestion date — it's your replay button.
• Model in layers: staging (clean, 1:1 with source) → intermediate → marts (facts and dimensions), with tests between.
• Serve each consumer in its shape: BI aggregates, app lookups, ML features, reverse ETL.
• Make each stage take its processing date as input and write deterministically, so it can be re-run alone.
• When numbers are wrong, walk the stages in order instead of re-running everything.

Test yourself

There are 9 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.