🛡 Quality, Testing & Ops · Chapter 2

Testing pipelines: unit, integration, contract, data

Code tests prove your logic is right. Data tests prove today's input didn't break it. A pipeline needs both, because the input changes every night and the code doesn't.

Study this lesson free in the app →

The basics

Pipeline testing is checking, automatically, that a pipeline does what you meant — before it ships and every time it runs.

Why it exists: in normal software, if the code is correct and the tests pass, you are mostly done. In data, correct code can still produce wrong numbers because the input changed: the app team renamed fare to fare_amount, a city started sending fares in paise, a partner file arrived empty. So you need tests on the code and tests on the data.

The mental model — four kinds, four questions:

• Unit test — is my transformation logic right? (pure function, fake rows, milliseconds)
• Integration test — do the pieces work together against a real engine? (local Postgres/DuckDB/Spark, seconds to minutes)
• Contract test — does the producer still send what the consumer expects? (schema, types, required fields)
• Data test — is the actual data that just landed sane? (runs in production on every load)

The first three run in CI before merge. The fourth runs in the pipeline after each load, and it is the only one that sees real data.

Words to know: fixture, test pyramid, data contract, schema drift, quarantine.

Explained

Meghalaya Cabs' fare pipeline had 140 unit tests, all green.

On 3 November the driver app started sending fare in paise instead of rupees. Revenue on the dashboard jumped 100× to ₹38 crore for one day. The CFO forwarded the screenshot with one question mark.

Not one unit test was wrong. None of them could have caught it — they test the code, and the code hadn't changed.

The restaurant kitchen

A good kitchen checks four different things:

• The recipe — tasted in the test kitchen with sample ingredients. (unit)
• The line — does the grill hand off to plating without dishes going cold? (integration)
• The supplier agreement — the vegetable supplier promised 5 kg crates of washed potatoes, not 10 kg sacks of muddy ones. (contract)
• Tonight's delivery — open the crate and look before cooking. (data test)

A perfect recipe does not help when tonight's fish smells off. And checking tonight's fish does not help if the recipe has too much salt. You need all four because they fail in different places.

Where each test runs

Two gates. CI protects you from your mistakes. Data tests protect you from everyone else's.

Unit and contract, in pytest

# unit: pure logic, hand-made input, no database
def compute_fare(km, minutes, surge=1.0):
    base, per_km, per_min = 50, 14, 1.5
    return round((base + km * per_km + minutes * per_min) * surge, 2)

def test_surge_applies_to_whole_fare():
    assert compute_fare(10, 20, surge=1.5) == 330.0   # (50+140+30)*1.5

def test_zero_km_is_base_plus_time():
    assert compute_fare(0, 4) == 56.0

# contract: the event the app team promises to send
RIDE_COMPLETED = {
    "ride_id": str, "driver_id": str,
    "fare_inr": float, "city": str, "completed_at": str,
}

def test_sample_event_matches_contract(sample_event):
    for field, typ in RIDE_COMPLETED.items():
        assert field in sample_event, f"missing {field}"
        assert isinstance(sample_event[field], typ), f"{field} wrong type"

The contract test is most useful in the producer's CI — it fails their PR when they rename a field, before it reaches you.

Integration and data tests

# integration: run the REAL SQL against a real engine with seeded rows
import duckdb

def test_daily_revenue_sql_dedupes_retries():
    con = duckdb.connect()
    con.execute("CREATE TABLE rides AS SELECT * FROM 'tests/seed_rides.csv'")
    # seed contains ride R1 twice (a retried event)
    con.execute(open('sql/daily_revenue.sql').read())
    total = con.execute("SELECT revenue FROM daily_revenue WHERE d='2026-11-04'").fetchone()[0]
    assert total == 1240.0      # R1 counted once

-- data test: runs in production after every load; returns failing rows
SELECT ride_date, sum(fare_inr) AS revenue
FROM fact_rides
GROUP BY ride_date
HAVING sum(fare_inr) > 3 * (SELECT avg(daily) FROM revenue_last_28_days);

The paise bug from the hook fails that last query on the first run.

What each kind can and cannot see

               speed    runs on       catches                  blind to
─────────────────────────────────────────────────────────────────────────────
unit           ms       fake rows     wrong formula, edge      real input,
                                      cases, null handling     SQL dialect
integration    sec–min  seeded real   bad joins, dialect,      tomorrow's
                        engine        type casts, dedupe       input
contract       ms       schema/sample renames, type changes,   values that are
                                      missing fields           well-typed but wrong
data           min      production    bad input, drift,        logic bugs on
                        data          volume drops, dupes      paths data hasn't hit

Every row has a blind spot that another row covers. That is the argument for all four.

Thursday, 4:05 PM — the PR that looked fine

Asha changes the revenue model from JOIN payments to LEFT JOIN payments to include unpaid rides.

• Unit tests — pass. They test the fare function, not the SQL.
• Integration test — fails: the seeded data has a ride with two payment attempts (one failed, one captured). The LEFT JOIN now doubles that ride. Expected ₹1,240, got ₹1,560.

She adds WHERE payments.state = 'captured' OR payments.ride_id IS NULL — careful to keep unpaid rides — and the test goes green.

Without the integration test, this would have shipped and been found a week later by a finance analyst. Join fan-out is the bug integration tests exist for.

Gotchas

• Mocking the database in “integration” tests. If the SQL never runs on a real engine, you've written a unit test with extra steps.
• Data tests that only warn. A warning in a log nobody reads is no test. Decide: block, quarantine, or page.
• Testing on a copy of production. Slow, flaky, and leaks PII into CI. Use small seeded fixtures that contain the nasty cases on purpose: duplicates, nulls, midnight rides.
• Contract only on your side. You find out after the producer shipped. Put the check in their CI, or use a schema registry that rejects incompatible changes.
• 100% coverage as the goal. Coverage counts lines, not cases. One test with a duplicate payment beats twenty happy-path tests.

Words to know

In two minutes

• Code changes when you merge; data changes every run — so pipelines need tests on both.
• Unit tests check pure transformation logic on tiny hand-made inputs; they are fast and should be the majority.
• Integration tests run the real SQL or Spark job against a real (local) engine to catch dialect, join and type issues.
• Contract tests pin the agreement between producer and consumer: field names, types, required fields, allowed values.
• Data tests run inside the production pipeline on real data: nulls, uniques, ranges, row counts, reconciliations.
• Unit, integration and contract tests gate the merge in CI; data tests gate publishing to consumers.
• When a data test fails, stop or quarantine — never silently publish to the dashboard.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.