🛡 Quality, Testing & Ops · Chapter 6

Observability: lineage, logs, metrics, sane alerts

Monitoring tells you something is wrong. Observability lets you ask why — without shipping new code first. At 3 AM, that difference is the whole night.

Study this lesson free in the app →

The basics

Data observability is being able to answer “what is happening in my pipelines and why?” from the signals they already emit — without adding print statements and rerunning.

Why it exists: a modern data platform has hundreds of jobs and thousands of tables. When the revenue dashboard is wrong, the cause might be six hops upstream in a job someone else owns. Monitoring (dq-004) tells you a number is off. Observability tells you which upstream table changed, which run produced it, what it logged, and who else is affected.

The mental model — four signals:

• Metrics — numbers over time: run duration, rows in/out, lag, cost
• Logs — what happened in a run, as structured events with a run_id
• Lineage — the graph of which job reads which table and writes which table (sometimes down to columns)
• Alerts — the small set of conditions that deserve a human, routed to the owner

The first three are for investigating. The fourth is for interrupting — and should be rare.

Words to know: lineage graph, upstream / downstream, structured log, SLO, alert fatigue, runbook.

Explained

02:40. Forty-seven alerts arrive in four minutes: fct_rides stale, driver_payouts failed, city_dashboard stale, surge_features failed, finance_daily failed…

The on-call engineer opens them one by one. At 03:25 she finds the cause: one upstream source, app_events.ride_completed, stopped loading at 02:31.

With lineage, those 47 alerts would have been one incident with 46 downstream effects — and she'd have started at the root.

The hospital patient

A patient in the ICU isn't checked by a nurse asking “are you okay?” every hour.

• Metrics — heart rate, BP, oxygen on a screen, trending over time.
• Logs — the chart: what medicine, when, by whom.
• Lineage — the referral history: which doctor sent them, which tests fed which diagnosis.
• Alerts — the monitor beeps only for a few things: oxygen below 90, heart stopped. Not for every flicker.

A monitor that beeps for everything gets turned down by tired nurses. Then it misses the one that mattered. Alert design is a safety decision, not a technical detail.

A lineage graph — both directions matter

  app_events.ride_completed ──┐
                              ├─► stg_rides ─► fct_rides ─┬─► finance_daily
  gateway.payments ───────────┘                           ├─► driver_payouts
                                                          ├─► city_dashboard
  dim_drivers ────────────────────────────────────────────┘   └─► surge_features

  ◄──── walk UPSTREAM for root cause
        (finance_daily is wrong → what fed it?)

        walk DOWNSTREAM for blast radius ────►
        (I'm changing fct_rides.fare_inr → who breaks?)

Column-level lineage goes further: finance_daily.revenue comes from fct_rides.fare_inr, which comes from ride_completed.fare. Now a rename has a precise list of affected dashboards.

Logs and metrics a run should emit

import json, logging, time, uuid

log = logging.getLogger("pipeline")

def emit(event, **fields):
    log.info(json.dumps({"event": event, **fields}))

def run_fct_rides(partition):
    run_id, t0 = str(uuid.uuid4()), time.time()
    ctx = {"run_id": run_id, "job": "fct_rides", "partition": partition}
    emit("run_started", **ctx, inputs=["stg_rides", "dim_drivers"])

    rows_in = read_count("stg_rides", partition)
    good, rejected = transform(partition)
    write("fct_rides", good, partition)

    emit("run_finished", **ctx, status="success",
         rows_in=rows_in, rows_out=len(good), rows_rejected=len(rejected),
         duration_s=round(time.time() - t0, 1),
         outputs=["fct_rides"])

One line per event, JSON, with run_id everywhere. Now “show me every run of fctrides last week where rows_rejected > 0”_ is a query, not an afternoon of grep. The inputs / outputs fields are lineage for free — standards like OpenLineage formalise exactly this, and Airflow, dbt and Spark can emit it.

Metrics that catch problems early

Per job run:

• duration — creeping from 12 to 47 minutes over a month means a full scan or skew is coming
• rows_in vs rows_out — a join that suddenly drops 8% of rows
• rows_rejected — quarantine growth
• lag — event time to availability
• cost — warehouse credits or cluster-hours per run

Per dataset: freshness, volume, test pass rate.

Plot each as a time series. The incidents you prevent are the ones where a line was slowly bending for two weeks and somebody looked. Use p95 duration, not average — a job that's usually 10 minutes and sometimes 90 is a problem the average hides.

Is this alert worth a human?

Symptom-based alerting: page on what users experience (finance_daily not ready by 06:30), not on every internal cause (a retry happened). Causes go on dashboards for investigation.

The rename that almost shipped

Rohan opens a PR renaming fct_rides.fare_inr to fare_amount_inr — cleaner naming.

The CI step queries column lineage and comments on the PR:

fct_rides.fare_inr is read by: finance_daily.revenue, driver_payouts.base_earnings, surge_features.avg_fare_1h, and 3 Metabase dashboards (owner: finance-analytics).

Rohan keeps the old column as an alias for two weeks, announces the change in the data channel, and migrates consumers first. Lineage turned a Monday incident into a Thursday Slack message.

Gotchas

• Alerting on every failure. A task that retries and succeeds doesn't need a human. Alert on the outcome.
• One alert per downstream table. A single upstream outage becomes a storm. Group by lineage root.
• Unstructured logs. print("done!") can't be filtered by run or partition. JSON with run_id or it didn't happen.
• Lineage drawn by hand in a wiki. It's wrong within a month. Lineage must be emitted by the tools or parsed from SQL.
• Alerts without owners. Sent to #data-alerts with 40 members means owned by nobody.
• No runbook. The 3 AM responder shouldn't have to reverse-engineer what this alert means.

Words to know

In two minutes

• Monitoring answers ‘is it broken?’; observability answers ‘why, where, and who's affected?’.
• Metrics per run — duration, rows in and out, rejected rows, lag — make slow drifts visible before they become outages.
• Structured logs with run_id, dataset and partition let you filter one failing run out of millions of lines.
• Lineage gives two superpowers: root cause (walk upstream) and impact analysis (walk downstream before changing anything).
• Alert on symptoms consumers feel — stale or wrong data against an SLO — not on every internal cause.
• Every alert needs severity, an owner, deduplication and a runbook link; otherwise it trains people to ignore alerts.
• Use lineage to group a cascade of failures into one incident at the root.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.