🛡 Quality, Testing & Ops · Chapter 7

On-call: triaging a broken pipeline at 3 AM

At 3 AM your job is not to find the root cause. It's to stop bad data reaching people, tell them what's happening, and make the smallest safe change. Root cause is for daylight.

Study this lesson free in the app →

The basics

On-call means you're the person who responds when a pipeline breaks outside working hours. Triage is deciding, quickly, how bad it is and what to do first.

Why it matters: a broken pipeline at 3 AM has a deadline — Meghalaya Cabs' finance report goes out at 07:00, driver payouts at 09:00. A tired engineer making a big, clever change at 3 AM often turns a late report into a wrong one. A good on-call engineer follows a fixed order so tiredness can't skip steps.

The mental model — five verbs, in order:

1. Acknowledge — say you've got it, so nobody else duplicates
2. Assess — who's affected, by when, how badly?
3. Contain — stop bad data spreading: pause downstream, don't publish
4. Mitigate — the smallest safe action that restores service: rerun, rollback, serve yesterday's data with a banner
5. Communicate — short updates on a fixed schedule

Then, in daylight: diagnose properly, fix the root cause, write a blameless postmortem.

Words to know: severity, blast radius, mitigation vs fix, idempotent rerun, rollback, postmortem.

Explained

03:07, Wednesday. Your phone: fct_rides failed — unique test on ride_id, 6,214 failing rows.

Finance's revenue email goes out at 07:00. Driver payouts are computed at 09:00 from the same table.

You have four hours, one tired brain and full write access to production. The order you do things in matters more than how clever you are.

The kitchen fire

A pan catches fire in a restaurant kitchen during dinner service.

The head chef doesn't start by asking why the oil overheated. She:

1. Shouts “fire, I've got it” — so three people don't grab extinguishers. (acknowledge)
2. Checks: one pan, or the hood too? (assess)
3. Puts a lid on it and turns off the burner. (contain)
4. Moves orders to the other stove. (mitigate)
5. Tells the front: “Table 6 mains in 10 minutes.” (communicate)

The next morning, they look at the burner. Investigation happens after the fire is out, not during.

The first 30 minutes

Notice ‘find the root cause’ isn't on the list. Often the mitigation (roll back last night's deploy) works before you know exactly why.

Assess: four questions, two minutes

1. Who consumes it, and by when? Lineage downstream: finance email 07:00, payouts 09:00, ops dashboard continuously.
2. Wrong or late? Late data is annoying. Wrong data that gets published is dangerous — it drives payouts, prices and decisions, and it's hard to recall once people have seen it.
3. How much? One partition or every day since Monday? One city or all?
4. Is it getting worse? An hourly job that will write more bad partitions every hour needs containing now.

That gives you a severity. A reasonable scale:

• SEV1 — money or customer-facing data wrong, or a hard deadline will be missed
• SEV2 — internal reports late, workaround exists
• SEV3 — degraded, can wait until morning

SEV3 at 3 AM? Write it up, set a reminder, go back to sleep. That's a legitimate decision.

What changed? The commands

# 1. Recent deploys — the #1 cause
git -C ~/repos/mc-dbt log --since='18 hours ago' --oneline

# 2. Which runs and tasks failed, and since when
airflow dags list-runs -d fct_rides_daily --state failed
airflow tasks states-for-dag-run fct_rides_daily scheduled__2026-11-04T21:00:00+00:00

-- 3. Shape of the failure: where do the duplicates come from?
SELECT _source_file, _loaded_at, count(*) AS n
FROM stg_rides
WHERE ride_id IN (SELECT ride_id FROM stg_rides GROUP BY 1 HAVING count(*) > 1)
GROUP BY 1, 2 ORDER BY 2;

# 4. After fixing the input, rerun ONLY if the task is idempotent
airflow tasks clear fct_rides_daily -t 'load_stg_rides' \
  -s 2026-11-04 -e 2026-11-04 --downstream --yes

In this incident, query 3 shows the same partner file rides_0300.parquet loaded twice — at 02:58 and 03:01 — after a retry.

03:07–04:20, as it actually went

03:07 Page. Ack in #incidents: “On it — fctrides unique test failing.”_ 03:10 Lineage: feeds finance email 07:00 and payouts 09:00. Wrong data → SEV1. dbt build already skipped downstream, so nothing bad is published yet. Pause the 04:00 payouts DAG anyway. 03:18 What changed: no deploys. Query shows partner file loaded twice after a network retry; the loader appends. 03:25 Update: “SEV1. Duplicate ride rows from a double-loaded partner file. Nothing published. Mitigating. Next update 04:00.” 03:40 Mitigate: delete rows from the second _loaded_at for that file in stg_rides (checked count = 6,214 first, in a transaction), rerun fct_rides and tests. Green. 04:00 Update: “Resolved. Finance email on time. Payouts DAG resumed. Postmortem tomorrow: loader must be idempotent.” 04:20 Back to sleep.

Is it safe to just rerun?

# load_partner_rides task
def load(file):
    df = read_parquet(file)
    warehouse.execute("INSERT INTO stg_rides SELECT * FROM df")

The task failed at 02:58 after the INSERT committed but before it reported success. Airflow marks it failed.

Gotchas

• Debugging before containing. Forty minutes deep in logs while the hourly job writes three more bad partitions.
• Rolling forward at 3 AM. A new ‘quick fix’ PR deployed half-asleep. Roll back to the last known good version instead.
• Blind reruns. Rerunning a non-idempotent append creates the duplicates you're trying to fix.
• Silent heroics. Fixing it alone without updates; finance wakes up to no email and no explanation.
• Not escalating. 30 minutes with no progress on a SEV1 → call the next person. That's the process working, not you failing.
• Deleting without counting. Run the SELECT with the same WHERE first, check the count, then DELETE inside a transaction.
• No postmortem. If nothing changes afterwards, the same page comes next month.

Words to know

In two minutes

• Follow a fixed order: acknowledge, assess impact, contain, mitigate, communicate — then diagnose and fix in daylight.
• Impact first: which consumers, which deadline, wrong data or late data? Wrong is worse than late.
• Contain before you fix: stop downstream jobs and publishing so bad numbers don't spread.
• Ask ‘what changed?’ — deploys, upstream sources, schema, infrastructure, data volume — in that order of likelihood.
• Prefer the smallest reversible action: rerun an idempotent task, roll back a deploy, or publish yesterday's data with a clear note.
• Only rerun tasks you know are idempotent; rerunning an append-only load creates duplicates.
• Communicate on a schedule with impact, status, next update time — even when there's no news.
• Escalate on a time-box, not on pride; and finish with a blameless postmortem that changes something.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.