🔀 Pipelines & Orchestration · Chapter 8

Orchestrating failure: retries, alerting, SLAs

Your pipeline will fail. The only design choice is whether it fails loudly at 02:10 to the right person with a runbook — or quietly, discovered by the CFO at the 10 am review.

Study this lesson free in the app →

The basics

Orchestrating failure means deciding, in advance, what a pipeline does when something goes wrong: which errors it retries, which it fails fast on, who gets alerted and how urgently, and what promise — an SLA — it makes about data being fresh and correct by a certain time.

The problem it solves: a pipeline with no failure design either pages someone for every blip (so alerts get ignored) or for nothing (so bad data reaches dashboards). Both end the same way: people stop trusting the data.

The mental model: a hospital emergency ward. Triage first — a cough isn't a heart attack. Minor problems are treated on the spot (retry). Serious ones go straight to a specialist (fail fast + page). Everything has a target time to treatment (SLA). And if a patient didn't arrive who was expected, someone notices (dead man's switch).

Explained

At Meghalaya Cabs, driver_payouts must be correct by 08:00 IST — 6,000 drivers check the app for their money.

One night the bank API times out, a schema changed upstream, and the scheduler restarted. Retries should handle the first, an immediate page the second, and a freshness alarm the third. Most teams only handle the first.

The railway station master

A station master doesn't panic when one train is 3 minutes late — trains recover time. He doesn't wait politely when a signal has failed either; he calls the engineer at once.

He has a board: the 7:40 to Guwahati must leave by 7:55. He doesn't watch each train's engine — he watches the board.

And the most important alarm on his desk: no train has passed in 40 minutes. Nothing failed, nothing broke — something didn't happen. That's the alarm most pipelines don't have.

What happens when a task fails

Every arrow is a decision you should make at design time, not at 02:10.

Retries done right

Only retry what can heal: timeouts, 5xx, HTTP 429 (honour Retry-After), deadlocks, preemptible-node loss. Never retry auth failures, schema mismatches or validation failures — raise AirflowFailException and alert.

Exponential backoff with jitter: 1 min, 2, 4, 8… each randomised. Without jitter, 300 tasks that failed together retry together and knock the recovering API over again (thundering herd).

Cap the total: max_retry_delay and a sane retries so the whole retry schedule fits inside your SLA. Five retries with 30-minute backoff is a 15-hour delay.

Retries require idempotency (partition overwrite, MERGE, idempotency keys on external calls like payouts) — otherwise each retry duplicates.

Retry at the right level: retry the API call inside the task for sub-second blips; retry the task for minute-scale outages; re-run the DAG for multi-hour incidents.

Failure handling in Airflow

TRANSIENT = (requests.Timeout, requests.ConnectionError)

@task(retries=4, retry_delay=timedelta(minutes=2),
      retry_exponential_backoff=True, max_retry_delay=timedelta(minutes=20),
      execution_timeout=timedelta(minutes=30))
def pay_drivers(ds=None):
    batch = load_batch(ds)
    if not batch.validate():                    # bad data: retry won't help
        raise AirflowFailException(f'invalid payout batch {ds}')
    try:
        bank.submit(batch, idempotency_key=f'payout-{ds}')
    except bank.AuthError as e:                 # permanent
        raise AirflowFailException(str(e))
    # TRANSIENT errors propagate → Airflow retries with backoff

@dag(..., default_args={'on_failure_callback': page_if_critical},
     dagrun_timeout=timedelta(hours=4))

The idempotency_key is what makes retrying a payment safe: the bank recognises the second submit and doesn't pay twice.

SLAs for data: what to promise and measure

• SLA — the promise: "driver_payouts is complete and correct by 08:00 IST"
• SLO — the target you track: "met on ≥ 99% of days per quarter"
• SLI — the measurement: time of last successful publish; row count vs source

Data SLIs come in three kinds: freshness (newest data age), completeness (rows vs source / expected), correctness (tests passing, totals reconcile).

Measure at the table, not the task. A green DAG that loaded zero rows is a missed SLA. Check max(loaded_at) and counts directly — dbt source freshness, a final check task, or an observability tool (Elementary, Monte Carlo, Soda).

Airflow's own sla= (2.x) is measured from the run's scheduled time, ignores manual runs, and is notoriously quirky; Airflow 3 removed it in favour of deadline alerts. Most teams measure freshness outside the orchestrator.

Alert routing at Meghalaya Cabs

signal                          severity   route            when
──────────────────────────────────────────────────────────────────
payouts not published by 07:30  P1         page on-call     now
  (SLA 08:00)                                 + #payouts
fact_rides freshness > 3 h      P2         page, business   now
                                           hours only
unique test failed on mart      P2         block publish +  now
                                           Slack owner
marketing report DAG failed     P3         Slack channel    next day
single retry succeeded          —          no alert,        dashboard
                                           count it         only
no DAG run at all since 02:00   P1         dead man's       now
                                           switch → page

Rule: every page must be actionable and link a runbook. If the right response is "ignore it", it shouldn't be an alert.

At scale

Alert fatigue is the real outage. At 3,000 tasks a night, alerting on every failure means 40 Slack messages before breakfast and the one that matters scrolls past. Alert on SLA risk and data impact, not on task failures that retried successfully.

Dead man's switch. Failure callbacks can't fire if the scheduler is down, the DAG is paused or never got scheduled. Have an external check (e.g. a heartbeat ping each run to a monitoring service, or a freshness query from outside Airflow) that alerts when expected success didn't arrive.

Write-audit-publish. Write to a staging table/branch, run tests, then atomically swap/publish (Iceberg branches, Delta, a view swap). Failed audits never reach dashboards — consumers see yesterday's correct data, labelled with its age, instead of today's wrong data.

Error budgets. A 99% SLO allows ~3–4 missed days a year. Burn it fast and you prioritise reliability work over new pipelines — that's how SLOs change behaviour.

Words to know

In two minutes

• Design failure behaviour up front: classify errors, retry transient ones, fail fast on permanent ones.
• Use exponential backoff with jitter, capped so the full retry schedule fits inside the SLA; retries require idempotency.
• SLA is the promise, SLO the target, SLI the measurement — for data: freshness, completeness, correctness.
• Measure the tables, not the tasks: a green DAG with zero rows is still a miss.
• Route alerts by impact with runbooks; avoid alert fatigue by not alerting on self-healed retries.
• Add a dead man's switch outside the orchestrator to catch runs that never happened.
• Write-audit-publish keeps bad data away from consumers; serve last-good data with an honest timestamp.

Test yourself

There are 10 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.