🔀 Pipelines & Orchestration · Chapter 3

Airflow: DAGs, tasks, sensors, retries

A DAG that works on a good day is easy. The real Airflow skill is the five knobs that decide what happens on a bad one: sensor mode, timeouts, retries, pools and trigger rules.

Study this lesson free in the app →

The basics

Apache Airflow runs your pipeline as a DAG of tasks; each task is an instance of an operator — a template for one kind of work (run SQL, run a Python function, launch a Spark job, wait for a file). This lesson is the hands-on toolkit; pl-001 covered why tasks must be idempotent and backfill-safe.

The problem it solves: real pipelines wait for partners, call flaky APIs and share limited resources. Without explicit rules, a late file blocks workers for hours, a 503 kills the night's run, and ten DAGs hammer the same rate-limited API at once.

The mental model: each task is a worker on a factory floor with a rulebook:

• Sensor — "wait for the delivery, but don't stand at the door"
• Retries + timeouts — "if it fails, try again 3 times, and never take longer than 20 minutes"
• Pool — "only 4 people may use the forklift at once, across the whole factory"
• XCom — "pass a sticky note to the next worker — a note, not the parcel"
• Trigger rule — "start when all predecessors succeeded (or: when they're all done, whatever happened)"

Explained

Meghalaya Cabs' settlement DAG waits for Razorpay's daily file, calls their API for details and loads the warehouse.

On a bad night the file is 4 hours late, the API returns 503 twice, and three other DAGs want the same API. Whether that night ends in a page or in a clean 07:00 load is decided by a handful of parameters.

The courier desk at an office

An office expects a courier.

The receptionist could stand at the gate for six hours (that's a sensor in poke mode — one person blocked). Smarter: check every 10 minutes and do other work between checks (reschedule). Smartest: ask the guard to ring when it arrives (deferrable).

When the parcel comes, the receptionist doesn't carry the 20 kg box to accounts — she sends a note: "it's in room 4" (XCom passes a reference, not the data).

The office has one franking machine; at most one person uses it at a time, whatever department they're from (a pool).

And if the courier hasn't come by 6 pm, the receptionist stops waiting and tells the manager (a sensor timeout).

Operators and sensors

Action operators do work: BashOperator, PythonOperator / @task, SQLExecuteQueryOperator, SparkSubmitOperator, KubernetesPodOperator. Most come from provider packages (apache-airflow-providers-amazon, …-google).

Sensors wait for a condition: S3KeySensor, SqlSensor, HttpSensor, ExternalTaskSensor, @task.sensor.

Sensor knobs:

• poke_interval — seconds between checks
• timeout — total time to keep waiting; then AirflowSensorTimeout and the sensor fails without retrying
• mode='poke' holds a worker slot the whole time; mode='reschedule' frees it between checks
• deferrable=True (where supported) hands the wait to the triggerer process — no worker slot at all, thousands of waits on one process
• soft_fail=True — mark skipped instead of failed on timeout

Rule: heavy work belongs in Spark/the warehouse; operators should submit and watch.

Retries, timeouts, trigger rules

Retries (per task or in default_args): retries, retry_delay, retry_exponential_backoff, max_retry_delay. Callbacks: on_retry_callback, on_failure_callback.

Timeouts: execution_timeout caps one try of a task; for sensors, timeout caps the whole wait. Use dagrun_timeout on the DAG to cap a whole run.

Fail fast deliberately: raise AirflowFailException for errors a retry can't fix (bad credentials, schema mismatch) — it skips remaining retries. AirflowSkipException marks skipped.

Trigger rules decide when a task may start:

• all_success (default) — every upstream succeeded
• all_done — every upstream finished, any state → cleanup / notify tasks
• one_failed — at least one upstream failed → alert task
• none_failed / none_failed_min_one_success — used after branching (@task.branch), where skipped branches would otherwise skip the join

XCom and pools, precisely

XCom — small values stored in the metadata database (default backend), keyed by dag_id, task_id, run_id and key. A @task return value is pushed automatically; classic operators use ti.xcom_pull(task_ids='extract') or the template {{ ti.xcom_pull(task_ids='extract') }}.

• Pass paths, ids, counts — never DataFrames or result sets
• Need bigger? A custom XCom backend that writes to S3 and stores a reference

Concurrency guardrails:

• Pool — named set of slots shared across all DAGs; a task declares pool= and optionally pool_slots=. default_pool has 128 slots out of the box. priority_weight decides who goes first within a pool
• max_active_runs — runs of one DAG at once (critical for backfills)
• max_active_tasks — tasks of one DAG at once
• max_active_tis_per_dag — instances of one task across runs

Pools protect external resources (an API, a database); DAG-level limits protect Airflow from one DAG.

At scale

Sensors are the silent capacity killer. 300 poke-mode sensors waiting overnight = 300 worker slots doing nothing. Reschedule mode helps; deferrable operators fix it — the triggerer runs thousands of async waits in one process.

Dynamic task mapping (extract.expand(city=[...])) creates one task instance per item at runtime — 40 cities, 40 parallel tasks, each retried independently. Combine with a pool so 40 tasks don't open 40 connections to the replica.

Metadata DB health. Large XComs and millions of task instances bloat it; scheduler loops slow down. Keep XComs tiny and prune old metadata (airflow db clean).

Retries must be classified. At 2,000 tasks a night, a retry on a schema error wastes 35 minutes and delays the alert. Retry transient errors (timeouts, 5xx, throttling); fail fast on permanent ones.

Gotchas

• wait >> load(extract()) — only load depends on the sensor; extract has no upstream and runs immediately, before the file exists
• Poke-mode sensors with long waits — one slot each, all night
• No sensor timeout — the default is 7 days; a missing file is silently waited on for a week
• DataFrames through XCom — serialised into the metadata DB, slows the scheduler for everyone
• Retrying permanent errors — wrong password retried 3× with backoff just delays the page; raise AirflowFailException
• Branch join with default trigger rule — the skipped branch propagates skip to the join; use none_failed_min_one_success
• Jinja in a non-templated argument — renders literally as {{ ds }}; check the operator's template_fields

Wait without holding a slot. Retry only what can heal. Pass notes, not parcels. Pool shared resources.

reschedule/deferrable · execution_timeout + retries · XCom = paths · pool = cross-DAG limit.

Words to know

In two minutes

• Operators are task templates; sensors are operators that wait for a condition.
• Sensors: set poke_interval and a real timeout (default is 7 days); use reschedule mode or deferrable to avoid holding worker slots.
• Retries with exponential backoff heal transient errors; execution_timeout caps each try; AirflowFailException skips retries for permanent errors.
• XCom stores small values in the metadata DB — pass paths and ids, not data.
• Pools cap concurrent use of a shared external resource across all DAGs; max_active_runs/tasks cap one DAG.
• Trigger rules (all_success, all_done, one_failed, none_failed_min_one_success) control joins, cleanup and alerts, especially after branching.
• Wire dependencies explicitly — wait >> load(extract()) leaves extract unguarded.

Test yourself

There are 10 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.