🔀 Pipelines & Orchestration · Chapter 4

Scheduling, dependencies and the backfill problem

The run stamped 14 March starts on 15 March. Until that sentence feels obvious, every Airflow schedule, cross-DAG sensor and backfill you write will be off by one.

Study this lesson free in the app →

The basics

Scheduling in Airflow means deciding which slice of time each run is responsible for, and when it is allowed to start. Dependencies decide what must finish first — inside a DAG and across DAGs. The backfill problem is re-running many past slices without corrupting data or crushing your systems.

The problem it solves: pipelines process periods — yesterday's rides, the last hour's payments. A run can only process a period after it has ended, and re-runs must hit exactly the same period. Get this wrong and you double-count, miss days, or wait forever for an upstream run that will never match.

The mental model: each run is a labelled envelope. The label says "rides for 14 March" (the data interval — 14 Mar 00:00 to 15 Mar 00:00). You can only fill the envelope once 14 March is over, so it is opened on 15 March. The label never changes, no matter when you re-open it.

Explained

Meghalaya Cabs' daily_revenue DAG runs at 00:00. The run that starts at 00:00 on 15 March has ds = 2026-03-14.

Not a bug. It's the most important convention in Airflow — and the source of most of its off-by-one incidents.

The newspaper's date

The newspaper delivered on the morning of the 15th reports what happened on the 14th. The editors couldn't print it on the 14th — the day wasn't over.

If the printing press breaks and the paper is reprinted on the 17th, it still covers the 14th. Nobody prints the 17th's news on the 14th's edition.

And the sports supplement can't go out until the main edition is printed — but if the main edition is printed at midnight and the supplement checks at 2 am for an edition labelled 2 am, it waits forever. Labels have to match, not clocks.

Intervals, labels and start times

schedule='@daily', start_date=2026-03-01 (Asia/Kolkata)

data interval              ds / logical_date   run starts at
─────────────────────────────────────────────────────────────
[03-01 00:00, 03-02 00:00)   2026-03-01        03-02 00:00
[03-02 00:00, 03-03 00:00)   2026-03-02        03-03 00:00
[03-03 00:00, 03-04 00:00)   2026-03-03        03-04 00:00
[03-04 00:00, 03-05 00:00)   2026-03-04        03-05 00:00

Deployed 03-05 10:00:
  catchup=True  → runs for 03-01, 03-02, 03-03, 03-04 queued now
  catchup=False → only 03-04 (latest finished interval), then onward

The first run happens one full interval after start_date. Your task should read the window from data_interval_start / data_interval_end, never compute it.

Precisely (Airflow 2.x)

• Data interval [start, end) — what the run covers. Templates: {{ data_interval_start }}, {{ data_interval_end }}, {{ ds }}, {{ ds_nodash }}
• Logical date — the run's identity (formerly execution_date, deprecated since 2.2). For cron/timedelta schedules it equals data_interval_start
• Run after — a run is created when its interval ends
• Timezones — Airflow stores UTC. Give start_date a timezone (pendulum.datetime(2026, 3, 1, tz='Asia/Kolkata')) or @daily means midnight UTC = 05:30 IST
• Timetables — cron presets, timedelta, CronTriggerTimetable (runs at the cron time; logical date = trigger time, no preceding interval), custom timetables (e.g. business days)
• catchup — True by default in Airflow 2 (config catchup_by_default); Airflow 3 flips the default to False
• depends_on_past=True — a task waits for its own previous run to succeed; makes backfills sequential and one failure blocks all later days
• max_active_runs — how many intervals run in parallel; your main backfill throttle

Three ways to depend on another DAG

Sensor = loose coupling but schedules must line up. Trigger = producer knows its consumers. Dataset = neither knows the other's schedule (renamed assets in Airflow 3).

Cross-DAG dependencies in code

# 1. Pull: wait for upstream task with the *same logical date*
wait = ExternalTaskSensor(
    task_id='wait_rides_ingest',
    external_dag_id='rides_ingest', external_task_id='publish_silver',
    # upstream runs at 00:00, we run at 02:00 → shift to match ITS label
    execution_delta=timedelta(hours=2),
    mode='reschedule', timeout=4 * 3600)

# 2. Data-aware: producer declares what it updates
SILVER_RIDES = Dataset('s3://mc-lake/silver/rides')

@task(outlets=[SILVER_RIDES])
def publish_silver(): ...

# consumer DAG runs whenever that task succeeds
@dag(schedule=[SILVER_RIDES], start_date=pendulum.datetime(2026, 3, 1, tz='Asia/Kolkata'))
def daily_revenue(): ...

execution_delta is subtracted from this run's logical date to find the upstream one. Forget it when schedules differ and the sensor waits for a run that doesn't exist.

The six-month backfill

Task: a bug in fare rounding affected fact_rides from 1 Sep to 28 Feb — 181 days. Downstream: daily_revenue, driver_payouts, and an ML feature table.

What a careful engineer does:

1. Confirms every affected task is idempotent per interval (overwrite partition / MERGE)
2. Runs airflow dags backfill -s 2025-09-01 -e 2026-02-28 rides_model with max_active_runs at 4 so the warehouse and replica survive
3. Checks for cumulative tables — driver_lifetime_earnings built as yesterday + today must be rebuilt in order, or rebuilt from scratch
4. Cascades downstream: clears dependent DAGs for the same range (or ExternalTaskMarker clears them with --include-downstream)
5. Verifies counts and ₹ totals per day before and after, against the payment gateway

At scale: the backfill problem

Throughput vs blast radius. 181 days × 30 tasks = 5,430 task instances. Unthrottled, they fill every worker slot and starve tonight's production runs. Use max_active_runs, a dedicated pool, or run backfills on a separate queue.

Order matters only for state. Partition-overwrite tasks can run in any order, in parallel. Anything cumulative (running balances, SCD2 dimensions, depends_on_past) must go sequentially — or be redesigned to recompute from raw.

Late-arriving data is a permanent mini-backfill. If 2% of rides arrive up to 3 days late, design each run to reprocess a trailing window (the last 3 intervals) — or use a lookback in incremental models.

Backfill the cheapest layer. If raw is correct, re-run model and serve only. Re-extracting six months from a partner API may be rate-limited or impossible.

Words to know

In two minutes

• Every run covers a data interval [start, end); for normal schedules the logical date (ds) is the interval start.
• A run is created when its interval ends — the run labelled 14 March starts at 00:00 on 15 March.
• Tasks must read data_interval_start/end from the context so re-runs hit the same window.
• Set a timezone on start_date, otherwise @daily means midnight UTC (05:30 IST).
• catchup=True (Airflow 2 default) creates every missed interval; False creates only the latest finished one.
• Cross-DAG: ExternalTaskSensor (matching logical dates, execution_delta for offsets), TriggerDagRunOperator, or Datasets (data-aware).
• Backfills need idempotent tasks, throttling (max_active_runs, pools), sequential order for cumulative state, and downstream cascade.

Test yourself

There are 10 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.