🔀 Pipelines & Orchestration · Chapter 1

Airflow: DAGs, tasks and the backfill that must be safe

An orchestrator's job is not running things. It is running things again — after a failure, out of order, six months late — and getting the same answer.

Study this lesson free in the app →

The basics

Apache Airflow is an orchestrator: it runs your pipeline's steps in the right order, on a schedule, retries what fails, and shows you what happened. You describe the pipeline in Python as a DAG.

The problem it solves: a cron entry can start a script at 02:00. It can't say "load only after extract succeeded", retry a 503 three times, or re-run 13 March next week and get exactly the same answer. The hard part of scheduling isn't running things — it's running them again, safely.

The mental model: every run is stamped with the day it is for, not the day it happens to execute. A task asks "which day am I processing?", never "what time is it?". Combine that with writes that replace one partition, and re-runs become harmless.

Airflow orchestrates; heavy data processing belongs in Spark, the warehouse or another engine it triggers.

Explained

Anyone can write a script and a cron entry.

The question that separates a data engineer from someone with a cron entry is: what happens when it fails at 3am and you re-run it at 9? Twice the rows? Missing day? Or exactly right?

A DAG is a directed acyclic graph: tasks, and which must finish before which. Airflow's job is to work out what can run now, run it, retry what fails, and tell you.

The critical idea is the run date. Every run is stamped with the logical interval it is for — not the wall clock it happens to execute on. A task should ask "which day am I processing?" and never "what time is it?"

That single discipline is what makes a backfill possible.

The pieces

DAG — the pipeline definition. Python, evaluated on a schedule by the scheduler.

Task — one unit of work. An operator instance.

Logical date / data interval — the window this run is responsible for. Passed in as {{ ds }} or data_interval_start.

Sensor — a task that waits for something (a file, a partition, another DAG). Use reschedule mode, not poke, or it holds a worker slot for hours.

Pool / concurrency — the guardrail that stops your DAG taking every worker and starving everything else.

XCom — small values passed between tasks. Small. It goes through the metadata database, so it is not for data.

The shape that survives a re-run

@dag(schedule='@daily', start_date=datetime(2026, 1, 1), catchup=True,
     default_args={'retries': 3, 'retry_delay': timedelta(minutes=5)})
def rides_daily():

    @task
    def load(data_interval_start=None):
        day = data_interval_start.date()          # NOT date.today()
        rows = extract(day)
        # delete-then-insert for THIS partition only — the whole trick
        warehouse.execute('DELETE FROM rides WHERE ride_date = %s', [day])
        warehouse.insert('rides', rows)
        return len(rows)

    load()

Re-run it a hundred times for the same day and the table is identical every time. That property has a name — idempotency — and without it a backfill is a data corruption event with a progress bar.

14 March, 02:10. The daily load fails — the source API returned 503.

Airflow retries at 02:15, 02:20, 02:25. All fail. The API is down for maintenance until 06:00.

At 09:00 you clear the task. Airflow re-runs it with the same logical date — 13 March, not 14 March. It deletes 13 March's partition, reloads it, and the numbers are right.

Now imagine the task had used date.today(). At 09:00 it would have loaded 14 March's data into a run meant for the 13th, left the 13th empty, and nobody would notice for a week.

At 4,000 DAGs the failure modes change.

The scheduler becomes the bottleneck before the workers do. DAG files are parsed repeatedly — top-level code runs on every parse, so an API call at the top of a DAG file is an API call every thirty seconds, forever.

catchup=True on a DAG with a 2019 start date queues six years of runs the moment you deploy it. This takes production down and it happens regularly.

Sensors in poke mode each hold a worker slot. A hundred waiting sensors is a hundred workers doing nothing.

At that size people move the heavy work off the orchestrator entirely — Airflow submits a Spark job and watches, rather than processing anything itself.

date.today() anywhere in a task. The single most common Airflow bug. It makes backfills silently wrong rather than loudly broken.

Top-level code in a DAG file. It runs on every parse. Database connections and API calls belong inside tasks.

catchup=True without thinking. Check the start date before you deploy.

Passing DataFrames through XCom. It serialises into the metadata DB. Pass a path, not the data.

Retries on a non-idempotent task. You have automated the duplication.

An orchestrator's real job is the re-run.

Every task takes its date from the run, never from the clock. Every write is delete-then-insert for that partition only.

If you cannot re-run yesterday safely, you do not have a pipeline. You have a script that worked once.

Words to know

In two minutes

• Airflow schedules and coordinates tasks defined as a DAG; it is not where heavy data processing should happen.
• Each run owns a logical data interval; tasks must take their date from it, never from date.today().
• A daily run for a given day executes after that day ends.
• Idempotent tasks — delete-then-insert one partition — make retries and backfills safe.
• Retries on a non-idempotent task automate duplication.
• Top-level DAG-file code runs on every scheduler parse; keep connections and API calls inside tasks.
• Check catchup and start_date before deploy; cap backfills with max_active_runs and verify counts per day.
• Sensors should use reschedule mode; XCom carries small values like paths, not DataFrames.

Test yourself

There are 9 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.