◐ ai-teacher

🔀 Free course

Pipelines & Orchestration

Build a pipeline that reruns cleanly at 3 AM without you.

Start the course free →

What you will learn

  1. Airflow: DAGs, tasks and the backfill that must be safe
    An orchestrator's job is not running things. It is running things again — after a failure, out of order, six months late — and getting the same answer.
  2. Anatomy of a pipeline: extract, land, model, serve
    Every data pipeline you will ever build is the same four boxes. Engineers who can name the box a bug lives in fix it in ten minutes; the rest re-run everything and hope.
  3. Airflow: DAGs, tasks, sensors, retries
    A DAG that works on a good day is easy. The real Airflow skill is the five knobs that decide what happens on a bad one: sensor mode, timeouts, retries, pools and trigger rules.
  4. Scheduling, dependencies and the backfill problem
    The run stamped 14 March starts on 15 March. Until that sentence feels obvious, every Airflow schedule, cross-DAG sensor and backfill you write will be off by one.
  5. Incremental loads: watermarks and CDC
    Copying the whole rides table every night works until it takes nine hours. Copying only what changed works until you discover what 'changed' fails to see.
  6. dbt: models, refs, tests, incremental strategies
    dbt's whole idea fits in one line: every table is a SELECT statement in a file, and ref() turns a folder of those files into a dependency graph you can build, test and re-run.
  7. Data contracts and schema evolution
    Most pipeline outages are not caused by the pipeline. They're caused by an app engineer renaming a column on a Friday, with no idea that 40 tables and the CFO's dashboard depended on it.
  8. Orchestrating failure: retries, alerting, SLAs
    Your pipeline will fail. The only design choice is whether it fails loudly at 02:10 to the right person with a runbook — or quietly, discovered by the CFO at the 10 am review.

How the course works