◐ ai-teacher

🔀 Pipelines & Orchestration · Chapter 7

Data contracts and schema evolution

Most pipeline outages are not caused by the pipeline. They're caused by an app engineer renaming a column on a Friday, with no idea that 40 tables and the CFO's dashboard depended on it.

Study this lesson free in the app →

The basics

A data contract is an explicit, versioned agreement between whoever produces data (an app team, a service emitting events) and whoever consumes it (the data platform, analysts, ML). It pins down the schema (fields, types, nullability), the meaning (is fare in rupees or paise?), quality and freshness expectations, the owner, and how changes are allowed to happen. Schema evolution is the set of rules for changing a schema without breaking the people reading it.

The problem it solves: producers change their data for their own reasons. Without a contract, the data team discovers the change when a pipeline fails at 3 am — or worse, when it doesn't fail and silently loads wrong numbers.

The mental model: an API for data. Backend teams wouldn't remove a field from a public REST API without versioning it. A contract gives tables and event streams the same discipline, enforced by tooling (CI checks, schema registries), not by goodwill.

Words you must know 5

Backward compatible
new readers can read old data
Forward compatible
old readers can read new data
Breaking change
one that makes existing readers fail or misread
Schema registry
a service storing schema versions and rejecting incompatible ones
Expand and contract
add the new form, migrate, then remove the old

Explained

Meghalaya Cabs' app team changes fare from rupees to paise — still an integer, still called fare. Every schema check passes.

Revenue on the dashboard jumps 100×. The schema was compatible. The meaning wasn't. A data contract covers both.

The tiffin service agreement

A dabbawala collects tiffins from 300 homes and delivers to offices. It works because of an unwritten contract: tiffin at the gate by 9:30, name painted on the lid, standard box size.

One family starts using a bigger box. It doesn't fit the crate — a schema break, noticed instantly. Another family switches the painted code from the office's floor to the building number — the box still fits, but it goes to the wrong desk. That's a semantic break, noticed when someone goes hungry.

Sensible families tell the dabbawala a week ahead and paint both codes for a while. That's versioning and expand-and-contract.

What a contract contains

# contracts/ride_completed.v2.yaml
name: ride_completed
version: 2.1.0
owner: rides-backend (#rides-oncall)
schema:
  ride_id:     {type: string,  required: true, unique: true}
  driver_id:   {type: string,  required: true}
  fare_paise:  {type: long,    required: true, min: 0,
                description: "Final fare in paise, incl. GST"}
  city:        {type: string,  enum: [Shillong, Tura, Jowai, Nongpoh]}
  completed_at:{type: timestamp, timezone: UTC}
  tip_paise:   {type: long, required: false, default: 0}   # added 2.1
quality:
  freshness: "p99 < 5 min from completion to topic"
  completeness: ">= 99.9% of rides vs ops DB daily"
compatibility: BACKWARD_TRANSITIVE
change_policy: "breaking changes → new major version, 30 days dual-publish"

Notice: units in the field name, timezone stated, an owner with a channel, and a change policy.

Compatibility, precisely (Avro / Schema Registry)

What each change does in Avro:

change                          backward  forward
─────────────────────────────────────────────────
add field WITH default             ✓        ✓
add field without default          ✗        ✓
remove field that had a default    ✓        ✓
remove field without default       ✓        ✗
rename field (no alias)            ✗        ✗
int → long (promotion)             ✓        ✗
change type string → int           ✗        ✗

Rule of thumb: every new or removable field gets a default and you stay fully compatible. Protobuf's equivalent: never reuse or renumber a field tag; mark removed ones reserved.

How a change flows when contracts are enforced

The key shift: the breaking change is caught in the producer's CI, before deploy — not in the data team's pipeline after.

Renaming a column without an outage

Meghalaya Cabs wants fare → fare_paise (and to switch units). Expand and contract:

Week 0 — expand. Producer adds fare_paise alongside fare, writing both. Contract v2.0 marks fare deprecated with a removal date.

Weeks 1–3 — migrate. dbt staging reads coalesce(fare_paise, fare * 100). Each consumer moves to fare_paise; lineage shows who still reads fare.

Week 4 — contract. Lineage + query logs show zero readers of fare. Producer stops writing it; contract v3.0 removes it.

No night was lost. Compare: the Friday rename that broke 40 tables took a weekend and a post-mortem.

At scale

Enforce at the edges, both of them. Producer side: CI compatibility checks against a schema registry or contract files. Consumer side: ingestion validates and routes failures to a dead-letter queue / quarantine table with alerts — never drop silently.

In the warehouse: dbt model contracts (contract: {enforced: true}) fail the build if a model's columns or types drift from the YAML; model versions let fct_rides_v2 coexist with v1 during migration. Delta's schema enforcement rejects mismatched writes unless you opt into mergeSchema; Iceberg's ID-based columns make renames safe for the table — but not for downstream SQL that uses the old name.

Organisationally: contracts work when the producer owns them, with a named on-call. A contract that only the data team writes is just documentation.

Don't over-contract. Put contracts on the interfaces many people depend on (core events, marts), not on every staging table.

Gotchas

Words to know

Data contractA versioned, producer-owned agreement defining a dataset's schema, semantics, quality and change rules.
Backward compatibilityReaders using the new schema can read data written with older schemas.
Forward compatibilityReaders using an older schema can read data written with the new schema.
Schema registryA service that stores schema versions and rejects changes violating the compatibility mode.
Expand and contractIntroduce the new form alongside the old, migrate consumers, then remove the old form.
Semantic changeA change in meaning (units, timezone, codes) that keeps the schema technically identical.
Quarantine tableWhere records failing validation are stored for inspection instead of being dropped.

In two minutes

Test yourself

1. Under BACKWARD compatibility, which statement holds?

  1. Old consumers can read new data
  2. Producers must upgrade first
  3. Consumers using the new schema can read data written with the previous schema
  4. No fields may ever be added
Show the answer

Consumers using the new schema can read data written with the previous schema — BACKWARD means new readers handle old data, so you upgrade consumers first. FORWARD is the reverse.

2. Which Avro change is compatible both backward and forward?

  1. Adding an optional field with a default value
  2. Adding a field without a default
  3. Renaming a field without an alias
  4. Changing a field from string to int
Show the answer

Adding an optional field with a default value — New readers fill the default for old data; old readers ignore the extra field in new data.

3. A producer removes a field that had no default. Who breaks?

  1. Consumers on the new schema reading old data
  2. Nobody
  3. Only the schema registry
  4. Consumers still on the old schema reading new data (it isn't forward compatible)
Show the answer

Consumers still on the old schema reading new data (it isn't forward compatible) — Old readers expect the field and there's no default to fill in. New readers simply ignore the field in old records, so backward holds.

4. The app team switches fare from rupees to paise, keeping type long and the same name. What does a schema registry do?

  1. Rejects it as incompatible
  2. Accepts it — the schema is unchanged; only the meaning changed
  3. Converts values automatically
  4. Creates a new topic
Show the answer

Accepts it — the schema is unchanged; only the meaning changed — Schema checks see identical types. Semantic changes need contract-level rules: units in names, value-range checks, and a change policy.

5. Which is the safest way to rename fare to fare_paise?

  1. Expand and contract: add fare_paise, dual-write, migrate consumers, then remove fare
  2. Rename in place on a Friday
  3. Rename and set compatibility to NONE
  4. Ask consumers to update their queries after deploy
Show the answer

Expand and contract: add fare_paise, dual-write, migrate consumers, then remove fare — Dual-writing keeps every reader working while they migrate; lineage tells you when nobody reads the old field.

6. A Kafka topic keeps 90 days of data in Avro. Which compatibility setting protects a consumer that replays from the beginning?

  1. BACKWARD
  2. BACKWARD_TRANSITIVE
  3. FORWARD
  4. NONE
Show the answer

BACKWARD_TRANSITIVE — Plain BACKWARD only checks against the previous version. Transitive checks against all versions, which a full replay will encounter.

There are 10 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.

Learn it so it stays. Read it, test yourself, then recall it on two different days. The app schedules that for you, free.

Start this lesson →