🔥 Spark & Databricks · Chapter 8

Medallion architecture in practice

Bronze is your insurance policy, silver is your source of truth, gold is your product. Mix them up and every bug becomes a re-ingest.

Study this lesson free in the app →

The basics

Medallion architecture organises a lakehouse into three layers of tables — bronze, silver, gold — each a cleaner, more useful version of the one before.

Why it exists: if a pipeline goes straight from raw JSON to a revenue dashboard, every bug in parsing, deduplication or business logic is baked into the only copy you have. You cannot replay, cannot tell where a wrong number came from, and every new use case rebuilds the cleaning from scratch.

The layers:

• Bronze — raw data exactly as received, append-only, plus ingestion metadata. Never 'fixed'.
• Silver — cleaned, typed, deduplicated, conformed entities: one row per ride, one row per driver. The reusable source of truth.
• Gold — business-shaped tables for a purpose: daily revenue per city, driver payouts, a star schema for BI.

Mental model: a rice mill. Paddy arrives in sacks and is stored as delivered (bronze). It is husked, cleaned and graded (silver). Then it is packed into 5kg retail bags and 25kg restaurant sacks (gold). If a grading machine was miscalibrated, you regrade from the stored paddy — you don't call the farmers.

Explained

Meghalaya Cabs discovered in November that fares since August had been parsed as paise instead of rupees for Tura — ₹2.80 rides instead of ₹280.

Team A had no bronze layer: the raw app events were parsed on arrival and discarded. Recovery meant asking the app team for three months of logs.

Team B kept bronze. They fixed one line in the silver job, replayed August–November from bronze, rebuilt the gold revenue tables, and closed the ticket the same afternoon.

The rice mill

Paddy arrives in sacks from forty farms. The mill stores every sack as delivered, tagged with the farm and date. Nobody cleans it there.

Next, it is husked, de-stoned and graded into a clean store — one bin per variety, every grain accounted for.

Finally it is packed: 5kg bags for shops, 25kg sacks for restaurants, a sample box for the inspector.

When a grading machine turns out to have been miscalibrated for a month, the mill regrades from the stored paddy and repacks. Nobody calls the farmers. That stored paddy is bronze.

Meghalaya Cabs' lakehouse

Arrows point one way. If a consumer needs something new, it is built from silver, never by reaching back into bronze.

What each layer promises

Bronze — completeness. Everything received, in arrival order, untouched: the raw payload (often as a string or VARIANT column) plus _ingest_ts, _source_file, Kafka offset. Append-only. Partitioned by ingest date. Retained as long as you might need to replay.

Silver — correctness. Parsed and typed columns, deduplicated on a producer-generated key, one row per entity at a declared grain, conformed codes (SHG not Shillong/shillong), PII masked or tokenised. Loaded with MERGE, so re-running a batch changes nothing. Bad rows go to quarantine with a reason.

Gold — usefulness. Aggregates, denormalised wide tables, star schemas. Business definitions live here (what counts as 'completed revenue'). Each table has a named consumer, a refresh SLA and an owner.

The contract between layers is what lets you rebuild any layer from the one below.

Bronze → silver, idempotently

# BRONZE: land raw files incrementally (Databricks Auto Loader)
(spark.readStream.format('cloudFiles')
   .option('cloudFiles.format', 'json')
   .option('cloudFiles.schemaLocation', '/chk/bronze_rides/schema')
   .load('s3://mc-landing/ride_events/')
   .withColumn('_ingest_ts', F.current_timestamp())
   .withColumn('_source_file', F.col('_metadata.file_path'))
   .writeStream.option('checkpointLocation', '/chk/bronze_rides')
   .trigger(availableNow=True)
   .toTable('bronze.ride_events'))

# SILVER: latest state per ride, MERGEd so replays are harmless
def upsert_rides(batch, batch_id):
    w = Window.partitionBy('ride_id').orderBy(F.desc('event_ts'))
    latest = (batch.dropDuplicates(['event_id'])
                   .withColumn('rn', F.row_number().over(w))
                   .filter('rn = 1').drop('rn'))
    latest.createOrReplaceTempView('u')
    batch.sparkSession.sql('''
      MERGE INTO silver.rides t USING u s ON t.ride_id = s.ride_id
      WHEN MATCHED AND s.event_ts > t.event_ts THEN UPDATE SET *
      WHEN NOT MATCHED THEN INSERT *''')

The s.event_ts > t.event_ts guard means a late, older event can never overwrite a newer state.

At 100×

Bronze storage is the cheap part. Compressed raw events for Meghalaya Cabs: ~40GB/day, ~15TB/year — about ₹28,000 a month at ₹1.9/GB-month. Losing replayability costs more the first time you need it. Move old bronze to cheaper storage tiers rather than deleting it.

Silver is where compute goes. Dedup and MERGE on billions of rows need the target clustered on the merge key (or partitioned by date with the date in the MERGE condition) so each batch rewrites a few files, not the table.

Gold sprawl is the real risk. Every analyst wants their own table. Without owners, you get 400 gold tables, 30 definitions of 'revenue', and a nightly bill nobody can explain. Prefer a governed set of gold models (dbt or DLT) plus views.

Backfills at scale — replay bronze by partition, in parallel, into silver; then recompute only the gold partitions whose dates changed.

Gotchas

• Cleaning bronze in place — the moment you 'fix' bronze you've lost the ability to prove or replay what arrived
• Silver loaded with append — a retried batch doubles rows and gold revenue doubles with it; use MERGE on a business key
• Dedupe in gold — every gold table reinvents it slightly differently; dedupe once in silver
• Gold reading bronze for 'just one field' — now gold depends on raw parsing, and a bronze schema change breaks a dashboard
• Dropping bad rows silently — revenue is quietly 2% low for months; quarantine with a reason and alert on volume
• Layers for their own sake — a silver table identical to bronze, or a gold table identical to silver, is cost without value

Bronze: keep everything. Silver: get it right once. Gold: shape it for someone.

Fix in the layer that owns the logic, then rebuild downstream from the layer below.

Words to know

In two minutes

• Bronze holds raw data as received, append-only, with ingest metadata (source file, ingest time) — its job is replayability, not usability.
• Silver holds cleaned, typed, deduplicated entities keyed by business IDs, loaded with idempotent MERGE — the layer most consumers and ML should read.
• Gold holds purpose-built aggregates and dimensional models for BI and products; each gold table has an owner and a consumer.
• Data flows one way: bronze → silver → gold. Gold never reads bronze; fixes happen in the layer where the logic lives, then flow downstream.
• A logic bug is fixed by reprocessing from the layer below — silver from bronze, gold from silver — which only works if bronze kept everything.
• Bad records go to a quarantine table with a reason, not silently dropped; quality checks (e.g. DLT expectations) gate silver.
• It is a convention, not a law: add layers only when they earn their storage and compute; not every silver table needs a gold one.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.