🧱 Foundations · Chapter 8

Table formats: Delta, Iceberg, Hudi

A folder of Parquet files is not a table. It has no idea which files are current, who is writing, or what it looked like yesterday. A table format is the small ledger that answers all three.

Study this lesson free in the app →

The basics

A table format is a metadata layer that sits beside a set of data files (almost always Parquet) on object storage and defines exactly which files make up the table right now. The three open ones are Delta Lake, Apache Iceberg and Apache Hudi.

The problem it solves: object storage can't edit a file in place or rename a folder atomically. So on a plain lake, a reader can see half a write, two writers can clobber each other, an UPDATE means hand-rewriting files, and there's no "yesterday's version". Table formats fix this by never changing data files — they write new files and then atomically commit a new metadata version that points at them.

The mental model: a bank passbook. The money (data files) sits in the vault; the passbook (metadata log) says what you own. Each transaction is a new line. Reading an old line is time travel. If two clerks write at once, only one line can be next — the other retries.

Explained

Delta, Iceberg and Hudi solve the same core problem — ACID tables on object storage — and they converge more every year.

What interviewers probe is how each one tracks files, because that explains everything else: concurrency, time travel, partitioning, and why your table got slow.

The passbook, not the vault

Your money in a bank isn't stored in your name on each note. The vault holds notes; your passbook records what's yours.

Withdraw ₹500 and nobody shreds notes in front of you — a new line is written. Want to know your balance on 1 March? Read up to that line.

Two clerks can't both write line 47. The second one sees line 47 is taken, re-checks your balance, and writes line 48.

And once a year the bank archives very old passbook pages — after that, you can't ask about 2019. Data files are the notes. The table format is the passbook.

What all three give you

• Atomic commits — writers add new files, then publish one metadata change; readers see the old version or the new, never half
• Snapshot isolation + optimistic concurrency — writers don't lock; at commit they check for conflicting changes and retry if needed
• Row-level UPDATE / DELETE / MERGE — via copy-on-write or merge-on-read
• Time travel — query an older snapshot, roll back a bad write
• Schema enforcement and evolution — reject bad writes; add columns safely
• File-level statistics — min/max per file in metadata, so engines prune files without listing folders or opening footers

The differences are in how metadata is laid out and what each was optimised for.

How each one tracks files

DELTA LAKE                    ICEBERG                       HUDI
_delta_log/                   catalog (Glue/REST/Hive/Nessie)  .hoodie/ timeline
 000…000.json                   └─► current metadata.json     instants:
 000…001.json  add/remove        └─► snapshot               20260314.commit
 …                                └─► manifest list (avro)    …deltacommit
 000…010.checkpoint.parquet         └─► manifests (avro)      …compaction
                                        file + partition      …clean
Commit = write next                     + column stats      base files (parquet)
numbered JSON (put-if-absent)       └─► data + delete files  + log files (MoR)
                              Commit = atomic swap of the    Commit = complete an
                              catalog's metadata pointer     instant on the timeline

The differences that matter

Delta Lake (Databricks, Linux Foundation) — ordered JSON commit log plus Parquet checkpoints. Deepest Spark/Databricks integration: OPTIMIZE, Z-ORDER, liquid clustering, deletion vectors (MoR-style deletes), change data feed. Renaming/dropping columns needs column mapping enabled. UniForm can expose Delta tables as Iceberg.

Apache Iceberg (Netflix) — tree of metadata → manifest list → manifests, committed by swapping a pointer in a catalog. Hidden partitioning (days(ts), bucket(16, ride_id)): users filter on ts, Iceberg prunes — no ride_date column to remember. Partition evolution without rewriting old data. Columns tracked by ID, so renames are safe. CoW or MoR with position/equality delete files. The widest engine support (Spark, Trino, Flink, Snowflake, BigQuery, Athena…).

Apache Hudi (Uber) — built for upserts and incremental pulls. Timeline of instants; record keys plus an index to find which file holds a key. CoW or MoR tables (Parquet base + Avro log files, compacted asynchronously). Native incremental queries: "rows changed since commit X".

At scale: what slows tables down

Small files. Streaming writes every minute → thousands of tiny files and a bloated metadata layer. Schedule compaction (OPTIMIZE / rewrite_data_files / Hudi clustering & compaction).

Delete-file buildup (MoR). Fast writes, but each read merges deletes/logs. Without regular compaction, reads get slower every day.

Snapshot buildup. Every commit adds metadata and keeps old files. Expire snapshots (expire_snapshots, VACUUM) — this is also what makes deletion requests actually remove data.

Concurrent writers. Optimistic concurrency means two MERGEs touching the same files conflict and one retries. Design writers to touch disjoint partitions, or serialise them.

Choosing: on Databricks → Delta. Multi-engine / vendor-neutral / Snowflake + Spark + Trino sharing tables → Iceberg. Heavy record-level upserts from CDC with incremental consumers → Hudi is purpose-built (though Delta and Iceberg handle CDC MERGE well too).

Gotchas

• Writing to the folder directly with a plain Parquet writer — the files exist but aren't in the log, so the table ignores them (or worse, VACUUM deletes them)
• VACUUM with a tiny retention — breaks time travel and can delete files a long-running reader still needs
• Assuming time travel is a backup — it lasts only until snapshots expire, and it's on the same storage
• Over-partitioning — partitioning by ride_id or by hour on a small table creates millions of tiny files; Iceberg hidden partitioning or clustering is usually better
• Hudi read-optimized queries on MoR — they read base files only and miss updates still sitting in log files

Never edit files — add files, then commit a pointer.

Delta: numbered JSON log. Iceberg: catalog → metadata → manifests. Hudi: timeline + record index, upsert-first. CoW pays on write, MoR pays on read.

Words to know

In two minutes

• A table format is a metadata layer defining which immutable data files make up a table at each version.
• All three give atomic commits, snapshot isolation with optimistic concurrency, row-level DML, time travel and schema evolution.
• Delta: _delta_log of numbered JSON commits plus Parquet checkpoints every 10 commits; strongest on Databricks.
• Iceberg: catalog → metadata.json → manifest list → manifests → files; hidden partitioning, partition evolution, ID-based columns, widest engine support.
• Hudi: timeline + record keys and indexes; built for upserts, CoW/MoR tables, native incremental queries.
• Copy-on-write rewrites files on update (fast reads); merge-on-read writes delete/log files (fast writes, needs compaction).
• Operate them: compact small files, expire snapshots/VACUUM, and avoid conflicting concurrent writers.

Test yourself

There are 10 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.