🧱 Foundations · Chapter 6

Compression and encoding: why Parquet is 10x smaller

Writing Shillong one crore times costs 80 MB as text. Parquet stores it in a few kilobytes — not by compressing harder, but by noticing it's the same word.

Study this lesson free in the app →

The basics

Encoding rewrites a column's values into a smaller representation using what it knows about them — "these are the same 40 cities over and over", "these timestamps increase by a few seconds each". Compression is a generic algorithm (Snappy, gzip, zstd) that squeezes whatever bytes it is given.

The problem it solves: storage is cheap, but reading is not. Every byte a query scans costs I/O, network and money (BigQuery and Athena literally bill per byte scanned). Fewer bytes → faster, cheaper queries.

The mental model: Parquet does it in two layers, per column, per page:

1. Encode — dictionary, run-length, delta, bit-packing turn values into tiny integers
2. Compress — a codec squeezes the encoded page

Columnar layout is what makes step 1 possible: a page full of city values is highly repetitive; a CSV line mixing city, fare and timestamp is not.

Explained

A month of Meghalaya Cabs rides: 9 GB as CSV, about 900 MB as Parquet.

About half of that win comes from encoding — exploiting what the values are. The codec then works on far friendlier bytes. Understand the first part and you can make files smaller and queries faster by changing nothing but the sort order.

The cricket scorer's shorthand

A scorer doesn't write "no run, no run, no run, no run, one run, no run". He writes dots: . . . . 1 . — a dictionary of symbols instead of words.

A maiden over isn't six dots, it's a single M — a run-length: "nothing, six times".

The total isn't rewritten each ball; he notes the change — +1, +4 — a delta.

At the end of the day he folds the scorebook into an envelope to post it — that's compression, applied to an already tiny record. The shorthand did most of the work.

Parquet's encodings, precisely

Chosen per column chunk by the writer:

• Dictionary (RLE_DICTIONARY) — a dictionary page holds each distinct value once; data pages hold ids, stored with the RLE/bit-packing hybrid. Default for most columns. Writers fall back to PLAIN when the dictionary grows too large (~1 MB by default in parquet-mr/Spark) — e.g. a ride_id column
• RLE/bit-packing hybrid — repeated runs become (value, count); short mixed stretches are bit-packed. Also used for booleans and for definition/repetition levels
• DELTA_BINARY_PACKED — integers/timestamps stored as bit-packed differences from the previous value
• DELTA_BYTE_ARRAY — strings stored as "shared prefix length + suffix" (great for sorted URLs, ids with common prefixes)
• BYTE_STREAM_SPLIT — splits float bytes into streams so the codec compresses them better
• Nulls are not stored as values at all — definition levels record where they are

Then each page is compressed with the file's codec.

Where the 10× comes from

column       CSV text            Parquet encoding        approx size
                                                        (1 crore rows)
─────────────────────────────────────────────────────────────────────
city         "Shillong" ~8 B     dictionary + RLE        80 MB → KBs*
pickup_ts    "2026-03-14 10:03:27" delta-binary-packed   190 MB → ~15 MB
             19 B
fare_paise   "24500" ~5 B        dictionary/bit-packed   50 MB → ~12 MB
is_cash      "false" 5 B         RLE / bit-packed        50 MB → ~1 MB
notes (90%   "" + delimiter     definition levels only  10 MB → ~0.2 MB
 null)

* when sorted by city; ~7.5 MB of 6-bit ids if not
 then: Snappy/zstd on every page squeezes further

The trick isn't one algorithm. It's choosing the right one per column, which only columnar storage allows.

Codecs: the second layer

codec    speed      ratio    use for
──────────────────────────────────────────────────────
Snappy   very fast  modest   Spark's Parquet default; hot data
LZ4      very fast  modest   same niche as Snappy
zstd     fast       high     modern default; tunable levels
gzip     slow       high     archives, legacy

Crucial fact: Parquet compresses page by page, so a Parquet file stays splittable whatever codec you choose — workers split on row groups. That's different from a .csv.gz, where one gzip stream covers the whole file.

ORC works the same way (dictionary, RLE v2, then a codec — ZLIB by default).

At scale: the levers you actually pull

1. Sort before writing. Sorting by low-cardinality filter columns (city, ride_date, status) lengthens runs, shrinks files, and tightens min/max stats for skipping. Teams routinely see 20–40% smaller files from sort order alone.

2. Watch for dictionary fallback. High-cardinality strings (UUIDs) blow the dictionary limit and fall back to plain. That's expected — don't fight it, but don't expect them to compress well.

3. Pick zstd for cold, Snappy/LZ4 for hot. On object storage, bytes read usually cost more than CPU, so zstd often wins even for hot data — measure.

4. Store the right types. A fare as a STRING "245.50" defeats delta and bit-packing; store paise as INT64 or use DECIMAL. Timestamps as strings are the same mistake.

Gotchas

• Confusing encoding with compression — interviewers check you know they are separate layers
• Repartitioning randomly before writing — repartition(200) shuffles rows, destroying the sort; files grow noticeably
• Numbers and dates as strings — kills delta encoding and correct min/max ordering ('9' > '10' as strings)
• Assuming gzip makes Parquet unsplittable — it doesn't; compression is per page
• Max compression everywhere — gzip level 9 on hot data spends CPU on every read for a few % of bytes

Encode by meaning, then compress the bytes.

Dictionary for repeats, RLE for runs, delta for sequences, bit-pack the ids — and sort first so the runs are long.

Words to know

In two minutes

• Encoding uses knowledge of the values; compression is a generic byte squeezer applied afterwards.
• Columnar layout puts similar values together, which is what makes encoding effective.
• Parquet's main encodings: dictionary, RLE/bit-packing hybrid, delta binary packed, delta byte array, byte stream split.
• Dictionary falls back to plain when the dictionary gets too large (high-cardinality columns).
• Nulls cost almost nothing: definition levels mark them, no value is stored.
• Codecs compress per page — Snappy/LZ4 fast, zstd/gzip smaller — and Parquet stays splittable with any codec.
• Sorting before writing lengthens runs, shrinks files and sharpens min/max skipping.

Test yourself

There are 9 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.