🔥 Spark & Databricks · Chapter 9

Reading the Spark UI to find the bottleneck

Every slow Spark job has the reason printed in the UI. Most people look at the wrong tab.

Study this lesson free in the app →

The basics

The Spark UI is the web dashboard every Spark application serves (port 4040 on the driver, the 'Spark UI' link in Databricks, or the History Server after the job ends). It shows what actually ran: jobs, stages, tasks, the physical plan with real row counts, and executor health.

Why it matters: guessing is expensive. Doubling a cluster to fix skew wastes money; adding partitions to fix a driver-side bottleneck changes nothing. The UI tells you which kind of slow you have, and each kind has a different fix.

Mental model: a hospital chart. You don't start surgery because the patient 'feels slow' — you read temperature, blood pressure and bloods first. In Spark the vitals are: where the time went (which stage), how it was spread (task distribution), whether memory coped (spill, GC), and whether the plan was sensible (rows read vs needed).

Explained

Meghalaya Cabs' payout job went from 25 minutes to 2 hours. Three engineers, three theories: 'the cluster is too small', 'Delta is slow', 'the data grew'.

The Stages tab took ten seconds to settle it: one stage, 400 tasks, median 9 seconds, max 96 minutes, 31GB spilled by that single task. A new corporate fleet account had joined — 22% of all rides under one driver_id.

None of the three theories would have fixed it.

The ward round

A junior doctor sees a patient who 'isn't well' and wants to order every scan. The senior doctor reads the chart first: temperature normal, blood pressure normal, oxygen 88%.

That one number points to the lungs, and the next ten minutes go there instead of everywhere.

The Spark UI is the chart. Duration tells you which organ, the task distribution tells you if it's one spot or everywhere, spill and GC tell you if memory is struggling, and the plan tells you whether it's doing far more work than it should.

The diagnostic path

Top-down: application → query → stage → task → executor. Stop as soon as one metric explains the time.

Four stage signatures

                 Duration p25/med/p75/max    Spill(disk)  Verdict
A  skew          10s / 12s / 15s / 96min     31GB (1 task) one hot key
B  too big       6m / 7m / 8m / 9m           1.4TB        partitions too large
C  too many      8ms / 11ms / 15ms / 40ms    0            50k tiny tasks, overhead
D  healthy       40s / 52s / 61s / 80s       0            just volume

Executors tab                      Verdict
GC Time 41% of Task Time (red)     memory pressure: fat executors,
                                   over-caching, big UDF objects
Tasks: 312 failed, 6 executors lost   OOM or spot reclaim; check logs

A needs salting/AQE/broadcast. B needs more shuffle partitions or memory per task. C needs fewer partitions or compacted files. D needs more slots — the only case where a bigger cluster is the answer.

Reading the SQL tab plan graph

Scan parquet silver.rides
  number of files read: 41,280        ← expected ~120 for one day
  size of files read: 2.1 TB
  number of output rows: 9,812,004,551
      │
Filter (pickup_ts >= 2026-11-04 ...)
  number of output rows: 13,440,118   ← 99.9% thrown away AFTER reading
      │
Exchange hashpartitioning(driver_id, 200)
  shuffle bytes written: 1.9 GB
      │
SortMergeJoin
  spill size: 0

Read it bottom-up from the scan. The red flag isn't the join — it's 41,280 files and 9.8 billion rows read to keep 13 million. The filter is on pickup_ts while the table is partitioned by ride_date, so no partition pruning happened.

# confirm before changing anything
spark.table('silver.rides').filter(F.col('pickup_ts') >= '2026-11-04').explain('formatted')
# PartitionFilters: []   ← empty: nothing pruned

Fix: add ride_date = '2026-11-04' to the filter. Expected: files read drops to ~120.

The metrics that matter

Stage summary (per task):

• Duration and Shuffle Read Size — compare max with 75th percentile
• Spill (Memory) / Spill (Disk) — same data, deserialised vs on-disk size; any disk spill means execution memory was too small for the partition
• GC Time — high relative to duration = memory pressure
• Scheduler Delay / Task Deserialization Time — high relative to tiny durations = too many tasks or a huge closure
• Shuffle Read Blocked Time — waiting on remote fetches: network or overloaded executors
• Peak Execution Memory — how close tasks came to the limit

Event timeline — coloured bars per task. One long bar alone = straggler. Long green 'executor computing' everywhere = genuine work. Lots of orange/red slivers = overhead.

Jobs timeline — gaps where no job runs are driver time: listing millions of files, collect(), toPandas(), Python loops issuing one small job each.

AQE note — with AQE on, the SQL tab shows the final plan (isFinalPlan=true) with AQEShuffleRead nodes noting coalesced or skew-split partitions.

At 100×

Jobs with hundreds of stages — don't read them all. In the Stages tab, sort completed stages by duration; the top three usually own 80% of the time.

Long-running and finished apps — the live UI disappears with the driver. Enable event logs (spark.eventLog.enabled) and use the History Server; Databricks keeps them per cluster for a while.

Turn findings into alerts. Once you know the signatures, track them automatically: task-duration max/p75 ratio, spill bytes, GC fraction, files read per query. A pipeline whose max/p75 ratio crept from 2 to 20 over a month is telling you a hot key is growing before it fails.

Compare runs, not single runs. A 2-hour run means little alone; next to last week's 25-minute run with the same stage IDs, the stage that grew is the answer.

Gotchas

• Looking only at total job time — it hides which stage and which kind of slowness
• Reading the mean — a mean of 30s hides one 96-minute task; always compare max to percentiles
• Scaling up on skew or driver-bound jobs — idle executors, bigger bill
• Ignoring the jobs timeline — 20 minutes of gaps means the driver, not the cluster, is the bottleneck
• Blaming the write — with lazy evaluation, the write's job contains every upstream stage; look at stages, not the line of code
• Changing three settings at once — you'll never know which one helped; change one, rerun, compare the same metric
• Forgetting AQE rewrote the plan — the initial explain() may show a sort-merge join that ran as a broadcast

Words to know

In two minutes

• Start in the SQL/DataFrame tab: find the slow query, read its plan graph bottom-up, and compare rows and bytes per operator with what you expected.
• Then the Stages tab: sort by duration, open the slowest stage, read the summary metrics and the event timeline.
• Max ≫ 75th percentile in duration or shuffle read → skew. All percentiles high with Spill (Disk) → partitions too big for memory.
• Thousands of tasks of a few milliseconds, or scheduler delay dominating → too many partitions or small files.
• Executors tab: GC time above ~10% of task time → memory pressure (fat executors, over-caching, huge objects); failed tasks and lost executors → OOM or spot loss.
• Gaps in the jobs timeline with no stages running → the driver is busy (file listing, collect, Python loops, planning), not the cluster.
• A scan reading far more files or bytes than the filter implies → pruning or pushdown failed; check 'files read' and PushedFilters/PartitionFilters.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.