🔥 Free course

Spark & Databricks

Make a slow job fast, and explain exactly why it was slow.

Start the course free →

What you will learn

1. Spark: jobs, stages, and the shuffle that costs you everything
   Spark is fast until it shuffles. Almost every Spark performance problem is one question: how many times does the data cross the network?
2. Spark's execution model: driver, executors, stages
   Your cluster has 40 cores and your stage has 4 partitions. Thirty-six cores are being paid for to do nothing.
3. Transformations vs actions, lazy evaluation
   Your filter 'took 0.002 seconds'. It took nothing because it did nothing — yet.
4. Shuffles — the thing that makes jobs slow
   Every Exchange in your plan is a full copy of the data across the network. Count them before you run anything.
5. Partitioning, bucketing, broadcast joins
   Three tools, three different problems: partitioning skips files, bucketing skips shuffles, broadcasting skips the big side entirely.
6. Handling skew: salting and AQE
   199 tasks finish in 20 seconds. One runs for 70 minutes. Adding executors will change nothing.
7. Delta Lake: ACID, time travel, OPTIMIZE, VACUUM
   A Delta table is just Parquet files plus a folder of JSON receipts. Everything — ACID, time travel, rollback — comes from those receipts.
8. Medallion architecture in practice
   Bronze is your insurance policy, silver is your source of truth, gold is your product. Mix them up and every bug becomes a re-ingest.
9. Reading the Spark UI to find the bottleneck
   Every slow Spark job has the reason printed in the UI. Most people look at the wrong tab.

How the course works

• Each chapter is a short book: the basics, the explanation in several ways, the words to know, and a two-minute summary.
• Then you test yourself — quiz, puzzles — and recall it on two different days, so it stays.
• Blinks (one idea per screen), games and a daily plan keep you coming back. It is free.