🐍 Python for Data Engineers · Chapter 12

Pandas to PySpark: what changes and what breaks

The API looks familiar on purpose. The execution model underneath is nothing like Pandas, and that gap is where jobs die.

Study this lesson free in the app →

The basics

PySpark is the Python API for Apache Spark, an engine that runs DataFrame transformations across many machines. Its syntax deliberately resembles Pandas; its execution model does not.

The problem it solves: Pandas works on one machine and needs the data in RAM. When yesterday's rides are 2 TB in S3, you need an engine that splits the data into partitions, runs work on dozens of executors in parallel, and survives a machine dying mid-job.

The mental model: Pandas is a cook who does each step the moment you say it. Spark is a head chef who reads the whole recipe, optimises it, and only starts cooking when you ask to see the dish. Every DataFrame line just adds to a plan; nothing runs until an action. And each action cooks the whole recipe again from scratch, unless you cache().

Explained

# Pandas — runs now, on this machine, in this memory
df[df.status == "completed"].groupby("city").fare.sum()

# PySpark — runs nowhere yet, on machines you cannot see
df.filter(df.status == "completed").groupBy("city").sum("fare")

The second line builds a plan. Nothing executes until you ask for a result. Understanding that one sentence is most of the difference.

The cook and the head chef

Pandas is a cook. You say "chop the onions" and they chop the onions, now, in front of you. If you ask for something bigger than the counter, everything falls on the floor.

Spark is a head chef running forty kitchens. You hand over the whole recipe. They read it end to end, notice you only need three of the twelve ingredients, reorder the steps, decide which kitchen does what, and only then shout "go" — when you finally ask to see the dish.

Two consequences follow immediately.

Your recipe can be enormous — forty kitchens, not one counter.

And you cannot peek mid-recipe for free. Every .show(), .count() or .collect() makes the whole brigade actually cook, from the beginning, unless you told them to keep something aside.

What happens when you call an action

Nothing above happens until an action. Transformations are free; actions are the bill.

The translation table

# ---------- Pandas ----------            # ---------- PySpark ----------
df[df.fare > 100]                          df.filter(F.col("fare") > 100)
df[["city", "fare"]]                       df.select("city", "fare")
df["tax"] = df.fare * 0.05                 df = df.withColumn("tax", F.col("fare") * 0.05)
df.rename(columns={"a": "b"})              df.withColumnRenamed("a", "b")

df.groupby("city").fare.sum()              df.groupBy("city").agg(F.sum("fare").alias("revenue"))
df.merge(d, on="id", how="left")           df.join(d, on="id", how="left")
df.sort_values("fare", ascending=False)    df.orderBy(F.col("fare").desc())
df.drop_duplicates("ride_id")              df.dropDuplicates(["ride_id"])
df.fillna(0)                               df.fillna(0)

len(df)                                    df.count()          # a full job!
df.head()                                  df.show(5)          # an action
df.to_parquet(path)                        df.write.parquet(path)

# no Pandas equivalent — this is the important one
df.cache()          # keep the computed result for reuse
df.explain()        # read the physical plan

Notice len(df) vs df.count(). In Pandas it is a property lookup. In Spark it executes the entire pipeline to find out.

Why is this job 3× slower than it should be?

rides = spark.read.parquet("s3://raw/rides/")
clean = rides.filter(F.col("status") == "completed") \
             .join(drivers, "driver_id")

print(clean.count())
clean.groupBy("city").agg(F.sum("fare")).show()
clean.write.parquet("s3://marts/rides_clean/")

The four differences that actually matter

1. Lazy, not eager. Transformations build a plan; only actions (show, count, collect, write) run it. A bug in your filter may not surface until the write, three functions later.

2. Partitioned, not one block. Your data is split across executors. Anything requiring rows to meet — a join key, a groupBy key, an orderBy — triggers a shuffle: data moves across the network. Shuffles are the dominant cost in almost every slow job.

3. No index, no row order. There is no .iloc, no .loc, no meaningful row position. Row order is not preserved unless you explicitly orderBy — and doing so on a large frame forces a full sort.

4. The driver is a bottleneck, not a worker. collect() pulls every row into the driver's memory. On a large frame that is an instant OOM. toPandas() is the same trap wearing a friendlier name.

Lazy evaluation, step by step

This is why filter early / select early matters less than beginners think — Catalyst reorders it anyway — but it is still how you make your intent readable.

The three things that make Spark jobs slow

1. Shuffles. Every groupBy, join, distinct and orderBy moves data across the network. Mitigations: broadcast the small side of a join (F.broadcast(dim) — under ~10 MB by default), pre-partition on the join key, and aggregate before joining rather than after.

2. Skew. One city is 40% of the rides, so one task does 40% of the work while 199 tasks idle. The job takes as long as its slowest task. Mitigations: salt the hot key, enable Adaptive Query Execution (spark.sql.adaptive.enabled, on by default in recent versions), or split hot keys into their own job. Exactly the same skew lesson as Kafka partition keys.

3. Small files. Ten thousand 2 MB files means ten thousand tasks each doing almost nothing — scheduling overhead dominates. Mitigations: coalesce() before writing, OPTIMIZE/compaction on Delta, and partition columns with sensible cardinality (partitioning by event_id creates a directory per row).

Diagnosis order in the Spark UI: find the longest stage, look at its task duration distribution. Even bars = fine. One long bar = skew. Thousands of tiny bars = small files.

Words to know

In two minutes

• PySpark looks like Pandas but is lazy: transformations build a plan, and only actions (show, count, collect, write) run it.
• Each action recomputes the plan from the source; cache a DataFrame that feeds several actions, and unpersist it afterwards.
• Data is partitioned across executors; joins, groupBys, distinct and orderBy shuffle data over the network, and shuffles dominate cost.
• No index, no row order: forget iloc, and expect order only after an explicit orderBy.
• collect() and toPandas() pull every row into the driver — aggregate first, then bring back the small result.
• Slow jobs are almost always shuffle, skew or small files; read the longest stage's task-duration distribution in the Spark UI.
• Prefer built-in functions over UDFs: Python UDFs serialise every row and are opaque to the Catalyst optimiser.
• Before migrating, check whether DuckDB or Polars on one machine is enough.

Test yourself

There are 9 questions, 3 puzzles and 7 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.