🐍 Python for Data Engineers · Chapter 4

Lists, dicts, sets, tuples — choosing the right one

Four containers. Pick by the question you'll ask of the data — order, lookup, membership, or 'never change this'.

Study this lesson free in the app →

The basics

Python's four built-in containers — list, dict, set, tuple — all hold many values. They differ in which question is cheap to ask of the data.

The problem it solves: at 1,000 rows any choice works. At 1,000,000, checking membership in a list instead of a set turns a 4-second job into one that never finishes, silently.

The mental model: four ways to keep a wedding guest list.

• list — the arrival log. Ordered, duplicates allowed. Great for "who came third?", slow for "is Priya here?".
• dict — the reception index. Look up a key, get a value in one step.
• set — the hand stamp. Only "stamped or not?" — instant, no duplicates.
• tuple — the printed invitation. Fixed; because it can't change, it can be a dict key: (city, date).

Rule: if you write in inside a loop, the right-hand side should be a set or dict.

Explained

events = [e1, e2, e3]              # list  — order matters
by_id  = {"e1": e1, "e2": e2}      # dict  — find one instantly
seen   = {"e1", "e2"}              # set   — have I seen this?
point  = (25.57, 91.89)            # tuple — a fixed pair

All four hold things. The difference is which question is cheap, and at a million rows that difference is the whole job.

Four ways to keep a guest list

The list is the arrival log — names in the order they walked in. Perfect for "who came third?". Terrible for "is Priya here?", because you read from the top every time.

The dict is the reception desk index — look up a name, get their table number instantly. One step, whether there are 10 guests or 10 million.

The set is the hand stamp. You don't care about order or details, only: stamped or not? Instant, and it can't hold duplicates by construction.

The tuple is the printed invitation. Fixed, unchangeable, and because it can't change, it can itself be used as a dict key or dropped into a set.

The cost table — memorise this

| Operation | list | dict | set | tuple | | --- | --- | --- | --- | --- | | Get by position | O(1) | — | — | O(1) | | Get by key | O(n) | O(1) | — | O(n) | | x in ... | O(n) | O(1) | O(1) | O(n) | | Add | O(1) append | O(1) | O(1) | can't | | Keeps order | yes | yes (3.7+) | no | yes | | Allows duplicates | yes | keys no | no | yes | | Can be a dict key | no | no | no | yes |

The row that matters most is x in .... Checking membership against a list scans every element; against a set it's one hash lookup.

Inside a loop over a million events, that is the difference between four seconds and four hours. It is the most common real performance bug in Python data code, and it never raises an error — the program just stops finishing.

What you'll actually type

# ---- list: ordered, changeable ----
events = []
events.append(e)              # add to the end
events.extend(more)           # add many
events[0], events[-1]         # first, last
events[:5]                    # first five (a new list)
sorted(events, key=lambda e: e["time"])
len(events)

# ---- dict: key → value ----
by_id = {}
by_id["e1"] = event           # set
by_id.get("e9")               # None instead of KeyError
by_id.get("e9", 0)            # with a fallback
"e1" in by_id                 # checks KEYS, instantly
for key, value in by_id.items():
    ...

# ---- set: membership and dedup ----
seen = set()                  # NOT {} — that's an empty dict
seen.add("e1")
"e1" in seen                  # instant
unique_cities = set(all_cities)
set(a) & set(b)               # in both
set(a) - set(b)               # in a, not b

# ---- tuple: fixed ----
point = (25.57, 91.89)
lat, lon = point              # unpacking
by_cell = {(city, date): count}    # a compound key — only tuples can do this

Dedup with a set, step by step

Swap seen = set() for seen = [] and the logic is identical — but a million events would take hours instead of seconds.

Why is this slow?

seen = []
unique = []

for e in events:               # 1,000,000 events
    if e["id"] not in seen:
        seen.append(e["id"])
        unique.append(e)

It works perfectly on a 1,000-row test file — 0.01 seconds. On the real file it hasn't finished in an hour.

Gotchas

• {} is an empty dict, not an empty set. Use set()
• in against a list inside a loop — the quadratic trap above
• d["missing"] raises KeyError — use d.get(key) or d.get(key, default) when absence is normal
• Sets have no order — never rely on the order you get back; sort if you need one
• A list can't be a dict key — it's mutable, so it can't be hashed. Use a tuple
• list2 = list1 doesn't copy — both names tag the same list. Use list1.copy() or list(list1)
• Sorting a dict returns its keys — sorted(d) gives keys; you probably wanted sorted(d.items(), key=...)

Order → list. Lookup → dict. "Seen it?" → set. Fixed → tuple.

If you write in inside a loop, the right-hand side must be a set or a dict.

{} is a dict. set() is a set.

Words to know

In two minutes

• Pick the container by the question you'll ask: order → list, lookup by key → dict, 'seen it?' → set, fixed → tuple.
• x in list scans every element (O(n)); x in set or dict is one hash lookup (O(1)).
• A list membership check inside a loop is accidental O(n²) — fine on samples, endless on production data.
• Use .get(key, default) when a key may be missing; d[key] raises KeyError.
• {} is an empty dict; an empty set is set().
• Only hashable values can be dict keys or set members — use a tuple for compound keys like (city, date).
• b = a never copies a list; both names tag the same object. Use a.copy() or list(a).
• Sets have no order; dedup with a seen-set plus a list when order matters.

Test yourself

There are 11 questions, 3 puzzles and 6 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.