🏗 Scale & System Design · Chapter 3

Event Time vs Processing Time

A driver in a tunnel sends her 8:59 PM ping at 9:04. Which hour does it belong to?

Study this lesson free in the app →

The basics

Event time is when something actually happened, stamped by the device that saw it. Processing time is when your system got around to handling it. In a demo they match; in production they can be seconds or hours apart.

The problem it solves: a driver in a tunnel pings at 20:59 and the ping reaches Kafka at 21:04. Count it by processing time and 9 PM demand looks higher than it was, 8 PM lower, and surge pricing reacts to a rush that already ended. Every wrong hourly or daily report traces back to someone quietly using the wrong clock.

The mental model: postcards from a holiday. Sort them by the date written on the card (event time) and the album is right, but you must decide how long to wait for stragglers before closing a page. That decision is the watermark. Wait longer and you are more complete but less fresh; close sooner and you are fresher but must correct later.

Explained

Event time — when it actually happened, stamped by the producer.

Processing time — when your system finally saw it.

In a demo they are the same number. In production they differ by seconds, sometimes hours, and every wrong daily report traces back to someone quietly using the wrong one.

Postcards from a holiday

Friends post you postcards during a trip. Some take three days, one gets stuck in a sorting office for two weeks.

If you arrange them by the date written on the card, you reconstruct the holiday correctly — but you must decide how long to wait before declaring the album finished.

If you arrange them by the day they landed in your letterbox, the album is ready instantly and completely wrong.

Stream processing is that choice, made a million times a second. And watermarks are you saying: "anything posted before the 5th, I'll assume has arrived — I'm closing that page."

Windows, watermarks, allowed lateness

• Window — a bucket of time you aggregate over: tumbling (fixed, non-overlapping), sliding (overlapping), session (gap-based)
• Watermark — the engine's assertion that "no event older than T will still arrive". Usually max_seen_event_time − allowed_delay
• Allowed lateness — a grace period after the watermark during which late events still update a closed window
• Late/dropped data — arrivals past that grace period; route them to a side output rather than deleting them

The trade is stark and worth memorising: wait longer → more correct, less fresh. Close earlier → fresher, more corrections later. There is no setting that gives both.

The tunnel ping

event_time   20:59:02   ← when the driver was actually there
ingest_time  21:04:11   ← when Kafka accepted it
process_time 21:04:13   ← when your job handled it

window 20:00–21:00 aggregated by EVENT time  → ping counted ✅
window 21:00–22:00 aggregated by PROCESS time → ping counted ❌
                                                (9 PM demand overstated,
                                                 8 PM demand understated,
                                                 surge pricing misfires)

Always carry all three timestamps into the lake. You cannot reconstruct them later, and the gap between them is your best latency metric.

Why lateness is normal, not exceptional

At scale, late data is not an edge case — it is the weather:

• Phones lose signal in tunnels, lifts, basements, rural roads
• Apps buffer offline and flush hours later
• A regional broker fails over and replays a backlog
• A consumer is redeployed and catches up on 40 minutes of lag
• A partner sends yesterday's file at 3 AM

So mature platforms do two things. The streaming layer serves a fast approximate answer for operations (surge, ETA, live ops). The batch layer recomputes the same window later from immutable raw data and produces the number finance uses. Same logic, two latencies, and the batch result wins on disputes.

That pattern has a name — lambda architecture in its classic form, and increasingly a single engine running both modes over one table format (Delta/Iceberg). Knowing why it exists matters more than the label.

Event time = when it happened. Processing time = when you found out.

Aggregate by event time. Alert by processing time. Keep both, plus ingest time.

Watermark = "I'm closing this page of the album."

Gotchas

• Trusting client clocks — phone clocks are wrong, sometimes deliberately. Keep the client's event time and a server-received time, and clamp absurd values instead of silently accepting a 2027 timestamp
• Timezones — store UTC, convert at the display edge. A daily report defined in IST that aggregates UTC days is off by 5.5 hours and nobody notices for a quarter
• Watermark too aggressive — silently drops real events. Always emit a metric for dropped-late-record count
• Backfills that overwrite by processing time — re-running yesterday must rebuild yesterday's event-time partitions, or you smear corrected data into today
• Ignoring out-of-order within a partition — Kafka preserves order per partition, but the producer may have buffered events out of order before sending

The question they ask

"Your streaming job reports 5,000 rides in the 8–9 PM window. The batch job reports 5,040 for the same window. Who is right?"

"Both, at their own moment. The streaming number closed at the watermark, so it excludes 40 events that arrived after the grace period — typically drivers reconnecting after signal loss. The batch job recomputed from immutable raw data hours later with all events present. Operations uses the streaming number because a 1% error in real time is fine for surge decisions; finance uses the batch number because it must reconcile. What I would check is whether the gap is stable at ~1% or growing, because a growing gap means my watermark is mistuned or a producer is systematically late."

That last sentence — turning a number into a monitoring question — is what separates 10-year answers from 2-year answers.

Words to know

In two minutes

• Event time is when it happened, processing time is when you saw it; aggregate by event time.
• Keep event, ingest and processing timestamps in the lake — the gaps are your latency metric and cannot be rebuilt later.
• Windows bucket time: tumbling (fixed), sliding (overlapping), session (ends after a gap of inactivity).
• A watermark, usually max event time seen minus a delay, declares a window complete; later arrivals are late.
• Wait longer for more completeness, close earlier for freshness — no setting gives both. Route too-late events to a side output and count them.
• Late data is normal at scale: tunnels, offline buffers, failovers, consumer lag, partner files.
• Streaming gives a fast approximate number for operations; a batch recompute from raw gives the number finance trusts.
• Store UTC, define business days in the business timezone, and never trust client clocks blindly.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.