🧱 Foundations · Chapter 4
Batch vs streaming — how to choose
Streaming is not the grown-up version of batch. It is a more expensive answer to a question most dashboards never ask: how stale can this number be before someone loses money?
Study this lesson free in the app →
The basics
Batch processing collects data for a while and then processes the whole chunk at once — every night, every hour. Streaming processes each event within seconds of it happening, continuously, forever.
The problem it solves: every number in a company has a freshness requirement. Surge pricing at Meghalaya Cabs is useless if it is computed from rides an hour old. The monthly driver payout report is perfectly fine computed once a night. Choosing batch or streaming is choosing how much you pay for freshness.
The mental model: batch works on a bounded dataset — yesterday's rides, a file that has an end. Streaming works on an unbounded dataset — rides that will keep arriving as long as the company exists. Bounded data lets you sort, re-run and check totals. Unbounded data forces you to decide when an answer is "done" while more data keeps coming.
Words you must know 5
- Latency
- time from event to usable result
- Bounded / unbounded
- data with an end vs data without one
- Micro-batch
- tiny batches every few seconds; the middle ground
- Late data
- events that arrive after their window was computed
- State
- what a streaming job must remember between events (running counts, open windows)
Explained
Meghalaya Cabs wants three numbers: surge multiplier per area, fraud flags on UPI payments, and yesterday's revenue by city.
Only two of them need streaming. Picking which two — and saying why the third doesn't — is the whole skill.
The dhobi and the washing machine
A dhobi collects the whole colony's clothes on Sunday, washes them in one go, and returns them Tuesday. Cheap per shirt, efficient, predictable. If one shirt comes back stained, he re-washes the bundle.
A washing machine at home washes the shirt the minute you spill tea on it. Fast — but it runs for one shirt, someone must be home to run it, and it costs more per shirt.
Nobody washes every shirt the minute it gets dirty. You do that only for the shirt you need tonight. That is the decision: pay for immediacy only where someone needs it now.
Precisely
Batch — a job reads a bounded input (a day's partition), computes, writes, exits. Scheduled by an orchestrator. Re-running = re-reading the same input → same answer.
Streaming — a long-running job reads an unbounded log (Kafka, Kinesis), updates results per event or per window, never exits. Must manage state, checkpoints and late events.
Micro-batch — Spark Structured Streaming's default: run a small batch every N seconds over newly arrived data. Latency of seconds, batch-style simplicity.
The three costs you trade:
- Latency — hours vs seconds
- Money — a job that runs 20 min/day vs a cluster that runs 24×7
- Complexity — streaming adds ordering, watermarks, exactly-once and on-call
The decision, step by step
Start from the consumer of the number, never from the technology.
Meghalaya Cabs, sorted by freshness
use case who acts needs choose
──────────────────────────────────────────────────────────────
surge pricing pricing svc seconds streaming
UPI fraud flag payment svc < 2 s streaming
driver ETA / location rider app seconds streaming
ops dashboard (live) city manager ~1 min micro-batch
driver payouts finance daily batch
revenue by city leadership daily batch
ML training set data science weekly batchNotice: the left column is a business sentence. The right column falls out of it.
The ₹1.5 lakh dashboard
March. A product manager asks for a "real-time" revenue dashboard. The team builds Kafka → Flink → Postgres. A 3-node cluster runs 24×7: roughly ₹1.5 lakh a month plus one engineer's on-call week.
June. Someone checks the access logs. The dashboard is opened twice a day — at the 10 am stand-up and at 6 pm.
An hourly batch job on the existing warehouse would have cost about ₹3,000 a month and been at most 60 minutes stale at both viewings.
The question that would have saved ₹1.4 lakh/month: "What will you do differently if the number is 10 seconds old instead of 60 minutes old?"
At scale: you usually run both
Mature platforms rarely pick one.
Lambda architecture — a streaming path for fast, approximate answers and a batch path that recomputes the truth nightly. Accurate, but you maintain two codebases that must agree.
Kappa architecture — everything is a stream; to fix history you replay the log from the start through the same code. One codebase, but you need long retention and a replay-able job.
What most teams actually do: stream the handful of hot paths (pricing, fraud, ETAs) and land the same events into the lake, where batch jobs build the reporting tables. The stream is for acting; the batch is for accounting. Finance numbers come from batch because bounded data can be reconciled to the rupee.
Gotchas
- "Real-time" with no number attached. Always ask: real-time meaning 1 second or 1 hour?
- Streaming for a report read once a day. Pays 24×7 for a freshness nobody consumes
- Forgetting late data. A streaming count for 10:00–10:05 closes at 10:05; a ride from a phone in a tunnel arrives at 10:09. Without watermarks it is silently dropped
- Thinking batch can't be fast. Hourly or 15-minute batches cover most "near real-time" asks
- Underestimating ops. A batch job that fails is retried in the morning; a streaming job that fails is a page at 3 am with lag piling up
Words to know
| Batch processing | Processing a bounded chunk of data in one scheduled run, then exiting. |
|---|---|
| Stream processing | A long-running job that updates results continuously as each event arrives. |
| Latency | Time between an event happening and its effect being visible in the output. |
| Micro-batch | Running very small batches every few seconds over newly arrived data. |
| Late data | Events that arrive after the window they belong to has already been computed. |
| Lambda architecture | A streaming fast path plus a batch path that recomputes the correct result. |
| Kappa architecture | One streaming path; history is fixed by replaying the log through the same code. |
In two minutes
- Batch processes bounded chunks on a schedule; streaming processes an unbounded log continuously.
- The choice is about freshness: how fast does someone act on the number, and does acting faster change the outcome?
- Batch is cheaper, simpler to re-run and easy to reconcile; streaming buys seconds of latency with 24×7 cost and complexity.
- Micro-batch (Spark Structured Streaming) sits in the middle: seconds of latency, batch-style code.
- Streaming forces you to handle state, late events and checkpoints.
- Lambda runs a fast path plus a batch truth path; Kappa replays one stream for everything.
- Real platforms stream the hot paths (pricing, fraud, ETAs) and batch the reporting — stream to act, batch to account.
Test yourself
1. What fundamentally distinguishes the data a batch job reads from what a streaming job reads?
- Batch data is always smaller
- Batch data is bounded; streaming data is unbounded
- Streaming data is always JSON
- Batch data cannot contain timestamps
Show the answer
Batch data is bounded; streaming data is unbounded — A batch job reads something with an end (a day's partition) and can exit. A stream never ends, so the job must decide when a result is 'done'. Size is irrelevant — batch jobs routinely read terabytes.
2. Which Meghalaya Cabs use case most clearly needs streaming?
- Monthly driver payout statement
- Weekly ML training set
- Blocking a suspicious UPI payment before it is confirmed
- Yesterday's revenue by city
Show the answer
Blocking a suspicious UPI payment before it is confirmed — A fraud decision must happen in the payment path within seconds; a late answer is worthless. The others are consumed hours or days later.
3. Spark Structured Streaming's default execution model is best described as…
- Micro-batches triggered every few seconds
- True per-event processing
- A nightly batch
- A cron job over Kafka
Show the answer
Micro-batches triggered every few seconds — By default it processes newly arrived data in small batches on a trigger. That gives seconds of latency with batch-like semantics (a continuous mode exists but is experimental and rarely used).
4. A PM asks for a 'real-time' revenue dashboard. What should you do first?
- Provision Kafka and Flink
- Ask how often it is viewed and what decision changes with fresher data
- Build it in batch and don't tell them
- Refuse — revenue is a batch metric
Show the answer
Ask how often it is viewed and what decision changes with fresher data — 'Real-time' is a wish, not a requirement. The freshness SLA comes from who acts on the number and how fast. Often an hourly batch satisfies it at a fraction of the cost.
5. A streaming job computes rides per 5-minute window. A ride from 10:03 arrives at 10:09 after the 10:00–10:05 window was emitted. Without a watermark/late-data policy, what usually happens?
- Kafka rejects the event
- The window is automatically recomputed
- The event is dropped or lands in the wrong window, silently
- The job crashes
Show the answer
The event is dropped or lands in the wrong window, silently — Late events need an explicit policy: a watermark that keeps windows open for allowed lateness, or a correction path. Without one they vanish without an error.
6. Your team has one data engineer and no on-call rotation. The business needs data within 30 minutes. Best choice?
- A Flink cluster with exactly-once sinks
- Lambda architecture
- Daily batch
- A batch job every 15 minutes
Show the answer
A batch job every 15 minutes — A 15-minute batch meets a 30-minute SLA, is re-runnable, and needs no 24×7 operations. Streaming would buy latency nobody asked for and an on-call burden the team can't carry.
There are 9 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.
Start this lesson →