🧱 Foundations · Chapter 1

What Data Engineering Actually Does

Apps create data. Data engineers make that data trustworthy, timely and usable at scale.

Study this lesson free in the app →

The basics

Data engineering is building and running the systems that move data from where it is created to where someone makes a decision with it — reliably, on time and at a sensible cost.

The problem it solves: an app's database is designed to run the business right now — book a ride, take a payment. It is bad at answering "how much did Pune earn last quarter?" and will slow down if you make it try. Someone has to copy, clean and reshape that data into places built for questions, without losing or duplicating any of it.

The mental model: plumbing. Data flows through five stages — source, ingestion, storage, transformation, serving — and the data engineer guarantees it arrives clean, complete and fresh, and notices within minutes when a pipe bursts.

Explained

A rider taps Book. In that one second, a payment must not double-charge, a driver must be found, and by tomorrow morning the finance team must be able to say exactly how much the city earned.

One of those is app engineering. The rest is data engineering.

Friday, 9 PM, heavy rain

Thousands of riders open the app at once. Drivers ping their location every few seconds. Requests, payments, cancellations and ratings pour in.

The platform must, at the same moment:

• show nearby drivers in under a second
• match rider and driver fast
• never charge anyone twice
• keep every event for audit
• let operations see demand by city within minutes
• feed tomorrow's pricing and ETA models

No single database is good at all six. So the work gets split across specialised systems — and the person who designs that split is the data engineer.

In plain words

A data engineer builds the plumbing.

Someone else decides what the water is for. You make sure it arrives — clean, on time, in the right pipe, without leaks, and you know within minutes when a pipe bursts.

The five stages, precisely

Every data platform on earth is some version of this chain:

1. Source — apps, databases, APIs, devices, files
2. Ingestion — move data out of the source without hurting it
3. Storage — a durable, cheap, immutable landing zone (the lake)
4. Transformation — clean, join, model, aggregate into business meaning
5. Serving — warehouse tables, dashboards, APIs, ML features

Wrapped around all five: orchestration (what runs when), quality (is it right), observability (do we know when it breaks), and governance (who may see it).

The shape to memorise

apps ──▶ operational DB  (run the business NOW)
  │
  └────▶ event stream ──▶ real-time consumers  (seconds)
              │
              └────▶ raw lake ──▶ transforms ──▶ warehouse ──▶ dashboards / ML
                     (forever)     (minutes-hours)   (answers)

Read it left to right as speed decreasing, meaning increasing. The left is fast and dumb. The right is slow and smart.

Why big platforms look like this

At small scale you can get away with one Postgres and a cron job. It breaks for three reasons, always in this order:

1. Contention — analytics queries slow down the app that pays the bills
2. Volume — location pings arrive faster than rows can be indexed
3. History — the business wants 7 years of facts; the app only wants today's state

Splitting operational from analytical is not architectural fashion. It is what happens after the first outage caused by a dashboard.

OLTP runs the business now. Kafka carries the facts. The lake keeps the history. The warehouse answers the questions.

The mistake beginners make

Pointing the dashboard at the production database.

It works for a month. Then one analyst writes a GROUP BY over 400 million rows on a Friday evening, and bookings start timing out. Now it is an incident, not a query.

Words to know

In two minutes

• Data engineering owns the path from where data is produced to where a decision is made.
• Every platform is five stages: source → ingestion → storage → transformation → serving.
• Around them sit orchestration, quality, observability and governance.
• Operational databases run the business now; the lake keeps raw history; the warehouse answers questions.
• One Postgres breaks in a fixed order: contention, then volume, then history.
• Data bugs return plausible wrong numbers, so checks like freshness and reconciliation matter more than in app code.
• Senior answers talk about guarantees — freshness, correctness, cost, recoverability — not tool lists.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.