🛡 Quality, Testing & Ops · Chapter 1

The six data quality dimensions

“The data is wrong” is not a bug report. “Ride fares are 4% duplicated since Tuesday” is — and the six dimensions are how you get from one to the other.

Study this lesson free in the app →

The basics

Data quality is how fit a dataset is for the decision someone will make with it. It is not one property but six, and each one breaks differently.

Why it exists: a finance analyst at Meghalaya Cabs says revenue “looks off”. Without a vocabulary you cannot even start: off how? Missing rides? Doubled rides? Yesterday's rides not arrived yet? The six dimensions turn a vague complaint into a specific, testable claim.

The mental model: think of a ride row as a witness statement. Is it true (accuracy)? Is anything left out (completeness)? Does it agree with other witnesses (consistency)? Did it arrive in time to matter (timeliness)? Is it in the right format (validity)? Was it said only once (uniqueness)?

Only validity and uniqueness can be checked from the table alone. The others need a second source — that asymmetry is the whole lesson.

Explained

Meghalaya Cabs' finance dashboard says yesterday's revenue was ₹38.4 lakh. The payments team says ₹36.9 lakh.

Someone is wrong. But how wrong — duplicated rides, missing payments, a late batch, a bad fare formula?

Each of those is a different dimension, a different test, and a different team to call. Naming the dimension is the first step of every data incident.

The wedding guest list

You are checking the guest list for your cousin's wedding in Shillong.

• A phone number has 7 digits → validity (wrong format)
• Aunty Rina is listed twice → uniqueness
• Twelve families have no address → completeness
• The list says 340 guests, the caterer's list says 310 → consistency
• The final list reaches the caterer the morning of the wedding → timeliness
• The list says Mr. Lyngdoh is vegetarian; he is not → accuracy

Notice the last one. Nothing on the paper looks wrong. You only find it by asking Mr. Lyngdoh. That is why accuracy is the hardest dimension: the record looks perfect and is still false.

The six, on the rides table

Validity — fare_inr >= 0, status IN ('requested','accepted','completed','cancelled'), pickup_ts is a timestamp, phone matches ^[6-9][0-9]{9}$.

Uniqueness — one row per ride_id. Duplicates usually come from retried loads.

Completeness — driver_id not null for completed rides; ~95,000 rows a day expected, not 41,000.

Consistency — sum of completed fares in rides equals sum of captured amounts in payments for the same day; a ride's city agrees with its driver's home city table.

Timeliness — the latest ride is less than 30 minutes old when the ops dashboard reads it.

Accuracy — the fare stored is the fare the rider was actually charged on their UPI statement.

What each dimension needs to be checked

dimension      what you need              example check
──────────────────────────────────────────────────────────────
validity       the table + rules          fare_inr >= 0
uniqueness     the table                  count(*) = count(distinct ride_id)
completeness   the table + expectation    null% driver_id < 0.1%
                                          rows >= 80% of 7-day avg
consistency    a SECOND table/system      sum(rides.fare) = sum(payments)
timeliness     a clock + an SLA           now() - max(event_ts) < 30 min
accuracy       ground truth (reality)     audit 200 rides vs UPI receipts

Read it top to bottom: each row needs more outside information than the one above. That is also roughly the order of how hard they are to automate.

One SQL query per dimension

-- validity: rows breaking a rule
SELECT count(*) FROM rides WHERE fare_inr < 0 OR fare_inr > 20000;

-- uniqueness: duplicate keys
SELECT ride_id, count(*) FROM rides GROUP BY ride_id HAVING count(*) > 1;

-- completeness: null rate on a required column
SELECT avg(CASE WHEN driver_id IS NULL THEN 1.0 ELSE 0 END)
FROM rides WHERE status = 'completed';

-- consistency: two systems, one number
SELECT (SELECT sum(fare_inr) FROM rides    WHERE ride_date = '2026-11-04' AND status='completed')
     - (SELECT sum(amount_inr) FROM payments WHERE pay_date  = '2026-11-04' AND state='captured') AS gap;

-- timeliness: how stale is the newest row?
SELECT now() - max(event_ts) AS lag FROM rides;

Accuracy has no query. It is a sample-and-audit process — which is exactly the point.

Wednesday, 10:40 AM

The ops head messages: “Bengaluru rides dropped 55% yesterday?”

You check in this order:

1. Timeliness — max(event_ts) for Bengaluru is 21:10 yesterday. The late-evening file never landed. Not a business drop.
2. Completeness — row count 41,200 vs 7-day average of 92,000. Confirms missing data, not fewer rides.
3. Uniqueness — no duplicates; fine.

Answer in ten minutes: “It's a late-arriving batch after 21:10, not a real drop. Backfill running, numbers correct by noon.”

Without the dimensions you would have spent an hour arguing about demand.

Gotchas

• Valid is not accurate. A fare of ₹250 passes every rule and can still be wrong. Range checks never prove truth.
• Null is not always incomplete. driver_id is legitimately null on a ride cancelled before acceptance. Write the rule per status, not per column.
• Consistency needs a defined time boundary. Rides by ride_date, payments by pay_date — a ride at 23:58 paid at 00:01 creates a gap that is not an error.
• Quality is per use. A 2-hour lag is fine for the monthly finance report and a disaster for the surge-pricing model.
• Measuring everything equally. Put strict checks on the 10 tables that feed money and decisions; light checks elsewhere.

Valid, unique, complete, consistent, timely, accurate — each needs more of the outside world than the last.

Validity and uniqueness: the table alone. Accuracy: reality itself.

Words to know

In two minutes

• Data quality is fitness for a specific use, measured on six dimensions — not one vague 'is it good'.
• Validity and uniqueness are checkable from the table itself: types, ranges, formats, duplicate keys.
• Completeness needs an expectation: how many rows or which columns should be there.
• Consistency needs a second table or system to compare against — rides vs payments, warehouse vs source.
• Timeliness needs a clock and an SLA: data that is correct but six hours late can still fail the dashboard.
• Accuracy is hardest — it needs ground truth, so it is sampled and audited, not fully automated.
• Every complaint maps to a dimension, and every dimension maps to a specific test you can write.

Test yourself

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.