◐ ai-teacher

☁️ AWS for Data · Chapter 1

AWS for data engineers: which service, and why

AWS has about nine services that could do your job. Picking between them is the interview question, and the answer is always about scale, latency and who maintains it.

Study this lesson free in the app →

The basics

The AWS data stack is the set of managed services you combine to collect, store, transform and query data in Amazon's cloud.

The problem it solves: you need somewhere cheap to keep terabytes of data and some way to run computation over it, without buying servers. AWS gives you a dozen ways to do the second part — and the real skill is picking the right one for the size, frequency and latency of the job.

The mental model: S3 is the floor. Data lives there as files, organised by a naming convention. Every other service is just a different way of bringing compute to those files: SQL with no cluster (Athena), managed Spark (Glue), Spark you control (EMR), a short function when something happens (Lambda), or a warehouse for constant queries (Redshift).

Words you must know 5

S3
object storage; your data lake is S3 plus folder-style prefixes
Serverless
no cluster to run; you pay per use (Athena per TB scanned, Glue per DPU-hour)
Partition prefix
dt=2026-03-14/ in the path, so engines can skip whole days
Parquet
a columnar, compressed file format; usually 5–10× less to scan than CSV
Small-files problem
millions of tiny objects that make every read slow and costly

Explained

"Where would you run this?"

EMR, Glue, Lambda, Athena, Redshift, Kinesis, Step Functions, MWAA, ECS.

The wrong answer is naming one. The right answer is naming the two you were choosing between, and what decided it.

S3 is the floor everything else stands on. Object storage: cheap, effectively unlimited, eleven nines of durability. Your data lake is S3 plus a naming convention.

Everything above it is a way of getting compute onto that data:

Athena — SQL straight over S3. No cluster. Pay per TB scanned. Glue — managed Spark, plus a catalog of what your S3 files mean. EMR — Spark/Hadoop clusters you actually control. Lambda — a function, up to 15 minutes, event-driven. Redshift — the warehouse: columnar, MPP, for repeated analytical queries.

How the pieces usually line up

Almost every AWS data platform is this shape. What varies is which box each service fills — and that is the decision you get asked to justify.

The choices that actually get asked

Athena vs Redshift. Athena is serverless and charges per TB scanned — ideal for occasional queries over a lake. Redshift keeps a cluster and is cheaper per query once the same data is queried constantly. Rule of thumb: exploratory and irregular → Athena. Dashboards hitting it all day → Redshift.

Glue vs EMR. Glue is managed Spark with no cluster to run; you pay per DPU-hour and give up version control and fine tuning. EMR gives you the cluster, the Spark version, spot instances and roughly half the cost at steady scale — and you now own it.

Lambda vs anything else. 15-minute ceiling, no state. Perfect for "a file landed, register it". Wrong for processing 200GB, however tempting the pricing looks.

S3 is where the money quietly goes. Not storage — requests and scanned bytes. A million tiny files costs far more to list and read than a thousand large ones, and Athena bills you per byte scanned, so file layout is a cost decision.

Three rules that pay for themselves immediately:

  1. Parquet, not CSV. Columnar and compressed — usually 5–10× less scanned.
  2. Partition by date in the prefix: s3://lake/rides/dt=2026-03-14/. Athena and Spark both prune on it.
  3. Compact small files. Target 128MB–1GB per file. The small-files problem is the most common cause of an inexplicably slow lake.

Spot instances on EMR cut compute cost by 60–80% for fault-tolerant batch work.

Millions of tiny files in S3. Slow, expensive, and it creeps up on you.

CSV in the lake. You pay to scan every column of every row, forever.

No partitioning in the prefix. Every query is a full scan.

Reaching for Lambda because it looks cheap, then fighting the 15-minute limit for a fortnight.

Redshift running 24/7 for a query an hour. That is an Athena workload.

Credentials in code. Use IAM roles. Every AWS interview has a version of this question.

S3 is the platform. Everything else is a way to put compute on it.

Parquet · partitioned by date · files of 128MB+.

Athena for irregular, Redshift for constant. Glue for managed, EMR for control and cost. Lambda when it finishes in 15 minutes and holds no state.

"Design a pipeline on AWS for 50GB of daily events."

"Events land in Kinesis or straight to S3 as raw JSON, partitioned by date in the prefix. A Glue job converts each day to Parquet in a curated bucket — that alone usually cuts scan cost by an order of magnitude — and registers the partition in the Glue Catalog.

For 50GB a day I'd start with Glue rather than EMR: no cluster to run, and at that size the cost difference doesn't justify owning one. I'd move to EMR with spot instances if it grew past a few hundred GB a day, where the saving becomes real.

Querying: Athena over the curated bucket for analysts, since it's serverless and the usage is irregular. Redshift only if we end up with dashboards hammering the same tables all day.

Orchestration with Step Functions or MWAA, and the important property throughout is that each day's job can be re-run and overwrite just its own partition."

Words to know

S3Object storage. The foundation of an AWS data lake.
AthenaServerless SQL over S3. Billed per TB scanned.
GlueManaged Spark plus a metadata catalog of what your S3 files contain.
EMRManaged Hadoop/Spark clusters you control and tune.
RedshiftAWS columnar MPP data warehouse.
Partition prefixFolder-style keys such as dt=2026-03-14/ that let engines skip data.
Small-files problemMany tiny objects making listing and reading slow and expensive.

In two minutes

Test yourself

1. Analysts run a handful of ad-hoc queries a week over 5TB in S3. Athena or Redshift?

  1. Redshift, it is faster
  2. Athena — serverless, and you only pay for the scans
  3. Neither, use EMR
  4. Lambda
Show the answer

Athena — serverless, and you only pay for the scans — Irregular usage does not justify a running cluster. Redshift wins when the same data is queried constantly.

2. What most reduces Athena cost?

  1. A bigger cluster
  2. Parquet plus date partitioning
  3. More IAM roles
  4. Reserved instances
Show the answer

Parquet plus date partitioning — Athena bills per byte scanned. Columnar format and partition pruning cut the bytes, often by 10×.

3. Why avoid millions of small files in S3?

  1. Storage cost
  2. Listing and read overhead dominates, making everything slow and expensive
  3. S3 has a file limit
  4. They cannot be encrypted
Show the answer

Listing and read overhead dominates, making everything slow and expensive — Per-request overhead and per-file open cost dwarf the actual data. Compact to 128MB–1GB.

4. A 10 MB file lands in S3 each hour and must be validated and registered in the catalog within seconds. What do you use?

  1. An EMR cluster polling S3 every minute
  2. A Redshift COPY on a schedule
  3. A Lambda triggered by the S3 ObjectCreated event
  4. A Glue crawler every hour
Show the answer

A Lambda triggered by the S3 ObjectCreated event — Small, event-driven, stateless and done in seconds — the textbook Lambda job. A crawler every hour misses the 'within seconds' requirement, and a polling EMR cluster pays for 24 hours of compute to do a few seconds of work.

5. An Athena table sits over 3 TB of raw CSV with no partition prefix. A query filters on one day. What does Athena scan?

  1. All 3 TB — with no partitions in the path it cannot skip any file
  2. Only that day's rows
  3. Only the columns named in SELECT
  4. Nothing — Athena caches the table
Show the answer

All 3 TB — with no partitions in the path it cannot skip any file — Without a partition prefix there is no way to know which files hold that day, and CSV has no column layout or statistics to skip with. At $5/TB that's ~$15 a query. Column pruning only works with columnar formats like Parquet.

6. S3 promises eleven nines of durability. Does that mean you can skip versioning or backups?

  1. Yes — data in S3 can never be lost
  2. Only if you store it in Glacier
  3. Only for files larger than 128 MB
  4. No — durability protects against disks failing, not against your own job deleting or overwriting data
Show the answer

No — durability protects against disks failing, not against your own job deleting or overwriting data — Durability is AWS not losing the bytes you stored. A buggy job that overwrites a partition, or someone running aws s3 rm --recursive, is a perfectly durable deletion. Versioning and lifecycle rules protect against that.

There are 8 questions, 3 puzzles and 5 recall cards for this lesson in the app — the recall cards come back on later days, just before you would forget.

Learn it so it stays. Read it, test yourself, then recall it on two different days. The app schedules that for you, free.

Start this lesson →