Skip to content

Primary SAA curriculum

Analytics Data Lake On S3

A beginner-friendly AWS data lake by separating durable storage, metadata, transformation, occasional SQL, warehouse analytics, streaming ingestion, and dashboards.

6 min read

After this, you will understand

This scenario turns a list of analytics services into one clear pipeline: keep the source data, describe it, improve it, query it, and present the result.

Article guideprerequisites, mental models, and concepts

Article overview

intermediateCloudCertificationData

Three useful mental models

In plain terms

S3 keeps the data, Glue describes and prepares it, Athena or Redshift answers questions, and QuickSight presents the answers.

Decision pressure

Teams query production databases for analytics, dump unorganized files into S3, repeatedly scan inefficient raw files, or choose Athena and Redshift as if they solve the same workload.

Exam-ready model

Build the lake in layers, retain raw data that can be processed again, publish optimized curated data, and select each query service from how the data will be queried.

Think before reading

What is the simplest way to reduce Athena query cost?

Make each query read fewer bytes: organize files by common filters such as date, use compressed formats that group values by column, and select only the required columns.

Connected learning

These lessons add useful context to the current core lesson.
  1. 1On-Premises Migration To AWSAWS Scenario

A Data Lake Separates Analytics From Production

An online retailer wants to analyze orders, logs, click events, partner files, and spreadsheets together. Heavy scans and joins should not compete with customer transactions on the production database, and not every source fits one operational schema.

A data lake creates a separate analytics foundation. It is not one AWS service or simply a large bucket. S3 stores the data; metadata makes it discoverable; processing makes it trustworthy and efficient; query engines answer questions; presentation tools show the answers.

The Architecture In One View

The complete design keeps data flow separate from metadata and governance, and keeps dashboards away from the production database.

The smallest useful mental model has five jobs:

  • Keep the source — private S3 raw zone. Batch exports, logs, and partner files land independently of production. Preserve the original input long enough to audit and replay it after a transformation bug. S3 durability does not make the objects clean or understandable.
  • Describe it — Glue Data Catalog. Catalog tables record schema, format, partitions, and S3 locations. The bytes and rows remain in S3. A Glue crawler can infer and register metadata; it does not clean records or prove that an inferred schema has the right business meaning.
  • Prepare it — Glue ETL. Jobs validate, standardize, deduplicate, and write new curated data rather than overwriting the only raw copy. Compressed, column-oriented Parquet lets compatible queries skip unneeded columns. Date or another common filter can define partitions so a query skips unrelated S3 paths.
  • Ask questions — Athena or Redshift. Athena supplies serverless SQL directly over cataloged S3 data for occasional or ad hoc questions. Redshift supplies a data warehouse for repeated, complex, concurrent analytics with governed models and predictable performance requirements.
  • Present answers — QuickSight. QuickSight builds dashboards from an appropriate query engine or prepared dataset. It is the presentation layer, not storage, ETL, or query compute.

The trust path commonly becomes raw → curated → published. Raw preserves what arrived, curated holds validated and query-efficient data, and published datasets expose stable business definitions. Those names are conventions; separate locations, permissions, owners, and pipelines make the trust boundaries real.

Read deeper into raw, curated, and published trust zones

Follow One File From Arrival To Answer

  1. A daily order export lands in the private S3 raw zone. Analytics no longer scans the production database, and the original file remains replayable.
  2. A Glue job reads the raw file, applies the approved validation and transformation rules, and writes compressed Parquet into a curated date partition.
  3. The Glue Data Catalog describes the curated table, its columns, format, partitions, and S3 location. It points to the objects; it does not contain their rows.
  4. An analyst submits SQL to Athena. Athena reads the catalog metadata, uses the date filter to find relevant partitions, scans the required S3 objects and columns, and writes query results to a protected S3 results location.
  5. If the same logic becomes a frequent, complex workload for many BI users, curated data can feed a Redshift warehouse instead. QuickSight then presents the prepared answer without making every dashboard refresh rescan raw data.

The most important Athena performance and cost question is: how many bytes must this query read? Compression reduces bytes, Parquet can skip unused columns, partitions can skip unrelated paths, and narrow SQL avoids selecting unnecessary data. Excessive partitions and huge numbers of tiny files create their own overhead.

Read deeper into Parquet, compression, and partition pruning

Choose The Query And Ingest Paths From The Workload

Both services support SQL. Athena fits occasional questions over S3; Redshift fits repeated, concurrent warehouse workloads with predictable performance needs.

Do not choose from the word “SQL” alone. Ask where the data lives and how the workload behaves. Athena is the first thought for intermittent serverless SQL over S3. Redshift is the first thought for a modeled warehouse serving repeated complex queries and many concurrent BI users.

Likewise, do not add streaming because the destination is a data lake. A complete nightly file is batch input and can land in S3. For continuous events, choose Kinesis Data Streams when custom consumers need a retained stream, ordering within shards, or rereading during retention. Choose Amazon Data Firehose when the main job is managed buffering and delivery to S3 or another supported destination.

Read deeper into the Athena versus Redshift decision

SAA Recognition Signals And Traps

  • “Durable, low-cost repository for many data formats” points to an S3 data lake, but organization, metadata, governance, and processing are what keep it from becoming a data swamp.
  • “Persistent table, schema, partition, and location metadata for S3 data” points to the Glue Data Catalog. “Infer and populate that metadata” points to a Glue crawler.
  • “Serverless ETL or data integration” points to Glue jobs; a crawler does not perform that cleaning or transformation.
  • “Run serverless SQL directly over S3” points to Athena. “Reduce Athena cost” points first to reducing bytes scanned through columnar formats, compression, useful partitions, and narrow queries.
  • “Repeated complex BI with warehouse models and many concurrent users” points to Redshift. Not every dashboard or SQL requirement needs a warehouse.
  • “Multiple custom consumers process a retained real-time stream” points to Kinesis Data Streams; “managed streaming delivery to S3” points to Data Firehose.
  • “Dashboards and visualizations” points to QuickSight, but it still needs a query engine or prepared dataset beneath it.
  • Do not delete the raw source before the realistic replay window, expose raw sensitive data to dashboard users, or assume a catalog entry makes bad data trustworthy. Use Lake Formation when the requirement specifically calls for centralized fine-grained lake permissions across teams, accounts, tables, columns, or rows.

One-Minute Review

Keep the source, describe it, prepare and optimize it, ask questions with the right query engine, and present the answers.
sources -> private S3 raw zone keeps replayable originals
-> Glue Data Catalog describes locations and schemas
-> Glue ETL writes trusted, compressed, partitioned Parquet
-> Athena answers occasional S3 questions OR Redshift serves repeated warehouse BI
-> QuickSight presents the result

If you remember only one thing: keep storage, metadata, transformation, query, and presentation as separate jobs, then choose Athena or Redshift from the query workload—not from the word SQL.

Finished reading?

Your reading history is saved in this browser so you can continue later.

Recommended Next

Backup vs Replication Recovery DesignAWS Architecture Scenarios22 min read

This applies the foundation mental models to a real architecture decision instead of a service inventory.

Optional exploration

These links add context, but they do not replace the recommended next lesson.

Arcflow Plus is coming — review drills, research breakdowns, more AI. Get one email at launch.

One email at launch. That's it.