Data Engineering learning roadmap
Build one data platform end to end: ingest, transform, schedule, deploy and monitor it.
Start with short learning sessions here. Continue coding and project work on your laptop with the same GitHub account.
Start this pathWhat you’ll build
Payments data platform
A settlement platform built on synthetic payments. It loads, models, checks and recovers data. You build and run it on your laptop first, then deploy it to AWS with Terraform from GitHub Actions, using Free plan services.
Tap a milestone to see what you’ll build on your laptop.
1Ingestion Pipeline
Write a one-page design doc, then load the payments API into PostgreSQL in Docker with a saved cursor, and rerun it without creating duplicates.
What to show: docker-compose.yml, the ingestion script, its tests and a rerun that adds no duplicates.
2Warehouse Modeling
Design a star schema for payments, keep one dimension’s history with SCD Type 2, build a cumulative activity table and record why this design fits.
What to show: A bus matrix with the grain and design choice, the schema SQL, history tests and a cumulative table.
3Transform with dbt
Turn raw tables into tested staging models, a merchant snapshot and an incremental payments mart.
What to show: dbt models, key and relationship tests, one business-rule test and generated docs.
6Big Data with Spark
Run a PySpark job on a large ledger, compare it with a plain baseline and fix one slow shuffle or skew.
What to show: The Spark job, its explain plan, timings before and after, and the same totals as the baseline.
7Streaming with Kafka
Stream payment events through Kafka, survive a consumer crash and replay without duplicate rows.
What to show: Producer and consumer code, a crash and replay test, and notes on ordering and delivery.
Tools: Airflow, Terraform, AWS ops, CI + runbook.
What you'll build
Payments data platform
A settlement platform built on synthetic payments. It loads, models, checks and recovers data. You build and run it on your laptop first, then deploy it to AWS with Terraform from GitHub Actions, using Free plan services.
Data
Synthetic, so you can publish everything. The template includes a ledger generator and a mock settlement API with pagination, rate limits, failures and a late-batch mode.
One GitHub repo for the whole path. You connect it once, and each module adds a verified milestone.
You need: A laptop with 8 GB of RAM (16 GB makes Spark and Kafka easier), Docker, about 15 GB of free disk, and an AWS account on the Free plan from Run and Monitor onward. If AWS will not accept your card, a GitHub-only route gets you through the same checks.
Tap a milestone to see what you’ll build on your laptop.
1Ingestion Pipeline
Write a one-page design doc, then load the payments API into PostgreSQL in Docker with a saved cursor, and rerun it without creating duplicates.
What to show: docker-compose.yml, the ingestion script, its tests and a rerun that adds no duplicates.
2Warehouse Modeling
Design a star schema for payments, keep one dimension’s history with SCD Type 2, build a cumulative activity table and record why this design fits.
What to show: A bus matrix with the grain and design choice, the schema SQL, history tests and a cumulative table.
3Transform with dbt
Turn raw tables into tested staging models, a merchant snapshot and an incremental payments mart.
What to show: dbt models, key and relationship tests, one business-rule test and generated docs.
6Big Data with Spark
Run a PySpark job on a large ledger, compare it with a plain baseline and fix one slow shuffle or skew.
What to show: The Spark job, its explain plan, timings before and after, and the same totals as the baseline.
7Streaming with Kafka
Stream payment events through Kafka, survive a consumer crash and replay without duplicate rows.
What to show: Producer and consumer code, a crash and replay test, and notes on ordering and delivery.
Tools: Airflow, Terraform, AWS ops, CI + runbook.
A partner API failed and seven days of payments arrived late. Show how your platform loads them, checks the totals and recovers without counting anything twice.
Module by module
- 01
1Ingestion Pipeline
Plan the pipeline, then load data from APIs, files and databases into PostgreSQL, with reruns that never duplicate rows.
Write a one-page design doc, then load the payments API into PostgreSQL in Docker with a saved cursor, and rerun it without creating duplicates.
Plan it, then run PostgreSQL · Ingest from APIs and files · Incremental loads from databases
22hIngestion Pipeline
Write a one-page design doc, then load the payments API into PostgreSQL in Docker with a saved cursor, and rerun it without creating duplicates.
- 02
2Warehouse Modeling
Design facts and dimensions, keep history when records change, build cumulative tables and choose between the main warehouse designs.
Design a star schema for payments, keep one dimension’s history with SCD Type 2, build a cumulative activity table and record why this design fits.
Facts, dimensions and grain · History: SCDs and cumulative tables · Choosing a design
22hWarehouse Modeling
Design a star schema for payments, keep one dimension’s history with SCD Type 2, build a cumulative activity table and record why this design fits.
- 03
3Transform with dbt
Turn raw tables into clean, tested models with dbt, and run them separately in dev and prod.
Turn raw tables into tested staging models, a merchant snapshot and an incremental payments mart.
Layered models and sources · Tests, snapshots and incremental models · Macros, packages and environments
20hTransform with dbt
Turn raw tables into tested staging models, a merchant snapshot and an incremental payments mart.
- 04
4Scheduling with Airflow
Run the pipeline on a schedule, retry and alert on failures, and rerun past days safely.
Run ingestion, checks and dbt as an Airflow DAG with retries, an alert and a seven-day backfill.
DAGs, TaskFlow and connections · Sensors, retries and alerts · Backfills
22hScheduling with Airflow
Run ingestion, checks and dbt as an Airflow DAG with retries, an alert and a seven-day backfill.
- 05
5Cloud Data Lake
Build a data lake locally with Parquet and DuckDB, then define it on AWS in Terraform: S3 layers, a Glue catalog and Athena, checked in CI.
Build the lake locally with Parquet and DuckDB, then define it on AWS in Terraform: S3 layers, a Glue table, Athena and least-privilege roles, checked in CI.
Terraform, S3 and IAM · Lake layers, Parquet and partitions · Query the lake: DuckDB and Athena
26hCloud Data Lake
Build the lake locally with Parquet and DuckDB, then define it on AWS in Terraform: S3 layers, a Glue table, Athena and least-privilege roles, checked in CI.
- 06
6Big Data with Spark
Process large data with PySpark, read its query plan and fix one slow join or skewed key.
Run a PySpark job on a large ledger, compare it with a plain baseline and fix one slow shuffle or skew.
DataFrames and Spark SQL · Partitions, shuffles and Parquet · Faster joins and skew
24hBig Data with Spark
Run a PySpark job on a large ledger, compare it with a plain baseline and fix one slow shuffle or skew.
- 07
7Streaming with Kafka
Send events through Kafka, share them across consumers and recover from a crash without losing or duplicating data.
Stream payment events through Kafka, survive a consumer crash and replay without duplicate rows.
Topics, partitions and schemas · Consumer groups and offsets · Delivery guarantees and windows
24hStreaming with Kafka
Stream payment events through Kafka, survive a consumer crash and replay without duplicate rows.
- 08
8Run and Monitor
Run the pipeline on a schedule locally, then deploy it to AWS from GitHub Actions, stop it on bad data and alert when data goes stale.
Run the pipeline on a schedule locally, then deploy it to AWS from GitHub Actions: a scheduled Lambda, S3, and a freshness check at a live URL with alerts.
Deploy and schedule on AWS · Stale and bad data checks · Alerts and runbooks
26hRun and Monitor
Run the pipeline on a schedule locally, then deploy it to AWS from GitHub Actions: a scheduled Lambda, S3, and a freshness check at a live URL with alerts.
- 09
9Data Platform Capstone
Ship the whole platform with checks, lineage, alerts, a recovery runbook and docs another engineer can follow.
Release the platform live on AWS with green CI, lineage, alerts, a recovery runbook, an updated design doc and a walkthrough video.
Lineage and monitoring · Recovery runbook · Docs for the next engineer
24hData Platform Capstone
Release the platform live on AWS with green CI, lineage, alerts, a recovery runbook, an updated design doc and a walkthrough video.