Data Engineering learning roadmap

Build the infrastructure. Design robust pipelines, manage massive datasets, and ensure data quality and accessibility.

8 modules About 108 hoursFree curated resources

What you will learn

  1. 01

    Local Data Platform

    Run the project locally, then build an API-to-PostgreSQL pipeline that validates inputs and reruns without duplicate data.

    Docker & PostgreSQL · API ingestion · Idempotency & quality

    10h
  2. 02

    Data Modeling

    Design an analytics-ready star schema with declared grain, dependable keys, and a justified history strategy.

    Grain & dimensional models · Keys & SCDs · Warehouse performance

    12h
  3. 03

    dbt

    Build layered dbt transformations with tested sources, documented lineage, and separate development and production targets.

    Layered dbt models · Tests & lineage · Environment discipline

    12h
  4. 04

    Workflow Orchestration

    Schedule the platform as an observable Airflow DAG that retries safely, backfills correctly, and exposes failures.

    DAGs & data intervals · Retries & alerts · Safe backfills

    12h
  5. 05

    Cloud & Infrastructure as Code

    Deploy the batch platform with least-privilege identities, separated environments, and documented storage and query costs.

    Terraform & environments · IAM & secrets · Storage, warehouse & cost

    14h
  6. 06

    Spark — Advanced

    Use PySpark for a justified large-data workload, inspect its execution plan, and remove avoidable shuffle or skew.

    PySpark · Partitioning & shuffles · Performance tuning

    16h
  7. 07

    Real-time Streaming

    Build a recoverable Kafka event flow and explain its ordering, replay, consumer-group, and delivery guarantees.

    Topics · Consumer groups · Delivery semantics

    16h
  8. 08

    Production Readiness Capstone

    Ship the cumulative data platform with quality gates, lineage, observability, recovery procedures, and documentation another engineer can use.

    Data observability · Failure recovery · Technical documentation

    16h