Data Engineering Quest

Tracks

1. 01🧭Foundation — Why Data Engineering Matters

   0/6 lessons

   The 80% problem, the modern stack, and the mental model.

   Before any code, the why. What data engineering actually is, why spreadsheets aren't it, what the modern Python data stack looks like in 2026, and the mental model that separates a script that ran once from a pipeline that runs every night.

   Lesson list (6)

   • 01
     Why Data Engineering Matters~12 min · foundation, career, mental-model
   • 02
     The Modern Python Data Stack (April 2026)~14 min · foundation, ecosystem, tools
   • 03
     Why Spreadsheets Break Down~10 min · foundation, anti-pattern, history
   • 04
     The DE Mental Model — Schemas, Lineage, Idempotency, Contracts~14 min · foundation, mental-model, principles
   • 05
     Pandas vs Polars vs DuckDB vs SQL — When to Reach for Each~13 min · foundation, tools, decision-framework
   • 06
     The Production Mindset — Rerunnable, Observable, Reviewable~12 min · foundation, production, mindset

   ✓Quiz · 4 questions→

2. 02🔢Frames — NumPy + Pandas Core

   0/7 lessons

   Arrays, DataFrames, and the mental shift from loops to vectorization.

   The two libraries every Python data person eventually masters. Why ndarray is fast, what broadcasting actually means, how Series and DataFrames relate, the .loc/.iloc trap, GroupBy split-apply-combine, joining tables, and the method-chaining style that makes transformations readable.

   Lesson list (7)

   • 01
     NumPy Arrays vs Python Lists — Why ndarray~13 min · numpy, ndarray, vectorization
   • 02
     Vectorization & Broadcasting — The Mental Shift~14 min · numpy, broadcasting, vectorization
   • 03
     Pandas DataFrame and Series — Labeled Tables~12 min · pandas, dataframe, series
   • 04
     Selection, Filtering, and the .loc / .iloc Trap~13 min · pandas, indexing, loc, iloc, gotcha
   • 05
     GroupBy — Split, Apply, Combine~14 min · pandas, groupby, aggregation
   • 06
     Joining Tables — merge() and the Join-Key Gotchas~13 min · pandas, merge, join, gotcha
   • 07
     Method Chaining and assign() — Readable Transforms~11 min · pandas, style, method-chaining

   ✓Quiz · 4 questions→

3. 03💾Storage — File Formats That Actually Matter

   0/6 lessons

   CSV, Excel, Parquet, Arrow, DuckDB, and how to pick.

   The format you write to disk decides how fast the next stage runs and how much it costs you. CSV is fine for humans and disastrous for analytics. Parquet is the analytics default. Arrow is the wire format. DuckDB makes any of them queryable as SQL. Pick on purpose, not by accident.

   Lesson list (6)

   • 01
     CSV — When It's Fine and When It Lies~12 min · csv, format, gotcha
   • 02
     Excel — The Type-Confusion Tax~9 min · excel, format, gotcha
   • 03
     Parquet — Columnar Binary, Why It Matters~13 min · parquet, columnar, format
   • 04
     Apache Arrow — The Glue Format~10 min · arrow, format, interop
   • 05
     DuckDB — SQL on Files, Without a Database Server~12 min · duckdb, sql, olap
   • 06
     Choosing a Format for Production~10 min · format, decision-framework

   ✓Quiz · 4 questions→

4. 04🚰Pipelines — ETL Patterns That Survive

   0/6 lessons

   Extract, transform, load — and why Polars exists.

   The patterns that make a pipeline a pipeline instead of a script: clean stage boundaries, explicit configuration, idempotent writes, and the discipline to design for the second run before you write the first. Plus when to reach for Polars instead of Pandas, and the real difference between batch and streaming.

   Lesson list (6)

   • 01
     Extract — Reading from APIs, Files, and Databases~13 min · etl, extract, api
   • 02
     Transform — Cleaning, Casting, Deriving~12 min · etl, transform
   • 03
     Load — Writing to Warehouses or Downstream Files~12 min · etl, load, idempotency
   • 04
     Polars — When Pandas Hits a Wall~13 min · polars, performance, lazy
   • 05
     Batch vs Streaming — The Real Tradeoff~11 min · batch, streaming, architecture
   • 06
     Idempotency in Practice~11 min · idempotency, production, patterns

   ✓Quiz · 4 questions→

5. 05🛡️Quality — Bad Data Dies at the Gate

   0/6 lessons

   Schemas, expectations, profiling, and the discipline of validation.

   How to make sure the data you're shipping downstream is actually what your contract says it is. Pandera for schema validation, Great Expectations for cross-team data tests, ydata-profiling for exploratory profiling, plus type-drift detection and structured logging that makes incidents debuggable.

   Lesson list (6)

   • 01
     Schema Validation with Pandera~13 min · validation, pandera, schema
   • 02
     Great Expectations — Cross-Team Data Tests~12 min · validation, great-expectations, data-contracts
   • 03
     Data Profiling — Knowing Your Data Before You Validate It~10 min · profiling, exploration, ydata-profiling
   • 04
     Catching Type Drift Before It Bites~10 min · validation, drift, monitoring
   • 05
     Null and Outlier Handling — Strategies, Not Defaults~11 min · nulls, outliers, imputation
   • 06
     Observability — Logging, Metrics, and the Run Log~11 min · observability, logging, monitoring

   ✓Quiz · 4 questions→

6. 06🎼Orchestration — Airflow, Dagster, Prefect

   0/5 lessons

   The systems that decide what runs, when, and what to do when something fails.

   Why you can't skip having an orchestrator forever. Airflow's DAG model. Dagster's asset-first approach. Prefect's Pythonic flows. The actual differences, the actual tradeoffs, and how to pick one for a real team.

   Lesson list (5)

   • 01
     Why You Need an Orchestrator (Eventually)~11 min · orchestration, fundamentals
   • 02
     Apache Airflow — DAGs, Operators, the Scheduler~14 min · airflow, orchestration
   • 03
     Dagster — Asset-First Orchestration~12 min · dagster, assets, orchestration
   • 04
     Prefect — The Pythonic Alternative~11 min · prefect, flows, orchestration
   • 05
     Choosing — The Practical Decision Framework~10 min · orchestration, decision

   ✓Quiz · 4 questions→

7. 07🗺️Modeling — How Analytics Tables Are Designed

   0/5 lessons

   Star schemas, slowly changing dimensions, dbt, and lineage.

   Why your analytics tables are shaped the way they are (or should be). Dimensional modeling, fact and dimension tables, star vs snowflake, slowly changing dimensions, dbt as the modern transformation layer, and lineage — knowing exactly where every number on a dashboard came from.

   Lesson list (5)

   • 01
     Dimensional Modeling — Facts and Dimensions~13 min · modeling, kimball, dimensional
   • 02
     Star vs Snowflake — When and Why~9 min · modeling, schema, kimball
   • 03
     Slowly Changing Dimensions — Type 1 vs Type 2~12 min · modeling, scd, history
   • 04
     dbt — Transformations as Code~13 min · dbt, modeling, transformations
   • 05
     Lineage — Knowing Where Every Number Came From~10 min · lineage, modeling, observability

   ✓Quiz · 4 questions→

8. 08🏭Production — The Real World

   0/6 lessons

   Backfills, contracts, monitoring, PII, cost — the things that decide whether a pipeline lasts.

   The transition from "works on my machine" to "runs every night for two years." Backfills and time-travel reads. Data contracts between teams. PII and access control. Cost — when DuckDB beats Snowflake (and vice versa). Monitoring that catches problems before stakeholders do. And a candid look at the data engineering career.

   Lesson list (6)

   • 01
     Monitoring Pipelines That Run at Night~12 min · monitoring, production, alerting
   • 02
     Backfills, Reruns, and Time-Travel Reads~11 min · backfill, production, idempotency
   • 03
     Data Contracts Between Teams~11 min · contracts, production, process
   • 04
     PII, Compliance, and Access Control~11 min · pii, security, compliance
   • 05
     Cost — When DuckDB Beats Snowflake (and Vice Versa)~10 min · cost, production, tools
   • 06
     The Data Engineering Career Reality~10 min · career, production, meta

   ✓Quiz · 4 questions→