GPU Compute Quest

Tracks

1. 01🔧CUDA Setup — Get NVCC Talking

   0/5 lessons

   Driver, toolkit, NVCC: the three-layer stack you actually have to install

   CUDA C++ looks like plain C++ but adds three syntax extensions that change everything: function qualifiers, the triple-angle launch, and SIMT built-ins. Before any of it works, you need NVCC, a matching driver, and a Visual Studio or Linux toolchain to glue host and device code together. This track gets you from a blank Windows or Linux box to a green nvcc --version and a working hello kernel.

   Lesson list (5)

   • 01
     What CUDA C++ Really Is~12 min · cuda, nvcc, simt, qualifiers, intro
   • 02
     Driver, Toolkit, NVCC — The Three Layers~14 min · cuda, driver, toolkit, nvcc, compatibility
   • 03
     Setting Up on Windows + RTX 4090~14 min · cuda, windows, rtx, visual-studio, setup
   • 04
     Setting Up CUDA on WSL or Linux~14 min · cuda, wsl, linux, ubuntu, setup
   • 05
     Hello CUDA Example~12 min · cuda, hello, kernel, printf, first-program

   ✓Quiz · 5 questions→

2. 02🍎Metal Setup — Apple Silicon Path

   0/5 lessons

   Metal Shading Language, Xcode 26 toolchain, Unified Memory advantage

   Metal is Apple's CUDA — same SIMT mental model, different vocabulary, and one architectural advantage that genuinely changes how you write code: Unified Memory. Same DRAM physically backs the CPU and GPU, so cudaMemcpy disappears. This track installs the Metal toolchain, ships a hello kernel in pure Swift, and gives you a CUDA ↔ Metal cheat sheet so you can read either codebase in five minutes.

   Lesson list (5)

   • 01
     What Metal Shading Language Is~12 min · metal, msl, apple-silicon, kernel, intro
   • 02
     Why Apple Silicon Unified Memory Changes Everything~14 min · metal, unified-memory, apple-silicon, architecture
   • 03
     Setting Up on macOS~10 min · metal, macos, xcode, setup, command-line
   • 04
     Hello Metal Example~14 min · metal, hello, kernel, swift, first-program
   • 05
     CUDA ↔ Metal Cheat Sheet~10 min · metal, cuda, cheatsheet, comparison

   ✓Quiz · 5 questions→

3. 03📐Linear Algebra for AI

   0/4 lessons

   Why every model collapses to dot product + GEMM — and how the roofline keeps you honest

   Every neural net at compile time becomes a sequence of dot products and matrix multiplies. That's not a metaphor — it's literally what XLA, TensorRT, and torch.compile lower to. Once you internalize that, choosing kernels and reading roofline plots becomes mechanical. This track builds the AXPY/GEMV/GEMM mental ladder, explains why GPUs love Level-3 BLAS, and shows you the arithmetic intensity that decides memory-bound vs compute-bound.

   Lesson list (4)

   • 01
     Why AI Collapses to Two Primitives~12 min · linalg, dot-product, gemm, primitive
   • 02
     Mental Pictures: Vector, Matrix, Tensor~10 min · linalg, vector, matrix, tensor, intuition
   • 03
     AXPY, GEMV, GEMM — The Only Three Ops~12 min · linalg, blas, axpy, gemv, gemm
   • 04
     Memory Layout & Roofline~12 min · linalg, row-major, column-major, roofline, coalesced

   ✓Quiz · 5 questions→

4. 04🐜Hello GPU Reinforced

   0/4 lessons

   Same hello kernel, deeper understanding — ant colonies, warps, divergence

   You already saw a hello kernel run. This track replays it with the proper mental model: the GPU is an ant colony, blocks are squads, warps march in lockstep. Then we expand it to a configurable battalion that lets you feel block size, grid size, and warp divergence in your fingertips. By the end, the launch geometry stops feeling magical and starts feeling like a sergeant calling formations.

   Lesson list (4)

   • 01
     The Ant Colony Mental Model~10 min · gpu, mental-model, warp, block, hierarchy
   • 02
     Hello CUDA Revisited~12 min · cuda, hello, battalion, global-id
   • 03
     Hello Metal Revisited~12 min · metal, hello, battalion, swift
   • 04
     Common GPU Parallelism Gotchas~12 min · gpu, gotchas, warp-divergence, scheduler

   ✓Quiz · 5 questions→

5. 05➡️Vectors — The First Real Workload

   0/5 lessons

   Embarrassingly parallel arithmetic, the 9-step GPU workflow, VRAM capacity zones

   Vector add is the smallest workload that reveals everything: the host/device memory split, the launch grid, the bandwidth ceiling, the VRAM cliff. We walk the 9-step workflow once for CUDA, once for Metal, then variations (sub/mul/div) that look identical but expose latency differences. The track closes by showing how numpy / PyTorch / MLX hide all this — and why you should still know what's underneath.

   Lesson list (5)

   • 01
     What Vectors Are & Why GPUs Love Them~10 min · vectors, embarrassingly-parallel, intro
   • 02
     The 9-Step GPU Workflow~10 min · workflow, lifecycle, memory-management
   • 03
     Vector Addition: CUDA & Metal~12 min · vectors, cuda, metal, first-real-kernel
   • 04
     Operator Variations & Capacity Zones~10 min · vectors, operators, vram, capacity
   • 05
     Where numpy / PyTorch / MLX Hide These Primitives~12 min · vectors, numpy, pytorch, mlx, abstraction

   ✓Quiz · 5 questions→

6. 06🔢Matrices — Where Tiling Earns Its Keep

   0/5 lessons

   Coalesced access, reduction, naive vs tiled GEMM, the matrix_ops CLI

   Matrices are where 'just throw threads at it' stops working. Coalesced reads matter. Tiling matters. Reduction trees matter. This track builds row-major intuition, walks reduction patterns from warp-shuffle to block-tree, and finishes with the dramatic 9× speedup from naive to tiled GEMM. The matrix_ops CLI we ship is the lab bench you'll actually run on your machine.

   Lesson list (5)

   • 01
     Matrix Layout & Coalesced Access~12 min · matrices, row-major, coalesced, leading-dimension
   • 02
     Reduction — The Heartbeat of Matrix Math~14 min · matrices, reduction, warp-shuffle, softmax
   • 03
     Matrix Add & 2-D Indexing~10 min · matrices, 2d-launch, indexing, leading-dimension
   • 04
     Naive GEMM vs Tiled GEMM~16 min · matrices, gemm, tiling, shared-memory
   • 05
     Matrix Ops CLI — Seven Personalities~12 min · matrices, cli, benchmark, tooling

   ✓Quiz · 5 questions→

7. 07📚BLAS 101 — Stop Hand-Rolling

   0/5 lessons

   What BLAS is, cuBLAS + MPS cheat sheets, calling BLAS from Python, precision playbook

   Vendor BLAS routines have decades of tuning baked in. Even your best hand-rolled GEMM hits ~5% of peak; cuBLAS hits 80%+. This track teaches when to call BLAS, how to call it from CUDA, Metal, and Python, and the precision tradeoff (FP32 vs FP16/BF16 vs Int8/Int4) that decides whether your model fits at all on a given GPU.

   Lesson list (5)

   • 01
     What Is BLAS~10 min · blas, history, intro
   • 02
     cuBLAS & MPS Cheat Sheets~12 min · cublas, mps, gemm, alpha-beta
   • 03
     Hands-On BLAS Demo~12 min · blas, benchmark, comparison
   • 04
     Calling BLAS from Python~12 min · blas, numpy, pytorch, mlx, python
   • 05
     Precision Playbook on Apple Silicon~12 min · precision, fp16, bf16, int8, int4, apple-silicon

   ✓Quiz · 5 questions→

8. 08🔬BLAS Deep Dive — What Vendors Actually Do

   0/5 lessons

   Five tuning pillars, Nsight Compute, Apple Instruments, modern GEMM tiling

   Open the hood. Vendor BLAS hits 90% of peak through five pillars: blocking, micro-kernels, split-K, heuristic algo pickers, epilogue fusion. We read real Nsight Compute reports, real Apple Instruments traces, and decode the three-tier tile pyramid that both ecosystems converge on. By the end you can look at a profiler row and say 'I know exactly which pillar is breaking.'

   Lesson list (5)

   • 01
     Library Secrets at a Glance~12 min · blas, tiling, tensor-core, split-k, epilogue-fusion
   • 02
     Nsight Compute Profiling~14 min · nsight, profiling, occupancy, cuda
   • 03
     Apple Instruments Profiling~14 min · instruments, metal, profiling, apple-silicon
   • 04
     Modern GEMM Architecture~14 min · gemm, tiling, tensor-core, matrix-coprocessor, architecture
   • 05
     Admin Cheat Sheet~12 min · blas, ops, checklist, production

   ✓Quiz · 5 questions→