Transformer Quest

Tracks

1. 01⚡Why Transformers

   0/10 lessons

   Parallel sequence modeling and the bottleneck it broke

   What problem the Transformer actually solved, why attention beats recurrence, and where the 2017 design fits in modern AI's lineage.

   Lesson list (10)

   • 01
     Sequence Modeling Before Transformers: RNNs and LSTMs~22 min · history, rnn, lstm, background
   • 02
     The Parallelization Problem: Why GPUs Hated RNNs~18 min · parallelism, gpu, rnn
   • 03
     Long-Range Dependencies and the Vanishing Gradient~18 min · long-range, vanishing-gradient, rnn
   • 04
     The Attention Insight: Direct Pairwise Access~20 min · attention, intuition, core-idea
   • 05
     The 2017 Paper: 'Attention Is All You Need'~16 min · history, paper, vaswani-2017
   • 06
     Impact: From Translation to GPT, BERT, and the Modern LLM Stack~16 min · history, gpt, bert, llm
   • 07
     Beyond Text: Vision, Audio, Biology~14 min · vision, audio, biology, multimodal
   • 08
     The Scaling Hypothesis and What It Got Right~16 min · scaling, chinchilla, kaplan
   • 09
     Encoder, Decoder, Encoder-Decoder — Three Shapes, Three Roles~14 min · encoder, decoder, architecture-shapes
   • 10
     Roadmap: What Tracks 2-8 Will Build~8 min · roadmap, course-overview

   ✓Quiz · 4 questions→

2. 02🔤Tokenization

   0/12 lessons

   How text becomes integers

   BPE, WordPiece, SentencePiece, and the practical consequences — why GPT can't count letters, why non-English costs more per query.

   Lesson list (12)

   • 01
     Why Tokenization? The Bridge from Text to Tensors~12 min · tokenization, fundamentals
   • 02
     Character vs Word vs Subword: Three Strategies, One Winner~16 min · bpe, subword, vocabulary
   • 03
     Byte-Pair Encoding (BPE) Step by Step~18 min · bpe, algorithm, gpt
   • 04
     WordPiece: BERT's Tokenizer~12 min · wordpiece, bert
   • 05
     SentencePiece: Language-Agnostic Tokenization~14 min · sentencepiece, unigram, multilingual
   • 06
     Vocabulary Size: 32K to 262K and the Tradeoffs~12 min · vocabulary, scaling, multilingual
   • 07
     Special Tokens: Scaffolding for the Model~12 min · special-tokens, chat-format
   • 08
     Token IDs and the Embedding Lookup~10 min · token-ids, embedding
   • 09
     Tokenizer Quirks: Why Your LLM Can't Count R's in 'strawberry'~12 min · limitations, edge-cases
   • 10
     Multilingual Tokenization and the Efficiency Gap~10 min · multilingual, korean, cjk
   • 11
     tiktoken and Hugging Face Tokenizers — Tools of the Trade~10 min · tooling, tiktoken, huggingface
   • 12
     Token Economics: Why Pricing Is Per Token~10 min · pricing, economics, inference

   ✓Quiz · 4 questions→

3. 03📐Embeddings & Position

   0/12 lessons

   Token IDs to dense vectors with order

   How models turn integers into geometry, and how position is injected — sinusoidal, learned, RoPE, ALiBi, and YaRN-scaled extension.

   Lesson list (12)

   • 01
     Word Embeddings: Token IDs Become Dense Vectors~12 min · embeddings, fundamentals
   • 02
     Embedding Dimension (d_model): 768 to 12,288~10 min · d-model, scaling, capacity
   • 03
     Semantic Arithmetic: king − man + woman ≈ queen~10 min · semantics, intuition
   • 04
     The Position Problem: Attention Is Permutation-Invariant~12 min · positional, permutation-invariance
   • 05
     Sinusoidal Positional Encoding (Original Paper)~14 min · sinusoidal, positional
   • 06
     Learned Positional Embeddings (BERT, GPT-2)~10 min · learned-pos, bert, gpt-2
   • 07
     RoPE: Rotary Position Embeddings (the Modern Standard)~18 min · rope, rotation, modern
   • 08
     ALiBi: Attention with Linear Biases~10 min · alibi, extrapolation
   • 09
     Context Extension: Why It's Hard, and What Works~14 min · long-context, extension
   • 10
     YaRN and NTK-Aware Scaling: How Llama 3 Got to 128K~12 min · yarn, ntk, long-context
   • 11
     Input Pipeline: Putting It All Together~8 min · input-pipeline, summary
   • 12
     Visualizing Embeddings: Make the Geometry Real~10 min · visualization, interpretability

   ✓Quiz · 4 questions→

4. 04🎯Attention Mechanism

   0/13 lessons

   Q, K, V — and the engineering that scales them

   Scaled dot-product attention, multi-head, causal masking, KV cache, GQA, Flash Attention. The deepest track — read it twice.

   Lesson list (13)

   • 01
     Intuition: Attention as Soft Dictionary Lookup~12 min · intuition, attention, qkv
   • 02
     Q, K, V Projections: Where the Asymmetry Comes From~14 min · qkv, projections
   • 03
     Scaled Dot-Product Attention: Q · K^T → softmax → · V~14 min · scaled-dot-product, core
   • 04
     Why Divide by √d_k? The Scaling Factor Explained~10 min · scaled-dot-product, softmax
   • 05
     Multi-Head Attention: Parallel Subspaces~16 min · multi-head, mha
   • 06
     Head Count Across Models: From 8 to 96~8 min · heads, model-survey
   • 07
     Self-Attention vs Cross-Attention~10 min · self-attention, cross-attention
   • 08
     Causal Masking: How Decoder-Only Models Cheat-Proof Their Training~12 min · causal-mask, decoder
   • 09
     Attention Patterns: What Heads Actually Learn~12 min · interpretability, induction-heads
   • 10
     KV-Cache: Why Generation Doesn't Recompute Everything~14 min · kv-cache, inference
   • 11
     O(n²) Attention Complexity: The Long-Context Cliff~10 min · complexity, long-context
   • 12
     Efficient Attention: Flash Attention, Sliding Window, Sparse~16 min · flash-attention, sliding-window, sparse
   • 13
     GQA and MQA: Shrinking the KV Cache~12 min · gqa, mqa, memory

   ✓Quiz · 5 questions→

5. 05🧱The Transformer Block

   0/12 lessons

   Pre-LN, RMSNorm, SwiGLU, residual — the unit cell

   How attention plus FFN plus normalization plus residual become the building block stacked 80 times. Plus MoE and GQA in context.

   Lesson list (12)

   • 01
     Layer Normalization: Pre-LN vs Post-LN~12 min · layer-norm, stability
   • 02
     The Feed-Forward Network: Position-Wise Compute~12 min · ffn, fundamentals
   • 03
     Activations: ReLU → GELU → SwiGLU~12 min · activations, swiglu
   • 04
     Residual Connections: Why You Can Stack 80 Layers~10 min · residual, depth
   • 05
     The Full Block: Putting Everything Together~12 min · block, synthesis
   • 06
     Depth vs Width: Where to Spend Parameters~10 min · depth, width, scaling
   • 07
     Parameter Count Calculator: Reading Any Model Card~10 min · params, model-card
   • 08
     The Four Numbers: d_model, d_ff, n_heads, n_layers~8 min · hyperparameters, synthesis
   • 09
     Dropout and Regularization (and Why It's Out of Fashion)~8 min · dropout, regularization
   • 10
     The Output Head: From Hidden States to Logits~8 min · output-head, weight-tying
   • 11
     Mixture of Experts: Sparse Compute, Dense Knowledge~14 min · moe, mixtral, deepseek
   • 12
     GQA in the Block: A Concrete Example~8 min · gqa, block-summary

   ✓Quiz · 4 questions→

6. 06🔥Training & Generation

   0/12 lessons

   Loss, schedules, decoding, and alignment

   Pretraining objectives, mixed precision, decoding strategies, RLHF / DPO / GRPO / Constitutional AI — the full training and serving stack.

   Lesson list (12)

   • 01
     Pre-Training Objectives: Next-Token vs Masked LM~12 min · pretraining, objectives
   • 02
     Autoregressive Generation: One Token at a Time~10 min · generation, decoding
   • 03
     Training Data: Trillions of Tokens~8 min · data, scale
   • 04
     Loss Function: Cross-Entropy~8 min · loss, cross-entropy
   • 05
     Learning Rate: Warmup + Cosine Decay~10 min · lr, schedule
   • 06
     Batch Size and Gradient Accumulation~10 min · batch, gradient-accumulation
   • 07
     Mixed Precision: BF16, FP16, FP8~10 min · mixed-precision, bf16, fp8
   • 08
     Decoding Strategies: Greedy, Beam, Top-k, Top-p~12 min · decoding, sampling
   • 09
     Temperature: How Much Should the Model Improvise?~8 min · temperature, sampling
   • 10
     Repetition Penalty and Stop Conditions~8 min · repetition, stop-sequences
   • 11
     RLHF and DPO: Aligning Models to Be Helpful~16 min · rlhf, dpo, alignment
   • 12
     GRPO and Constitutional AI: Alignment Beyond DPO~14 min · grpo, constitutional-ai, reasoning

   ✓Quiz · 4 questions→

7. 07🌳Model Families

   0/11 lessons

   GPT, Llama, Mistral, Claude, Gemini, and the alternatives

   How each major model family fits the architectural template, and where state-space alternatives (Mamba, RWKV) actually compete.

   Lesson list (11)

   • 01
     Encoder-Only: BERT, RoBERTa, and the Embedding Era~12 min · bert, encoder-only, embeddings
   • 02
     Decoder-Only: GPT to Llama, the Universal Shape~12 min · gpt, llama, decoder-only
   • 03
     Encoder-Decoder: T5, BART, Whisper~10 min · t5, bart, whisper, encoder-decoder
   • 04
     The GPT Lineage: GPT-1 to GPT-5~10 min · gpt, openai, history
   • 05
     The Llama Lineage: Open-Weight Frontier~10 min · llama, meta, open-weight
   • 06
     Mistral Family: Efficient European Open Weights~10 min · mistral, mixtral, moe
   • 07
     Gemini Family: Multimodal-First from Google~10 min · gemini, google, multimodal
   • 08
     Claude Family: Constitutional AI in Practice~10 min · claude, anthropic, alignment
   • 09
     Small Models: Phi, Gemma, Qwen, and the Cost-Quality Frontier~12 min · phi, gemma, qwen, small-models
   • 10
     Vision Transformers and Multimodal Models~12 min · vit, multimodal, vision
   • 11
     State-Space Alternatives: Mamba, RWKV, and What Comes Next~12 min · mamba, rwkv, ssm

   ✓Quiz · 4 questions→

8. 08🔧Practical Understanding

   0/12 lessons

   Reading model cards, sizing GPUs, picking the right tool

   Memory math, quantization, benchmarks, scaling laws in 2026, inference optimization, and how to keep up.

   Lesson list (12)

   • 01
     Reading a Model Card: The Five Questions~10 min · model-card, evaluation
   • 02
     Parameter Count Intuition: 1B to 1T~8 min · params, intuition
   • 03
     Memory Math: Bytes per Parameter~10 min · memory, deployment
   • 04
     Quantization: Q4, Q5, Q8 in Practice~12 min · quantization, deployment
   • 05
     Context Window Implications~10 min · context-window, long-context
   • 06
     Benchmark Literacy: MMLU to GPQA to HLE~12 min · benchmarks, evaluation
   • 07
     Scaling Laws Today: Beyond Chinchilla~10 min · scaling-laws, chinchilla
   • 08
     Emergent Capabilities and Test-Time Compute~10 min · emergence, test-time-compute
   • 09
     The Chinchilla Trap: Optimize for Where the Money Is~8 min · chinchilla, production
   • 10
     Inference Optimization: Speculative Decoding, PagedAttention, Continuous Batching~14 min · inference, vllm, speculative
   • 11
     Prompting vs RAG vs Fine-Tuning: When to Use Which~12 min · prompting, rag, fine-tuning
   • 12
     Staying Current: A Working Reading Loop~8 min · meta, research-tracking

   ✓Quiz · 4 questions→