Eval Quest

Tracks

1. 01🎯Why Evals Matter

   0/9 lessons

   From vibes-based inspection to repeatable evidence

   The mindset shift from 'looks good to me' to 'here is the measurement that protects this behavior.' Every later track depends on this discipline taking root.

   Lesson list (9)

   • 01
     The Vibes-Based Problem~22 min · evals, quality, fundamentals
   • 02
     What Evaluation Means for LLMs~22 min · evals, definitions, fundamentals
   • 03
     Evaluation vs Testing~22 min · evals, testing, ci
   • 04
     The Evaluation Lifecycle~22 min · evals, lifecycle, process
   • 05
     Types of Evaluation~22 min · evals, taxonomy, fundamentals
   • 06
     What to Evaluate~22 min · evals, scope, fundamentals
   • 07
     The Baseline Problem~22 min · evals, baseline, fundamentals
   • 08
     The Cost of Not Evaluating~18 min · evals, risk, business
   • 09
     Evals in the AI Workflow~18 min · evals, workflow, ci

   ✓Quiz · 5 questions→

2. 02🗃️Datasets and Golden Cases

   0/7 lessons

   The dataset is the ceiling on every evaluation that follows

   Curate the inputs that represent your real workload. No grader, framework, or judge can rescue an eval suite built on the wrong data.

   Lesson list (7)

   • 01
     What Is a Test Dataset?~22 min · datasets, fundamentals
   • 02
     Golden Datasets~22 min · datasets, ground-truth, annotation
   • 03
     Dataset Size and Diversity~18 min · datasets, coverage, sampling
   • 04
     Synthetic Test Data Generation~22 min · datasets, synthetic, scale
   • 05
     Adversarial Examples~22 min · datasets, adversarial, edge-cases
   • 06
     Annotation Guidelines and Versioning~22 min · datasets, annotation, versioning
   • 07
     Storage Formats and Maintenance~18 min · datasets, infrastructure, ops

   ✓Quiz · 5 questions→

3. 03📊Deterministic Metrics

   0/6 lessons

   The cheap, fast, reproducible measurements that should run first

   Exact match, BLEU, ROUGE, BERTScore, regex, format checks, composites. The metrics you should always exhaust before reaching for an expensive LLM judge.

   Lesson list (6)

   • 01
     Exact Match and Contains~18 min · metrics, deterministic, fundamentals
   • 02
     BLEU and ROUGE~22 min · metrics, deterministic, translation, summarization
   • 03
     BERTScore: Semantic Similarity~22 min · metrics, semantic, embeddings
   • 04
     Perplexity, F1, and Classification Metrics~18 min · metrics, classification, perplexity
   • 05
     Regex and Format Validation~18 min · metrics, format, structured-output
   • 06
     Custom and Composite Scores~18 min · metrics, composite, weighted

   ✓Quiz · 5 questions→

4. 04⚖️LLM-as-Judge

   0/7 lessons

   Using AI to evaluate AI — power and pitfalls

   When deterministic graders cannot reach the question, an LLM judge can. But judges have biases, costs, and calibration problems. This track is how to wield them without being wielded by them.

   Lesson list (7)

   • 01
     The Concept: AI Evaluating AI~22 min · judges, llm-as-judge, fundamentals
   • 02
     Judge Prompt Design~22 min · judges, prompt-engineering
   • 03
     Pointwise vs Pairwise Evaluation~18 min · judges, comparison, calibration
   • 04
     Reference-Based vs Reference-Free~18 min · judges, reference
   • 05
     Multi-Criteria Evaluation~18 min · judges, multi-axis
   • 06
     Judge Bias and Calibration~22 min · judges, bias, calibration
   • 07
     Cost Management for Judge Calls~18 min · judges, cost, ops

   ✓Quiz · 5 questions→

5. 05🔧Frameworks and Platforms

   0/7 lessons

   promptfoo, DeepEval, Braintrust, lm-eval-harness, RAGAS, Inspect AI

   The tools you'll actually use in production. Each has a sweet spot — pick for the job, not for the brand.

   Lesson list (7)

   • 01
     promptfoo: CLI-First Eval Runner~22 min · frameworks, promptfoo, cli
   • 02
     DeepEval: Pytest-Native Python Framework~22 min · frameworks, deepeval, pytest, python
   • 03
     Braintrust: Observability and Eval Platform~22 min · frameworks, braintrust, observability, platform
   • 04
     lm-evaluation-harness (EleutherAI)~18 min · frameworks, lm-eval-harness, benchmarks
   • 05
     RAGAS: RAG-Specific Evaluation~22 min · frameworks, ragas, rag
   • 06
     Inspect AI: Agent and Safety Evaluation~18 min · frameworks, inspect-ai, agents, safety
   • 07
     Framework Selection Guide~18 min · frameworks, selection, decision

   ✓Quiz · 5 questions→

6. 06🏆Public Benchmarks

   0/7 lessons

   Reading the leaderboards without believing them

   MMLU, HumanEval, GSM8K, HellaSwag, Chatbot Arena, MTEB, ARC-AGI. What each measures, when it lies, and how to use them for model selection without confusing them with product evals.

   Lesson list (7)

   • 01
     MMLU: Massive Multitask Language Understanding~18 min · benchmarks, mmlu, knowledge
   • 02
     Code Benchmarks: HumanEval, MBPP, SWE-bench~18 min · benchmarks, code, humaneval, swe-bench
   • 03
     GSM8K and Math Reasoning~18 min · benchmarks, gsm8k, math, reasoning
   • 04
     HellaSwag and TruthfulQA~18 min · benchmarks, hellaswag, truthfulqa, common-sense
   • 05
     Chatbot Arena and MT-Bench~18 min · benchmarks, arena, mt-bench, human-eval
   • 06
     MTEB, ARC-AGI, and Modern Frontier Benchmarks~18 min · benchmarks, mteb, arc-agi, embeddings
   • 07
     Benchmark Limitations and Custom Benchmarks~18 min · benchmarks, limitations, custom

   ✓Quiz · 5 questions→

7. 07🔬System and Agent Evals

   0/6 lessons

   Evaluating compositions, not single calls

   RAG, agents, chatbots, code assistants, A/B production tests, regression suites. Real systems are pipelines; their evals must be too.

   Lesson list (6)

   • 01
     RAG Evaluation~22 min · systems, rag, retrieval
   • 02
     Agent Evaluation~22 min · systems, agents, tools, trajectories
   • 03
     Chatbot and Conversation Evaluation~18 min · systems, chatbot, multi-turn
   • 04
     Code Generation Evaluation~18 min · systems, code, execution
   • 05
     Online Evaluation: A/B Testing in Production~18 min · systems, ab-testing, production
   • 06
     Regression Testing and the Eval-Improvement Cycle~18 min · systems, regression, ci

   ✓Quiz · 5 questions→

8. 08🛡️Safety and Operations

   0/6 lessons

   Red teaming, calibration, cost-quality, building eval culture

   Where evaluation meets the rest of the company — security, reliability, cost discipline, and the team practices that make all of it stick.

   Lesson list (6)

   • 01
     Red Teaming~22 min · safety, red-team, adversarial
   • 02
     Consistency and Calibration~18 min · safety, consistency, calibration
   • 03
     Bias and Hallucination Detection~22 min · safety, bias, hallucination
   • 04
     Cost-Quality Tradeoff Analysis~18 min · safety, cost, tradeoff, routing
   • 05
     Meta-Evaluation and Building Eval Culture~18 min · safety, meta-eval, culture, team
   • 06
     Open Challenges in LLM Evaluation~18 min · safety, research, frontier

   ✓Quiz · 5 questions→