Eval Quest

Tracks

1. 01🎯왜 eval이 중요한가

   0/9 lessons

   vibe 기반 검사에서 재현 가능한 증거로

   '느낌상 괜찮네'에서 '이 동작을 보호하는 측정이 여기 있다'로의 사고 전환. 이후 모든 트랙은 이 규율이 박혔다고 전제하고 시작해.

   Lesson list (9)

   • 01
     vibe 기반 문제~22 min · evals, quality, fundamentals
   • 02
     LLM 에서 evaluation 이 뜻하는 것~22 min · evals, definitions, fundamentals
   • 03
     Evaluation vs Testing~22 min · evals, testing, ci
   • 04
     Evaluation Lifecycle~22 min · evals, lifecycle, process
   • 05
     Eval 의 종류~22 min · evals, taxonomy, fundamentals
   • 06
     뭘 평가할 거야~22 min · evals, scope, fundamentals
   • 07
     Baseline 문제~22 min · evals, baseline, fundamentals
   • 08
     Eval 안 하는 비용~18 min · evals, risk, business
   • 09
     AI Workflow 안의 eval~18 min · evals, workflow, ci

   ✓퀴즈 · 5 문제→

2. 02🗃️Dataset 과 Golden Case

   0/7 lessons

   Dataset 은 이후 모든 evaluation 의 천장

   실제 workload 를 대표하는 input 을 curate 해. grader, framework, judge 가 아무리 좋아도 잘못된 데이터 위 eval suite 는 못 살려.

   Lesson list (7)

   • 01
     Test Dataset 이 뭐야?~22 min · datasets, fundamentals
   • 02
     Golden Dataset~22 min · datasets, ground-truth, annotation
   • 03
     Dataset Size 와 Diversity~18 min · datasets, coverage, sampling
   • 04
     Synthetic Test Data Generation~22 min · datasets, synthetic, scale
   • 05
     Adversarial Example~22 min · datasets, adversarial, edge-cases
   • 06
     Annotation Guideline 과 Versioning~22 min · datasets, annotation, versioning
   • 07
     Storage Format 과 Maintenance~18 min · datasets, infrastructure, ops

   ✓퀴즈 · 5 문제→

3. 03📊Deterministic Metric

   0/6 lessons

   먼저 돌려야 할 싸고 빠르고 재현 가능한 측정

   Exact match, BLEU, ROUGE, BERTScore, regex, format check, composite. 비싼 LLM judge 부르기 전에 다 써먹어야 할 metric 들이야.

   Lesson list (6)

   • 01
     Exact Match 와 Contains~18 min · metrics, deterministic, fundamentals
   • 02
     BLEU 와 ROUGE~22 min · metrics, deterministic, translation, summarization
   • 03
     BERTScore: Semantic Similarity~22 min · metrics, semantic, embeddings
   • 04
     Perplexity, F1, Classification Metric~18 min · metrics, classification, perplexity
   • 05
     Regex 와 Format Validation~18 min · metrics, format, structured-output
   • 06
     Custom 과 Composite Score~18 min · metrics, composite, weighted

   ✓퀴즈 · 5 문제→

4. 04⚖️LLM-as-Judge

   0/7 lessons

   AI 로 AI 평가 — 힘과 함정

   Deterministic grader 가 닿지 못하는 질문에 LLM judge 가 닿아. 하지만 judge 는 편향, 비용, calibration 문제 있어. 이 트랙은 judge 에 휘둘리지 않으면서 휘두르는 법.

   Lesson list (7)

   • 01
     개념: AI 가 AI 평가~22 min · judges, llm-as-judge, fundamentals
   • 02
     Judge Prompt Design~22 min · judges, prompt-engineering
   • 03
     Pointwise vs Pairwise Evaluation~18 min · judges, comparison, calibration
   • 04
     Reference-Based vs Reference-Free~18 min · judges, reference
   • 05
     Multi-Criteria Evaluation~18 min · judges, multi-axis
   • 06
     Judge 편향과 Calibration~22 min · judges, bias, calibration
   • 07
     Judge Call 비용 관리~18 min · judges, cost, ops

   ✓퀴즈 · 5 문제→

5. 05🔧Framework 와 Platform

   0/7 lessons

   promptfoo, DeepEval, Braintrust, lm-eval-harness, RAGAS, Inspect AI

   Production 에서 실제로 쓸 도구들. 각각 sweet spot 있어 — brand 가 아니라 일에 맞춰 골라.

   Lesson list (7)

   • 01
     promptfoo: CLI-First Eval Runner~22 min · frameworks, promptfoo, cli
   • 02
     DeepEval: Pytest-Native Python Framework~22 min · frameworks, deepeval, pytest, python
   • 03
     Braintrust: Observability 와 Eval Platform~22 min · frameworks, braintrust, observability, platform
   • 04
     lm-evaluation-harness (EleutherAI)~18 min · frameworks, lm-eval-harness, benchmarks
   • 05
     RAGAS: RAG-Specific Evaluation~22 min · frameworks, ragas, rag
   • 06
     Inspect AI: Agent 와 Safety Evaluation~18 min · frameworks, inspect-ai, agents, safety
   • 07
     Framework 선택 가이드~18 min · frameworks, selection, decision

   ✓퀴즈 · 5 문제→

6. 06🏆Public Benchmark

   0/7 lessons

   Leaderboard 안 믿으면서 읽기

   MMLU, HumanEval, GSM8K, HellaSwag, Chatbot Arena, MTEB, ARC-AGI. 각각 뭘 측정하고, 언제 거짓말하고, model 선택에 어떻게 — product eval 과 헷갈리지 않고 — 쓰는지.

   Lesson list (7)

   • 01
     MMLU: Massive Multitask Language Understanding~18 min · benchmarks, mmlu, knowledge
   • 02
     Code Benchmark: HumanEval, MBPP, SWE-bench~18 min · benchmarks, code, humaneval, swe-bench
   • 03
     GSM8K 와 Math Reasoning~18 min · benchmarks, gsm8k, math, reasoning
   • 04
     HellaSwag 와 TruthfulQA~18 min · benchmarks, hellaswag, truthfulqa, common-sense
   • 05
     Chatbot Arena 와 MT-Bench~18 min · benchmarks, arena, mt-bench, human-eval
   • 06
     MTEB, ARC-AGI, Modern Frontier Benchmark~18 min · benchmarks, mteb, arc-agi, embeddings
   • 07
     Benchmark 한계와 Custom Benchmark~18 min · benchmarks, limitations, custom

   ✓퀴즈 · 5 문제→

7. 07🔬System 과 Agent Eval

   0/6 lessons

   Single call 이 아니라 composition 평가

   RAG, agent, chatbot, code assistant, A/B production test, regression suite. 실제 system 은 pipeline; 그 eval 도 그래야.

   Lesson list (6)

   • 01
     RAG Evaluation~22 min · systems, rag, retrieval
   • 02
     Agent Evaluation~22 min · systems, agents, tools, trajectories
   • 03
     Chatbot 과 Conversation Evaluation~18 min · systems, chatbot, multi-turn
   • 04
     Code Generation Evaluation~18 min · systems, code, execution
   • 05
     Online Evaluation: Production A/B Testing~18 min · systems, ab-testing, production
   • 06
     Regression Testing 과 Eval-Improvement Cycle~18 min · systems, regression, ci

   ✓퀴즈 · 5 문제→

8. 08🛡️Safety 와 Operations

   0/6 lessons

   Red teaming, calibration, cost-quality, eval culture 만들기

   Evaluation 이 회사 나머지 — 보안, reliability, 비용 규율 — 와 만나는 곳, 그리고 다 stick 하게 하는 팀 practice.

   Lesson list (6)

   • 01
     Red Teaming~22 min · safety, red-team, adversarial
   • 02
     Consistency 와 Calibration~18 min · safety, consistency, calibration
   • 03
     Bias 와 Hallucination Detection~22 min · safety, bias, hallucination
   • 04
     Cost-Quality Tradeoff Analysis~18 min · safety, cost, tradeoff, routing
   • 05
     Meta-Evaluation 과 Eval Culture 만들기~18 min · safety, meta-eval, culture, team
   • 06
     LLM Evaluation 의 Open Challenge~18 min · safety, research, frontier

   ✓퀴즈 · 5 문제→