Evals Before Prompts — Why Working Backward Wins

~16 min · evaluation, process

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Write the test before the prompt

The strongest workflow for serious prompt work: before you write the prompt, write 20 representative inputs and the expected outputs (or pass/fail criteria). The prompt is then a search for something that satisfies the eval. Without the eval, you'll iterate on whatever output looks pleasant on the example you happened to try.

What an eval set looks like

Inputs — drawn from real traffic, not made up.
Pass criteria — programmatic where possible (must_contain, must_not_contain, schema match), human-judged where necessary.
Coverage — happy path, edge cases, hostile inputs, refusal cases.
Stable identifiers — each case has an id so you can track it across versions.

Why this beats prompt-by-feel

You know when the prompt is done — when the eval passes.
You catch regressions when a 'small tweak' breaks something.
The eval becomes the prompt's spec — onboarding the next person is easier.

Code

Eval-first prompt workflow·python

# 1. Write the eval
cases = [
    {"id": "happy-1", "input": "...", "must_contain": ["approved"]},
    {"id": "refund-1", "input": "...", "must_not_contain": ["approved"]},
    {"id": "empty",    "input": "",     "must_match_schema": EmptyError},
    # ...20 total
]

# 2. Iterate the prompt
while True:
    score = run_eval(cases, prompt)
    if score == len(cases): break
    prompt = improve(prompt, score.failures)

External links

Exercise

Pick a prompt you're about to ship or update. Before changing it, write 20 eval cases drawn from real traffic. Run the current prompt. Note the actual pass rate.

Progress

Progress is local-only — sign in to sync across devices.

← 🛡️ Defending Prompts Against Real AdversariesQuiz · 4 questions Next →Golden Sets — Building and Maintaining Them

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.