The Vibes-Based Problem

Why "looks good to me" never scales

Every AI product is born the same way. Someone writes a prompt, glances at three outputs, says "this is good," and ships. That moment of subjective inspection is fine for exploration — and useless for everything that follows: regression control, model upgrades, prompt iterations, retrieval changes, tool revisions, multi-version A/B tests. The instant a second person looks at the system, your taste stops being authoritative.

The failure modes pile up fast:

It does not scale. You cannot manually inspect 10,000 production outputs per week.
It does not reproduce. Your "good" is not your colleague's "good," and neither is your future self's "good."
It does not convince. A stakeholder asking "did this prompt change improve quality?" deserves a number, not a vibe.
It does not catch regressions. A change that fixes 80% of cases while silently breaking 3% of edge cases is invisible to spot-checking, but devastating in production.

Evals are the cure

An evaluation turns subjective taste into repeatable evidence about a specific behavior. It pins down: what should be true, on which inputs, measured how, with what threshold for shipping. Once written down, anyone on the team — including the future-you who forgot why the prompt looks the way it does — can re-run the measurement and decide whether a change is safe.

Principle: If you cannot measure it, you cannot protect it. The first eval you write is the first piece of behavior you have promised to keep working.

The operator pattern

Before you change a prompt, model, retrieval pipeline, or tool definition, write down — in plain words — the behavior you must not regress and how you will detect it. This single habit separates teams that ship LLM features confidently from teams that ship and pray.

Common failure mode: The biggest eval failure is not a low score. It is measuring something easy while the product breaks somewhere important. Score 100% on a metric that does not matter and the system can still be silently rotting.

Code

Eval intent note (write before any prompt change)·markdown

## Behavior to protect
- User task: <what is the user trying to accomplish?>
- Must be true: <what invariant must hold for this task?>
- Failure examples: <2-3 outputs that should NEVER appear>
- Success examples: <2-3 outputs that clearly satisfy the user>
- Metric or grader: <how will you measure pass/fail?>
- Release gate: <what threshold blocks shipping?>
- Owner: <who is accountable for this eval?>

The minimum viable eval loop·python

# The smallest possible eval — three lines you can write today.
test_cases = [
    {"input": "Translate 'hello' to French", "must_contain": "bonjour"},
    {"input": "Translate 'good morning' to German", "must_contain": "guten morgen"},
]
for case in test_cases:
    output = your_app(case["input"]).lower()
    assert case["must_contain"] in output, f"FAIL on {case['input']}: got {output!r}"
print("all pass")

The Vibes-Based Problem

Why "looks good to me" never scales

Evals are the cure

The operator pattern

Code

External links

Exercise

Progress

Comments 0