Verifier-Driven Reasoning

~18 min · reasoning, verification

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Reasoning is generation; verification is checking

The model's reasoning chain is itself an LLM output — confident-sounding but not always right. A verifier is a second pass that checks the reasoning against an external truth: a calculator, a unit test, a schema validator, a grounded search.

Useful verifiers

Code execution — for math, the model writes code and runs it. The output is checked.
Schema validation — JSON output validates against pydantic / zod.
Search verification — model claims a fact; a search tool checks it.
Self-critique pass — second LLM call asked to find errors in the first response.

The pattern

Generate → verify → if verifier rejects, regenerate with verifier feedback. This is the basis of agent loops, code-execution tools, and structured-output retry. Once you get used to writing prompts that interlock with a verifier, you stop trusting raw LLM output for high-stakes tasks.

Code

Generate → verify → regenerate·python

for attempt in range(3):
    out = generate(prompt)
    error = verify(out)        # returns None on success
    if error is None:
        return out
    prompt = f"{prompt}\n\nPrior attempt failed verification:\n{out}\nError: {error}\nFix and retry."
raise RuntimeError("verification failed after 3 attempts")

External links

Exercise

Add a verifier to one prompt that returns structured output. Loop generate → verify → regenerate up to 3 times on failure. Measure how often the second attempt succeeds.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousReasoning vs Output — Separating the Two Next →Step Compression — Don't Show Your Work When You Don't Have To

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.