Output Repair Loops

~14 min · outputs, repair

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Even strict mode misses things

Structured-output strict mode guarantees JSON syntactic validity. It does not guarantee that the values are semantically valid: enum mismatch, value out of range, business-rule violations. You still need a validator and a repair loop.

The pattern

Generate.
Validate against a Pydantic / Zod / JSON Schema validator with semantic rules.
If invalid, send the model the validation error and ask for a fix.
Cap retries — usually 2 or 3.
If still invalid, return a structured error to the user.

Why this is a prompt issue

Repair loops are cheaper when the prompt explains the validation rules upfront. "reason_codes must come from the enum [low_confidence, missing_evidence, policy_block, ok]; if no enum value applies, use 'ok' with notes" is much cheaper than letting the model discover the rule via a rejection.

Code

Validator-driven repair·python

from pydantic import ValidationError

for attempt in range(3):
    raw = call_model(prompt, messages)
    try:
        parsed = VerdictSchema.model_validate_json(raw)
        return parsed
    except ValidationError as e:
        prompt = f"{prompt}\n\nLast attempt failed validation:\n{raw}\nError:\n{e}\nReturn a corrected JSON."
raise RuntimeError("validation failed")

External links

Exercise

Add a 2-retry repair loop around an existing structured-output prompt. Log how often the second attempt succeeds. If high, harden the prompt with the missing rule.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousStreaming Output — Partial JSON and the Pipeline Next →Pydantic / Zod / Schemas as Contracts

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.