Regex and Format Validation

Format failures are bugs disguised as quality issues

If your prompt asks for JSON and the model returns markdown-wrapped JSON, your downstream pipeline crashes. That is not a quality failure — it is a format compliance failure, and it deserves its own metric.

What format validation catches

JSON parses correctly, with the expected keys and types.
Code blocks are well-formed and the listed language matches the content.
Citations follow the required pattern (numbered, bracketed, with sources).
Lists have the right cardinality (e.g. exactly 5 bullet points).
Output stays within length constraints (max 100 words).
Required tokens appear ("FINAL ANSWER:", "Step 1:", etc.).

Three levels of strictness

Regex match — pattern check. Fast, brittle.
Schema validation — Pydantic / JSON Schema / Zod. Catches type errors and missing fields.
Semantic validation — schema passed, but does the content satisfy invariants? (e.g. citation IDs reference real sources)

Principle: Format compliance is a separate axis from quality. Track them independently. A model that produces beautiful prose but breaks your JSON schema 10% of the time is broken.

Modern providers offer structured-output modes

OpenAI's response_format=json_schema, Anthropic's tool-use system, Google's controlled generation. They guarantee schema-valid output. If your task allows it, use them — format-failure rate drops to near zero. Then your eval focuses on semantic correctness.

Code

JSON schema validation with pydantic·python

from pydantic import BaseModel, ValidationError, conint, conlist
from typing import Literal

class MovieRecommendation(BaseModel):
    title: str
    year: conint(ge=1900, le=2030)
    genre: Literal["action", "comedy", "drama", "sci-fi", "thriller"]
    score: float
    reasons: conlist(str, min_length=1, max_length=5)

def validate_recommendation(output_str):
    try:
        rec = MovieRecommendation.model_validate_json(output_str)
        return True, rec
    except ValidationError as e:
        return False, e.errors()

ok, _ = validate_recommendation('{"title":"Inception","year":2010,"genre":"sci-fi","score":8.8,"reasons":["mind-bending","great cast"]}')
assert ok

Format-rate as its own metric·python

def format_compliance_rate(outputs, validator):
    total = len(outputs)
    passed = sum(1 for o in outputs if validator(o)[0])
    return passed / total

# Track this per release. A drop from 99% to 92% means the model started
# wrapping JSON in ```json fences again — fix the prompt or add a parser.
rate = format_compliance_rate(eval_outputs, validate_recommendation)
print(f"format compliance: {rate:.1%}")
assert rate >= 0.98, "format-compliance regression"

Regex assertion in promptfoo YAML·yaml

tests:
  - vars:
      query: 'Give me the latest stock price in JSON.'
    assert:
      - type: is-json
      - type: javascript
        value: |
          output.symbol && typeof output.price === 'number'
      - type: regex
        value: '"timestamp":\s*"\d{4}-\d{2}-\d{2}'

Regex and Format Validation

Format failures are bugs disguised as quality issues

What format validation catches

Three levels of strictness

Modern providers offer structured-output modes

Code

External links

Exercise

Progress

Comments 0