Prompt Regression Testing

Prompts rot silently

Prompts are code. They have bugs. They drift when the underlying model changes. They get tweaked for one use case and break another. Prompt regression tests are unit tests for prompts.

The shape of a prompt regression suite

Each test:

Calls the production prompt with a known input.
Asserts properties of the output: 'contains the user's name', 'is JSON-parseable', 'doesn't include forbidden phrase X', 'cites at least one source'.
Doesn't assert exact wording (LLMs vary). Asserts structural / semantic invariants.

Common regression patterns

JSON schema — the prompt is supposed to return JSON; the test validates against a schema.
Forbidden phrases — the answer must not say 'I cannot help with that' for a topic we explicitly support.
Citation density — the answer must reference at least N sources from the context.
Tone / persona — answer must be in the agreed voice (Pippa's sassy reading; a brand's professional tone).
Refusal correctness — for genuinely off-topic / unsafe inputs, the model must refuse.

Code

Prompt regression test (Python)·python

# tests/test_prompt_regression.py
import pytest, json
from myapp.llm import answer

@pytest.mark.eval
def test_returns_valid_json_for_summary_prompt():
    out = answer(prompt='summarize-doc', input={'doc': SAMPLE_DOC})
    parsed = json.loads(out)  # raises if not valid JSON
    assert 'summary' in parsed
    assert isinstance(parsed['summary'], str)
    assert len(parsed['summary']) > 100

@pytest.mark.eval
def test_does_not_refuse_on_supported_topic():
    out = answer(prompt='biology-q', input={'q': 'How does photosynthesis work?'})
    assert 'cannot help' not in out.lower()
    assert 'sorry' not in out.lower()

@pytest.mark.eval
def test_cites_at_least_one_source():
    out = answer(prompt='rag-q', input={'q': 'When was X founded?'})
    # answer must include a citation pattern like [1] or (source: ...)
    assert any(p in out for p in ['[1]', '(source:', 'According to'])

Prompt Regression Testing

Prompts rot silently

The shape of a prompt regression suite

Common regression patterns

Code

External links

Exercise

Progress

Comments 0