A/B Prompts in Production

~14 min · evaluation, ab-testing

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

The eval set is not the user

Golden-set evals are a proxy. The actual judge of prompt quality is real users on real traffic. A/B testing in production is how you confirm what evals suggest.

Wiring A/B

Hash on user_id (or request_id) to assign cohorts deterministically.
Track outcomes per cohort: success rate, downstream conversion, complaint rate, cost.
Run long enough for statistical significance — usually 1–2 weeks for low-volume systems.
Pre-register the metric and the decision rule ("if v_new is ≥ v_old on success rate at p < 0.05, ship it").

Pitfalls

Cohort imbalance from hashing on a non-uniform feature.
Novelty effects (new prompt looks better for first 3 days, then mean-reverts).
Spillover when the same user lands in both cohorts across sessions.
Outcome metrics that lag — wait for the full window.

Code

Hash-based cohort assignment·python

import hashlib

def cohort(user_id: str, experiment: str) -> str:
    h = hashlib.sha256(f"{experiment}:{user_id}".encode()).hexdigest()
    return "v_new" if int(h, 16) % 100 < 50 else "v_old"

prompt = PROMPTS[experiment][cohort(user_id, "support_v8")]

External links

Exercise

Set up an A/B test on one prompt change. Pre-register the metric, threshold, and duration. Run it. Honor the result even if it disappoints.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousMetric Design — Don't Measure What's Easy Next →Eval Drift — When Your Test Set Lies

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.