Online Evaluation: A/B Testing in Production

The most honest eval is real users

Offline evals tell you what your test set says. Online A/B tests tell you what your users do. Both matter, but online is the closest thing to ground truth — and the only way to measure things you cannot label offline (engagement, retention, monetization).

What online A/B tests measure that offline cannot

Engagement — did users keep talking to the new variant longer?
Task completion — what fraction completed their actual goal?
Satisfaction — explicit thumbs-up/down or implicit (no follow-up needed).
Latency tolerance — did users wait for slower-but-better responses, or churn?
Long-term retention — do users come back?

The methodology fundamentals

Random assignment at the user (not session) level.
Pre-registered metric — declare the primary metric before the test runs. Otherwise you'll find a metric that "won" by chance.
Sample size calculation — power analysis up front; don't peek at results.
Long enough to capture novelty effects — first-week effects often don't generalize.
Monitor guardrail metrics — even if your primary metric wins, check that latency, error rate, and cost didn't blow up.

Principle: Offline evals propose; online A/B tests dispose. Never ship a major change on offline numbers alone.

Statistical significance, not just direction

"Variant B was 2% better" means nothing without confidence intervals. With 1,000 users per arm, the noise floor on most metrics is ±5%. Compute the p-value (or the confidence interval) before declaring a winner. Tools like statsmodels, ABBA, or your platform's built-in stats handle this.

Code

Random assignment at the user level·python

import hashlib

def ab_assign(user_id, variant_count=2, salt="experiment-2026-q2"):
    """Stable, deterministic, uniform assignment."""
    h = hashlib.sha256(f"{salt}:{user_id}".encode()).hexdigest()
    bucket = int(h[:8], 16) % variant_count
    return bucket

# Same user → same bucket every call. Different experiment → different salt.
# Avoids 'session-level' assignment (different responses on different visits).

Two-proportion z-test for completion rate·python

from statistics import NormalDist

def two_prop_z_test(c_a, n_a, c_b, n_b):
    p_a, p_b = c_a / n_a, c_b / n_b
    p_pool = (c_a + c_b) / (n_a + n_b)
    se = (p_pool * (1 - p_pool) * (1/n_a + 1/n_b)) ** 0.5
    z = (p_a - p_b) / se
    p_value = 2 * (1 - NormalDist().cdf(abs(z)))
    return {
        "p_a": p_a, "p_b": p_b,
        "diff": p_a - p_b,
        "z": z,
        "p_value": p_value,
        "significant": p_value < 0.05,
    }

print(two_prop_z_test(c_a=420, n_a=1000, c_b=465, n_b=1000))
# diff=0.045 (4.5pp), p_value~0.04 → just barely significant

Online Evaluation: A/B Testing in Production

The most honest eval is real users

What online A/B tests measure that offline cannot

The methodology fundamentals

Statistical significance, not just direction

Code

External links

Exercise

Progress

Comments 0