Consistency and Calibration

Two reliability axes that live behind correctness

"The model is correct most of the time" can hide two distinct problems:

Consistency — does it give the same answer when you rephrase the question?
Calibration — does its expressed confidence match its actual accuracy?

Both matter. An inconsistent model erodes user trust ("I asked the same thing twice and got two different answers"). An uncalibrated model lies confidently — saying "I'm 95% sure" while being right only 60% of the time.

Measuring consistency

Take N rephrasings of the same question. Run them all. Score the answers' semantic similarity (BERTScore or embedding cosine). Average is your consistency score for that case. Average across cases is the system's consistency metric.

Measuring calibration

Have the model state confidence ("I'm X% sure"). Bin predictions by stated confidence. For each bin, compute actual accuracy. A well-calibrated model has actual accuracy ≈ stated confidence in each bin. Plot the deviation — that's the calibration curve.

Principle: A confidently wrong model is more dangerous than a tentatively wrong one. Calibration is a safety property, not a vanity metric.

Why frontier models are still poorly calibrated

RLHF training tends to over-confidence. Models learn that confident answers get higher human ratings, so they become more confident than warranted. The 2025-2026 generation is somewhat better but still typically over-confident on the "I'm sure" tail.

Code

Consistency test via rephrasings·python

from sentence_transformers import SentenceTransformer
import numpy as np

emb = SentenceTransformer("all-mpnet-base-v2")

def consistency(question_variants, model):
    answers = [model.complete(q) for q in question_variants]
    vecs = emb.encode(answers)
    # Pairwise cosine similarity, averaged
    n = len(vecs)
    sims = []
    for i in range(n):
        for j in range(i + 1, n):
            sims.append(np.dot(vecs[i], vecs[j]) / (np.linalg.norm(vecs[i]) * np.linalg.norm(vecs[j])))
    return float(np.mean(sims))

variants = [
    "What is the capital of France?",
    "Which city is France's capital?",
    "France's capital city is what?",
    "Name the capital of France.",
]
print(consistency(variants, my_model))  # ~0.95 for a calibrated factual QA

Calibration curve·python

from collections import defaultdict

def calibration_curve(predictions, n_bins=10):
    """predictions: list of (stated_confidence, was_correct)."""
    bins = defaultdict(lambda: [0, 0])  # (correct, total)
    for conf, correct in predictions:
        bin_idx = min(int(conf * n_bins), n_bins - 1)
        bins[bin_idx][0] += int(correct)
        bins[bin_idx][1] += 1
    curve = []
    for i in range(n_bins):
        c, t = bins[i]
        if t > 0:
            curve.append((i / n_bins, c / t, t))
    return curve

# Output: [(stated_conf_bin, actual_accuracy, n_samples), ...]
# A calibrated model has actual_accuracy ≈ bin midpoint.

Consistency and Calibration

Two reliability axes that live behind correctness

Measuring consistency

Measuring calibration

Why frontier models are still poorly calibrated

Code

External links

Exercise

Progress

Comments 0