"The model is correct most of the time" can hide two distinct problems:
Consistency — does it give the same answer when you rephrase the question?
Calibration — does its expressed confidence match its actual accuracy?
Both matter. An inconsistent model erodes user trust ("I asked the same thing twice and got two different answers"). An uncalibrated model lies confidently — saying "I'm 95% sure" while being right only 60% of the time.
Measuring consistency
Take N rephrasings of the same question. Run them all. Score the answers' semantic similarity (BERTScore or embedding cosine). Average is your consistency score for that case. Average across cases is the system's consistency metric.
Measuring calibration
Have the model state confidence ("I'm X% sure"). Bin predictions by stated confidence. For each bin, compute actual accuracy. A well-calibrated model has actual accuracy ≈ stated confidence in each bin. Plot the deviation — that's the calibration curve.
Principle: A confidently wrong model is more dangerous than a tentatively wrong one. Calibration is a safety property, not a vanity metric.
Why frontier models are still poorly calibrated
RLHF training tends to over-confidence. Models learn that confident answers get higher human ratings, so they become more confident than warranted. The 2025-2026 generation is somewhat better but still typically over-confident on the "I'm sure" tail.
Code
Consistency test via rephrasings·python
from sentence_transformers import SentenceTransformer
import numpy as np
emb = SentenceTransformer("all-mpnet-base-v2")
def consistency(question_variants, model):
answers = [model.complete(q) for q in question_variants]
vecs = emb.encode(answers)
# Pairwise cosine similarity, averaged
n = len(vecs)
sims = []
for i in range(n):
for j in range(i + 1, n):
sims.append(np.dot(vecs[i], vecs[j]) / (np.linalg.norm(vecs[i]) * np.linalg.norm(vecs[j])))
return float(np.mean(sims))
variants = [
"What is the capital of France?",
"Which city is France's capital?",
"France's capital city is what?",
"Name the capital of France.",
]
print(consistency(variants, my_model)) # ~0.95 for a calibrated factual QA
Calibration curve·python
from collections import defaultdict
def calibration_curve(predictions, n_bins=10):
"""predictions: list of (stated_confidence, was_correct)."""
bins = defaultdict(lambda: [0, 0]) # (correct, total)
for conf, correct in predictions:
bin_idx = min(int(conf * n_bins), n_bins - 1)
bins[bin_idx][0] += int(correct)
bins[bin_idx][1] += 1
curve = []
for i in range(n_bins):
c, t = bins[i]
if t > 0:
curve.append((i / n_bins, c / t, t))
return curve
# Output: [(stated_conf_bin, actual_accuracy, n_samples), ...]
# A calibrated model has actual_accuracy ≈ bin midpoint.
For your most factual feature, write 5 rephrasings of 20 questions. Measure consistency. Then for each answer, ask the model to state confidence; measure calibration. Both are user-trust signals — the values you'll learn are usually below what you assumed.
Progress
Progress is local-only — sign in to sync across devices.