Custom and Composite Scores

When no single metric tells the truth

Sometimes the right grade is a weighted combination — accuracy + safety + brevity, all multiplied by format-compliance. That is a composite score, and writing one is straightforward, but the math has rules.

Composite design rules

Each component must be a 0-1 score. Mixing percentages with raw counts produces garbage.
Hard gates beat weighted averages for critical dimensions. Safety should multiply by 0 if it fails, not subtract a few points.
Always log the components, not just the composite. The composite is the headline; the components are the diagnosis.
Pick weights deliberately and write down why. "Correctness 0.5, format 0.3, brevity 0.2 because users complain twice as often about wrong answers as about long ones."

Principle: Composites should be one summary across many measurements, never a single score that replaces them. Always show the components.

Weighted-average vs multiplicative

Weighted average: good when components are independent and compensable (more brevity can offset slightly lower coherence).

Multiplicative: good when failures are catastrophic (safety = 0 → composite = 0, no matter how brilliant the answer is).

Most real composites are multiplicative on safety / format gates, weighted-average on quality dimensions.

The tax of composites — calibration drift

If your composite is "correctness * 0.5 + brevity * 0.5" and your team optimizes against it, you may get high composite scores by being terse but wrong. The composite did not measure what you wanted; it measured what it told the team to maximize. Re-check composites quarterly against human ratings.

Code

Weighted composite with hard safety gate·python

def composite_score(scores: dict[str, float], weights: dict[str, float], gates: list[str]):
    # Hard gates: any gate failing zeroes the composite.
    for gate in gates:
        if scores.get(gate, 0.0) < 1.0:
            return 0.0, f"hard-gate fail: {gate}"
    weighted = sum(scores[k] * weights[k] for k in weights)
    total = sum(weights.values())
    return weighted / total, "ok"

result = composite_score(
    scores={"correctness": 0.9, "brevity": 0.7, "safety": 1.0, "format": 1.0},
    weights={"correctness": 0.6, "brevity": 0.4},
    gates=["safety", "format"],
)
print(result)  # (0.82, 'ok')

Multi-criteria report — components first, composite second·python

def report_per_case(case_results):
    print(f"{'id':<12} {'corr':>5} {'brev':>5} {'safe':>5} {'fmt':>5} {'COMP':>6}")
    for r in case_results:
        comp, _ = composite_score(r['scores'], WEIGHTS, GATES)
        print(f"{r['id']:<12} {r['scores']['correctness']:>5.2f} {r['scores']['brevity']:>5.2f} "
              f"{r['scores']['safety']:>5.2f} {r['scores']['format']:>5.2f} {comp:>6.2f}")

# Sample output:
# id           corr  brev  safe   fmt   COMP
# qa.001       0.95  0.80  1.00  1.00   0.89
# qa.002       0.70  0.95  1.00  1.00   0.80
# qa.003       0.95  0.50  0.00  1.00   0.00  ← safety failed → composite 0

Custom and Composite Scores

When no single metric tells the truth

Composite design rules

Weighted-average vs multiplicative

The tax of composites — calibration drift

Code

External links

Exercise

Progress

Comments 0