Cost Management for Judge Calls

Judge calls add up fast

A 500-case eval with a strong judge model can run $20-100 per execution. Run it on every PR and you are paying $1000-5000/month for evaluation. Manageable, but worth optimizing.

Six cost-cutting techniques

Cache by hash — same input + same prompt + same judge model = same answer. Cache the result. Re-runs of unchanged cases cost zero.
Cheap-judge first, expensive-judge second — use a cheap model for an initial verdict; only escalate to the expensive judge when the cheap one says fail or is uncertain.
Sample, do not exhaust — for online evals on production traffic, judge 1-5% of calls, not all of them.
Batch when possible — judge multiple cases per request when API supports it (Anthropic's batch API, OpenAI's batch endpoint).
Prefer structured-output mode — eliminates retry loops on malformed JSON, saving 5-15% in failed-call costs.
Model down for trivial axes — format-compliance and length checks don't need a frontier model. Use Haiku / GPT-mini / a small open model for those.

Principle: Match judge model strength to question difficulty. Frontier models for nuanced quality, small models for plumbing checks. Wasting Opus on regex-equivalent questions is just spending money.

Track cost per eval run

Most LLM frameworks expose token usage. Log it. The metric "cost per eval run" should appear on your eval dashboard. If a refactor doubles eval cost, someone should notice immediately, not on the next billing report.

Code

Hash-keyed judge cache·python

import hashlib, json, sqlite3
from pathlib import Path

DB = Path(".eval_cache.sqlite")
conn = sqlite3.connect(DB)
conn.execute("CREATE TABLE IF NOT EXISTS cache (k TEXT PRIMARY KEY, v TEXT, model TEXT, created_at INTEGER)")

def cache_key(prompt, model):
    h = hashlib.sha256()
    h.update(prompt.encode())
    h.update(model.encode())
    return h.hexdigest()

def cached_judge(prompt, model_id, judge_fn):
    k = cache_key(prompt, model_id)
    row = conn.execute("SELECT v FROM cache WHERE k=?", (k,)).fetchone()
    if row:
        return json.loads(row[0])
    out = judge_fn(prompt, model_id)
    conn.execute("INSERT OR REPLACE INTO cache VALUES (?,?,?,strftime('%s','now'))",
                 (k, json.dumps(out), model_id))
    conn.commit()
    return out

Two-tier judge — cheap escalate to expensive·python

def two_tier_judge(case, output, cheap, expensive):
    cheap_v = cheap(case, output)
    if cheap_v["verdict"] == "PASS" and cheap_v["confidence"] >= 0.9:
        return cheap_v  # cheap was confident; stop
    return expensive(case, output)

# In practice: 70-80% of cases settle at the cheap tier.
# Total cost for the suite drops 60-70%, with negligible quality loss
# because the expensive judge sees the hard cases.

Cost Management for Judge Calls

Judge calls add up fast

Six cost-cutting techniques

Track cost per eval run

Code

External links

Exercise

Progress

Comments 0