Cost-Quality Tradeoff Analysis

Mapping the Pareto frontier

Every LLM system sits on a tradeoff between cost and quality. Evaluation tells you exactly where on the curve you are — and where you could go.

The Pareto-frontier exercise

Pick a representative eval set.
Run the same eval against multiple models / prompts / configurations, capturing both quality (eval pass rate) and cost (per-1k-call dollar cost).
Plot quality vs cost. Each (model, config) is a point.
The "Pareto frontier" is the set of configurations where you can't improve quality without increasing cost. Points off the frontier are dominated — strictly worse on both axes.

What the chart usually shows

For most product tasks the curve is steep at the bottom (cheap models barely work, even modest cost increases buy big quality jumps) and flat at the top (frontier models cost 10x more for marginal quality). The right point on the curve is where your product's quality threshold meets the smallest acceptable cost.

Principle: Always run the cost-quality grid. The dominated configurations — points strictly worse on both axes — are usually visible only after you plot.

Model routing for advanced systems

Don't pick one model for everything. Route easy queries to cheap models, hard queries to expensive ones. A simple router (regex on query length / topic / classifier) plus a quality classifier can reduce inference cost 3-10x while preserving quality. RouteLLM, Martian, and the OpenAI Routing API are the standard tools.

Code

Pareto-frontier grid run·python

MODELS = [
    {"name": "haiku-4.5",      "cost_per_1k": 0.25, "quality": None},
    {"name": "gpt-5-mini",     "cost_per_1k": 0.15, "quality": None},
    {"name": "sonnet-4.6",     "cost_per_1k": 3.00, "quality": None},
    {"name": "opus-4.7",       "cost_per_1k": 15.00,"quality": None},
    {"name": "gpt-5",          "cost_per_1k": 5.00, "quality": None},
]

for m in MODELS:
    pass_rate = run_eval_suite(model=m["name"], dataset=DATASET)
    m["quality"] = pass_rate

# Sort by cost; for each successive point, check if quality > best so far.
# Points where quality ≤ a cheaper one are dominated — drop them.
frontier = []
best_q = -1.0
for m in sorted(MODELS, key=lambda x: x["cost_per_1k"]):
    if m["quality"] > best_q:
        frontier.append(m)
        best_q = m["quality"]
print("Pareto frontier:", frontier)

Simple router — cheap for easy, expensive for hard·python

def difficulty_classifier(query: str) -> str:
    if len(query) < 30 and "why" not in query.lower():
        return "easy"
    if any(k in query.lower() for k in ("explain", "compare", "analyze", "why")):
        return "hard"
    return "medium"

def route(query):
    diff = difficulty_classifier(query)
    return {
        "easy":   "haiku-4.5",
        "medium": "sonnet-4.6",
        "hard":   "opus-4.7",
    }[diff]

# Validate: run eval with the routed setup. Quality should approximate
# always-using-opus while cost approximates always-using-haiku for the
# easy share of traffic.

Cost-Quality Tradeoff Analysis

Mapping the Pareto frontier

The Pareto-frontier exercise

What the chart usually shows

Model routing for advanced systems

Code

External links

Exercise

Progress

Comments 0