The Baseline Problem

A score on its own is meaningless

"Our model scored 84% on the eval." Is that good? You cannot know without three reference points: what would a baseline get? what does the previous version of our system get? what is the human ceiling?

Without a baseline, every number is theatre. With baselines, the same 84% becomes either:

"GPT-3.5 baseline gets 78%, our system gets 84%, humans get 91%. Worth shipping. Headroom exists."
"GPT-3.5 baseline gets 88%. Our complicated RAG system gets 84%. We just paid for retrieval to make things worse."

Three baselines worth measuring

Trivial baseline — what does the dumbest possible approach score? "Always answer 'I don't know.'" "Return the first sentence of the most recently retrieved document." "Echo the user's question back." If your fancy system does not beat the trivial baseline, you have built nothing.
Strong baseline — what does a well-prompted base model with no extras score? Often this number is uncomfortably close to your over-engineered system.
Human baseline — what do skilled annotators score on the same task? This is the realistic ceiling. Beating humans is rare and usually a sign that your eval is wrong.

Principle: Always measure at least one baseline before celebrating a number. The first eval question is not "what did we score?" but "what is the floor?"

The over-engineering trap

Teams routinely add reranking, multi-step retrieval, agent loops, and fine-tuning before establishing what a single well-prompted call would have done. Half the time the simpler baseline matches the complex pipeline at 10× lower cost. Always run the baseline first, even when you are sure the complex thing will win.

Code

Trivial baselines that are surprisingly hard to beat·python

# 1) The 'I don't know' baseline — for QA datasets this can score 30%+
#    on calibration metrics because models that admit ignorance are
#    rewarded over confident hallucinations.
def idk_baseline(question):
    return "I don't have enough information to answer that."

# 2) The 'most-common-class' baseline for classification.
from collections import Counter
def majority_class_baseline(train_labels):
    return Counter(train_labels).most_common(1)[0][0]

# 3) The 'echo' baseline for summarization — return the first 2 sentences.
def echo_baseline(article):
    return ". ".join(article.split(". ")[:2]) + "."

Strong baseline — same model, no extras·python

# Run this BEFORE your fancy RAG / agents / fine-tuning pipeline.
# If the fancy thing doesn't beat this by a meaningful margin,
# you don't need the fancy thing.
def strong_baseline(question, model):
    return model.complete(
        system="You are a helpful assistant. Be concise and accurate.",
        prompt=question,
    )

A score on its own is meaningless

Three baselines worth measuring

The over-engineering trap

Code

External links

Exercise

Progress

Comments 0