Braintrust: Observability and Eval Platform

Evals + tracing + UI + datasets in one platform

Braintrust is a hosted (with self-hosted option) platform for LLM evaluation and observability. It combines: experiment tracking, dataset versioning, side-by-side comparison UI, prompt playground, online (production) evals, and CI/CD integration. Best fit when teams need a shared workspace and don't want to build dashboards.

What you get

Experiments — every eval run becomes a tracked experiment with full inputs/outputs/scores.
Datasets — versioned, UI-editable, importable from JSONL.
Side-by-side diff view — compare two experiments at the case level. Crucial for "did this prompt change help?"
Online evals — log production traffic, score it asynchronously, alert on regressions.
Prompt playground — iterate on prompts in the UI with eval signals attached.
SDKs — TypeScript, Python, plus integrations with promptfoo, OpenAI, Anthropic.

Where it shines

Teams. The shared workspace is the killer feature. Engineer A iterates a prompt; engineer B looks at the experiment 10 minutes later and sees what changed and why. The UI removes the "I have a notebook on my laptop" trap.

Where it doesn't fit

Solo developers iterating fast may find the SaaS layer overkill. Cost matters at scale (per-call pricing). Self-hosted exists but is operationally non-trivial.

Principle: Braintrust is the right pick when 'we have a shared eval workspace' is more valuable to your team than 'the eval lives in our repo.' The two are not exclusive — many teams use both.

Code

Install and run an experiment from Python·python

# pip install braintrust
from braintrust import Eval
from autoevals import LevenshteinScorer

def task(input):
    return your_app(input)

Eval(
    "my-app-eval",
    data=lambda: [
        {"input": "hello", "expected": "hi"},
        {"input": "good morning", "expected": "good day"},
    ],
    task=task,
    scores=[LevenshteinScorer],
)

Online eval on production traffic·python

from braintrust import init_logger, current_logger

logger = init_logger(project="production-app")

def handle_request(req):
    output = your_app(req.input)
    # Log to Braintrust — async scoring runs in the background
    current_logger().log(
        input=req.input,
        output=output,
        metadata={"user_id": req.user_id, "model": "gpt-4o"},
    )
    return output

Side-by-side diff via SDK·typescript

import { compareExperiments } from "braintrust";

const diff = await compareExperiments({
  baseExperimentId: "exp_v3_baseline",
  candidateExperimentId: "exp_v4_new_prompt",
});

console.log(`improved: ${diff.improved.length}, regressed: ${diff.regressed.length}`);
console.log("top regressions:", diff.regressed.slice(0, 5));

Braintrust: Observability and Eval Platform

Evals + tracing + UI + datasets in one platform

What you get

Where it shines

Where it doesn't fit

Code

External links

Exercise

Progress

Comments 0