Which framework for which job
You will likely use 2-3 frameworks across the lifetime of a serious LLM project. Choosing badly costs months. Use the matrix below.
Decision matrix
| Need | Pick |
|---|---|
| Prompt × provider matrix from CLI/YAML | promptfoo |
| Pytest-native Python evals with rich metrics | DeepEval |
| Shared team workspace + production observability | Braintrust |
| Standard NLP benchmarks (MMLU, HumanEval, GSM8K) | lm-evaluation-harness |
| RAG-specific decomposed metrics | RAGAS |
| Agents, tool use, code execution, safety evals | Inspect AI |
| Quick custom hack with no infra | Hand-rolled Python + JSONL |
Mixing frameworks is normal
Real teams typically run promptfoo for prompt iteration, DeepEval for unit-level RAG metrics in CI, Braintrust for team-wide observability, and lm-eval-harness when comparing base models. None of them tries to do all four well, and you don't need them to.
Principle: Frameworks are tools, not religions. Pick the smallest tool that solves the immediate problem, add another when the next problem doesn't fit.
Migration cost is real but manageable
JSONL datasets and OpenAI-style API conventions are the common substrate across frameworks. Keep your data in JSONL and your providers configurable, and switching frameworks becomes a one-week rewrite, not a rebuild.