promptfoo: CLI-First Eval Runner

The fastest way to run evals from the command line

promptfoo (open-source, v0.118+ as of mid-2026) is a CLI and TypeScript library for evaluating LLM applications. It supports 60+ providers, 40+ deterministic assertion types, 14 model-graded assertion types, and a declarative YAML configuration. Best fit when you want fast iteration without building Python infrastructure.

What you get out of the box

Side-by-side comparison of multiple prompts × multiple providers in a single run.
Assertions: contains, equals, is-json, contains-json, regex, similar (embedding cosine), llm-rubric, javascript, python.
Built-in red-teaming module with 50+ vulnerability categories (prompt injection, jailbreaks, OWASP LLM Top 10).
HTML report viewer (promptfoo view) and CSV export.
Caching, retry, and concurrency controls.

Where it shines

promptfoo's strength is the declarative config: a single YAML can describe 4 prompts × 3 models × 50 test cases × 6 assertions. That kind of matrix in code would take days; in YAML it takes minutes. Excellent for prompt-engineering iteration and for quick provider comparisons.

Where it doesn't fit

If your eval lives inside complex Python pipelines (custom retrievers, multi-agent loops, fine-tuning experiments), the YAML model becomes awkward. DeepEval or a hand-rolled harness fits better there.

Principle: promptfoo wins for prompt-and-provider matrix evals run from the command line. It is the right first tool, and frequently also the last.

Code

Install and initialize·bash

# Install globally or use npx
npm install -g promptfoo
# or
npx promptfoo@latest init

# Scaffold a project
promptfoo init

# Run the suite
promptfoo eval

# Open the HTML viewer
promptfoo view

promptfooconfig.yaml — full prompt × provider matrix·yaml

description: 'Translation eval — three prompts, two providers'

prompts:
  - "Translate '{{text}}' to {{language}}."
  - "You are a translator. Convert '{{text}}' into {{language}}."
  - file://prompts/system.txt

providers:
  - openai:gpt-4o-mini
  - anthropic:messages:claude-sonnet-4-5

tests:
  - vars:
      text: 'Hello world'
      language: 'French'
    assert:
      - type: contains
        value: 'Bonjour'
      - type: llm-rubric
        value: 'Translation is natural and fluent'
  - vars:
      text: 'Good morning'
      language: 'German'
    assert:
      - type: similar
        value: 'Guten Morgen'
        threshold: 0.6
      - type: javascript
        value: 'output.length < 200'

Custom Python assertion·yaml

tests:
  - vars: {input: 'extract emails from this text'}
    assert:
      - type: python
        value: |
          import re
          emails = re.findall(r'[\w.+-]+@[\w-]+\.[\w.-]+', output)
          return len(emails) > 0

promptfoo: CLI-First Eval Runner

The fastest way to run evals from the command line

What you get out of the box

Where it shines

Where it doesn't fit

Code

External links

Exercise

Progress

Comments 0