Model Testing

Models are non-deterministic; tests must be too

For local-trained or fine-tuned models, the testing problem differs from prompt regression. You're not asking 'does this prompt produce the right shape' — you're asking 'did the model regress on a held-out evaluation set'.

Standard model tests

Held-out accuracy — score on a frozen test set.
Latency / throughput — inference budget per request.
Output format — does the model still produce the agreed schema?
Slice metrics — accuracy on subgroups (Korean inputs, edge-case tokens, long context).
Drift — distribution of scores on a recent traffic sample, compared to a baseline.

CI shape

Model tests typically need GPU runners and model weights. Three options:

Self-hosted GPU runner — runs full eval on every PR.
Hosted cloud GPU (Modal, RunPod) — pay-per-eval, scale to zero.
CPU-only smoke + scheduled GPU full eval — affordable middle ground.

Don't use macOS runners for ML. They're 10× the cost of Linux + GPU isn't available.

Code

Model test job — CPU smoke + scheduled GPU full·yaml

name: model-eval
on:
  pull_request: { paths: ['models/**', 'src/**'] }
  push: { branches: [main] }
  schedule: [{ cron: '0 6 * * *' }]

jobs:
  smoke:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest    # CPU
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e '.[dev]'
      - run: pytest tests/model/test_smoke.py    # tiny held-out, CPU-runnable

  full:
    if: github.event_name != 'pull_request'
    runs-on: [self-hosted, gpu, linux-x64]
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e '.[dev]'
      - run: pytest tests/model/test_full.py    # GPU-required held-out

Models are non-deterministic; tests must be too

Standard model tests

CI shape

Code

External links

Exercise

Progress

Comments 0