Testing & Iterating on Skills

~14 min · skills, testing, iteration, evaluation

Level 0🌱 Novice

0 XP0/70 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Skills are code; treat them like code

Skills change behavior — silently, in ways that don't show up in your test suite. The defense: a tiny eval harness. A folder of inputs + expected-output sketches, a script that runs the Skill against each, you eyeball the diff. It's not unit testing, it's regression eyeballing — and that's enough to catch the surprises.

The four moments to run the eval: (1) when you change the Skill body, (2) when you change paths:, (3) when you upgrade the underlying model, (4) before merging a Skill change. Skip any of those and you're tuning blind.

Code

An eval harness for one Skill·text

.claude/skills/security-audit/
├── SKILL.md
└── tests/
    ├── inputs/
    │   ├── safe-route.ts          ← shouldn't flag anything
    │   ├── sql-injection.ts       ← should flag CWE-89 high
    │   └── auth-bypass.ts         ← should flag CWE-287 high
    ├── expected/
    │   ├── safe-route.expected.md
    │   ├── sql-injection.expected.md
    │   └── auth-bypass.expected.md
    └── run.sh

tests/run.sh — run + diff each input·bash

#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"

for input in inputs/*.ts; do
  name=$(basename "$input" .ts)
  expected="expected/${name}.expected.md"
  actual=$(claude -p "/security-audit $input" \
    --output-format text --allowedTools "Read")
  echo "=== $name ==="
  diff <(echo "$actual") "$expected" || echo "DIFF"
done

External links

Anthropic skill testing patterns

Exercise

Build a 3-input eval harness for one of your Skills. Capture the expected outputs once. Run the harness; eyeball the diffs. Then make a small change to the Skill and re-run. Note what shifted.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousScripts — Determinism Inside a Skill Next →Enterprise Distribution & Marketplace

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.