Production maturity ladder

Ladder 어디?

Demo — prompt이 inline string; output이 example에 좋아 보임.
Working — prompt이 파일에; basic eval set 존재; 한 모델, fallback 없음.
Reliable — versioned prompt; CI eval gate; logging; rollback flag; primary model에 fallback.
Operated — full telemetry; cost dashboard; rollout canaried; incident runbook practiced; failure를 test case로 turn하는 eval-mining pipeline.
Engineered — multi-provider adapter; per-request routing; cost-quality-latency reporting; red-team eval suite; quarterly prompt rewrite; documented invariant.

어떻게 climb

각 rung이 분기나 두 분기 work. "Engineered" 어디나 필요 X; 각 system을 failure cost에 match. Marketing-copy generator는 Working에 살 수 있어. Refund-decisioning agent는 Operated나 Engineered여야 해.

이 quest이 너 두는 곳

모든 트랙과 exercise complete했으면 Engineered용 부품 list 가져. 남은 work는 너의 specific system에 wiring, 그리고 절대 끝나지 X — 이 system들 살아 있고, provider 움직이고, threat evolve. Ongoing job에 welcome.

Code

Self-assessment checklist·markdown

Demo:
  [x] prompt produces something

Working:
  [ ] prompts in source control
  [ ] eval set ≥ 20 cases
  [ ] documented sampler params

Reliable:
  [ ] versioned prompts with frontmatter
  [ ] CI eval gate
  [ ] log retention policy
  [ ] rollback flag tested
  [ ] fallback chain ≥ 2 providers

Operated:
  [ ] dashboards: health, quality, routing
  [ ] rollouts canaried
  [ ] incident runbook quarterly drill
  [ ] failures → eval cases pipeline

Engineered:
  [ ] adapter pattern; provider-neutral application code
  [ ] per-request model + reasoning routing
  [ ] red-team eval suite
  [ ] quarterly prompt rewrites against eval baseline
  [ ] invariants documented in CLAUDE.md / equivalent

Production maturity ladder

Ladder 어디?

어떻게 climb

이 quest이 너 두는 곳

Code

External links

Exercise

Progress

댓글 0