Cost Management

~10 min · cost, budget, api

Level 0Apprentice

0 XP0/101 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

AI CI bills are different

Standard CI cost is runner minutes. AI CI also burns:

API tokens — every eval call hits Anthropic / OpenAI / etc.
GPU minutes — orders of magnitude more than CPU.
Cloud egress — when models or datasets are pulled in/out.
Model registry storage — checkpoints add up.

Discipline checklist

Cache eval results on (prompt SHA, model version, scorer SHA). Don't re-call the model on PRs that didn't change those.
Smoke vs full — small subset on PRs, full on main / nightly.
Skip on docs-only — paths-ignore on the AI workflow.
Concurrency — cancel-in-progress on PR re-pushes.
Budget alerts — set monthly spending caps at the API provider; have CI fail-fast if a budget circuit-breaker fires.
Self-hosted for the heavy lifting — homelab GPU is free per minute.

Visibility

Append the run's API token usage to the GitHub Step Summary. Over a quarter, you can see which prompts / which test patterns dominate your bill.

Code

Eval cache key on prompt + model + scorer·yaml

      - name: Compute eval cache key
        id: keys
        run: |
          prompt_sha=$(git ls-files prompts/ | xargs sha256sum | sha256sum | cut -d' ' -f1)
          scorer_sha=$(git ls-files evals/scorers/ | xargs sha256sum | sha256sum | cut -d' ' -f1)
          echo "prompt=$prompt_sha" >> $GITHUB_OUTPUT
          echo "scorer=$scorer_sha" >> $GITHUB_OUTPUT
      - uses: actions/cache@v4
        with:
          path: .eval-cache/
          key: eval-${{ vars.MODEL_VERSION }}-${{ steps.keys.outputs.prompt }}-${{ steps.keys.outputs.scorer }}
          # Same prompt + model + scorer = re-use prior eval results

External links

Exercise

Pull last month's API bill for your AI provider. Identify the top 3 cost drivers (which prompts? which tests? which schedules?). Apply at least one discipline (cache, smoke subset, paths-ignore) to the biggest one.