Latency is the metric, not pass rate
Most teams optimize CI for the wrong number. They obsess over pass rate (how often is main green?). That matters, but it is downstream. The upstream metric is feedback latency: how long from I pushed a change to I know if it broke anything.
Why latency matters: a 90-minute CI run is not just slow. It is a context-switch tax. By the time it finishes, you have moved on to a different problem, started a different branch, or gone to lunch. When the failure lands, you have to re-load the mental model. The actual cost of a 90-minute run is closer to a half-day than to 90 minutes.
Tiered feedback
Real teams stack feedback at multiple speeds:
- Editor (seconds) — type errors, lint warnings, format issues. LSP + format-on-save catches most of them before you commit.
- Pre-commit hook (seconds) — fast checks that should never reach CI: lint, format, typo check.
- CI (minutes) — the main test suite. Aim for under 10 minutes for the bulk; under 2 minutes for the smoke subset.
- Pre-deploy (minutes) — integration tests, end-to-end checks, performance budgets.
- Post-deploy (minutes to hours) — smoke tests in production, error rate dashboards, user-facing observability.
The faster the tier, the cheaper the failure. CI's job is to push as much feedback as possible to the cheaper tiers.