Training is the easy part
The first model takes a week. The next year takes the rest of the team. Most ML costs live outside training: labeling pipelines, feature pipelines, drift monitoring, retraining cadence, rollback procedures, on-call, alerting, and the documentation that lets a new engineer ship a fix without paging the original author.
The lifecycle bill
- Data — collection, cleaning, labeling, ongoing quality checks.
- Features — pipeline code, freshness SLAs, backfill stories.
- Training — compute, experimentation, hyperparameter search.
- Serving — latency, autoscaling, A/B framework, fallback path.
- Monitoring — input drift, output drift, calibration, business KPIs.
- Governance — model cards, audits, approvals, deprecation plan.
The honest plan
Estimate each line item before you commit to ML. If the team cannot staff monitoring, the model will silently degrade. If there is no rollback path, the first bad day becomes the last good week.