Training is the easy part
The first model takes a week. The next year takes the rest of the team. Most ML costs live outside training: labeling pipelines, feature pipelines, drift monitoring, retraining cadence, rollback procedures, on-call, alerting, and the documentation that lets a new engineer ship a fix without paging the original author.
The lifecycle bill
- Data — collection, cleaning, labeling, ongoing quality checks.
- Features — pipeline code, freshness SLAs, backfill stories.
- Training — compute, experimentation, hyperparameter search.
- Serving — latency, autoscaling, A/B framework, fallback path.
- Monitoring — input drift, output drift, calibration, business KPIs.
- Governance — model cards, audits, approvals, deprecation plan.
Hidden coupling multiplies the bill
A changed column can touch ingestion, feature code, training, serving, dashboards, and alert thresholds. If those dependencies are not explicit, a small upstream change becomes silent degradation. Delayed labels make this worse because the system can be wrong for weeks before performance metrics arrive.
The honest plan
Estimate each line item before you commit to ML. Name who owns data quality, when retraining happens, and the exact condition that rolls back to a previous artifact. If the team cannot staff monitoring, the model will silently degrade. If there is no rollback path, the first bad day becomes the last good week.
Plan the end too
Deprecation includes removing callers, deleting retained features and predictions on schedule, and returning dependent workflows to a safe fallback. A model without an owner and retirement condition becomes a ghost service that still carries cost and risk.