The Cost of Deep Learning

The bill is bigger than the GPU

People focus on training cost ("we spent X on H100 hours"), but the real bill includes data labeling, evaluation, monitoring, drift detection, retraining, and the engineer time to keep all of that honest. A good rule of thumb: training compute is roughly a fifth to a tenth of the lifetime cost of a serious deep-learning system.

Inference cost is the line item that bites in production. A 7B model at 1000 requests per second is a non-trivial GPU bill that recurs forever. Quantization, distillation, batching, KV-cache reuse, and serving frameworks (vLLM, TGI) exist to make that number tolerable.

Hidden costs

Data quality — labels are noisy, distributions shift, splits leak. Most "the model is broken" debugging turns out to be data debugging in disguise. Carbon footprint — large training runs are real environmental decisions. Reproducibility — a checkpoint without code, data, and config is a bag of weights you cannot trust.

Warning: If your team cannot answer 'who would notice if this model silently degraded by 5% tomorrow,' you do not have a model in production. You have a science project that survives by luck.

Make the cost legible

The cheap habit that prevents most disasters: log every training run with seed, code commit, dataset hash, hyperparameters, and final metrics. Rerun your eval suite weekly on a frozen test set. Surface drift like you would surface server errors. The cost of doing this is small. The cost of not doing it shows up the day someone asks why the model started behaving strangely.

Principle: Deep learning is operationally expensive, not just computationally expensive. Budgeting only the GPUs is how teams discover the rest of the bill the hard way.

Code

Lineage metadata for every checkpoint·python

import json, subprocess, hashlib, torch
from pathlib import Path
from datetime import datetime

def save_checkpoint(model, optimizer, metrics, cfg, path: Path):
    git_sha = subprocess.check_output(
        ["git", "rev-parse", "HEAD"], text=True
    ).strip()
    manifest = Path("data/train_manifest.txt").read_bytes()
    data_sha = hashlib.sha256(manifest).hexdigest()[:12]

    payload = {
        "model_state_dict": model.state_dict(),
        "optimizer_state_dict": optimizer.state_dict(),
        "metrics": metrics,
        "lineage": {
            "git_sha": git_sha,
            "data_sha": data_sha,
            "config": cfg.to_dict(),
            "timestamp_utc": datetime.utcnow().isoformat(),
        },
    }
    torch.save(payload, path)
    print(f"Saved {path} with git={git_sha[:8]} data={data_sha}")

The bill is bigger than the GPU

Hidden costs

Make the cost legible

Code

External links

Exercise

Progress

Comments 0