MLOps — Experiment Tracking, Versioning, Monitoring

The infrastructure that keeps models alive in production

A model that ships and then sits there is a model that quietly degrades. MLOps is the practice of treating ML models like any other production system: version them, monitor them, retrain them when reality drifts.

The four MLOps pillars

Experiment tracking — log every hyperparameter, metric, and artifact for every run. W&B, MLflow, Comet are the popular tools.
Model versioning — save model + training data + code + environment together. Git LFS, DVC, MLflow Model Registry, or just blob storage with manifest files.
Production monitoring — track latency, throughput, error rate, AND model behavior (prediction distribution, input distribution). Drift detection.
Retraining triggers — automated pipelines that retrain when monitoring signals degradation.

The minimum viable setup

For an indie project, you don't need Kubernetes-managed Kubeflow. You need:

One experiment tracker (W&B or MLflow). Pick one and use it consistently.
Model files saved with a manifest (date, dataset version, metrics).
Production logging that captures inputs and outputs (sampled — don't log every request).
A weekly script that compares production prediction distribution to training distribution. Alert on shifts.

That's it. The big-co MLOps stacks add power but are overkill for a team of 1-3. Start simple, add when needed.

Code

Weights & Biases — the popular tracker·python

# pip install wandb
import wandb

wandb.init(project="pytorch-quest", config={
    "lr": 1e-3,
    "epochs": 50,
    "batch_size": 32,
    "architecture": "ResNet50",
})

for epoch in range(50):
    train_loss = train_one_epoch(...)
    val_loss, val_acc = evaluate(...)

    wandb.log({
        "epoch": epoch,
        "train_loss": train_loss,
        "val_loss": val_loss,
        "val_accuracy": val_acc,
        "learning_rate": optimizer.param_groups[0]['lr'],
    })

wandb.save("best_model.pth")    # ship the model artifact too
wandb.finish()

MLflow — the open-source alternative·python

# pip install mlflow
import mlflow

mlflow.set_experiment("pytorch-quest")

with mlflow.start_run():
    mlflow.log_params({
        "lr": 1e-3,
        "epochs": 50,
        "model": "ResNet50",
    })

    for epoch in range(50):
        train_loss = train_one_epoch(...)
        mlflow.log_metric("train_loss", train_loss, step=epoch)

    mlflow.pytorch.log_model(model, "model")
    # mlflow ui  → http://127.0.0.1:5000 to browse runs

Production monitoring — sampled request logging·python

import json
import random
import time
from pathlib import Path

LOG_DIR = Path("/var/log/pippa-pred")
LOG_DIR.mkdir(parents=True, exist_ok=True)

def log_prediction(input_data, prediction, sample_rate=0.01):
    """Log roughly 1% of predictions for offline analysis."""
    if random.random() > sample_rate:
        return
    record = {
        'ts': time.time(),
        'input_summary': summarize_input(input_data),     # avoid logging raw inputs if PII
        'prediction': prediction,
    }
    fname = LOG_DIR / f"{time.strftime('%Y%m%d')}.jsonl"
    with open(fname, 'a') as f:
        f.write(json.dumps(record) + '\n')

# In your serving code:
# log_prediction(req, response.dict())

CI/CD shape — GitHub Actions example·python

# .github/workflows/ml-pipeline.yml
name: ML Pipeline

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 6 * * 0'       # weekly retrain on Sundays

jobs:
  train-evaluate-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - name: Install
        run: pip install -r requirements.txt
      - name: Tests
        run: pytest tests/
      - name: Train
        run: python train.py --config config.yaml
      - name: Evaluate
        run: python evaluate.py --threshold 0.90
      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: model
          path: outputs/model.pt

MLOps — Experiment Tracking, Versioning, Monitoring

The infrastructure that keeps models alive in production

The four MLOps pillars

The minimum viable setup

Code

External links

Exercise

Progress

Comments 0