Lightweight Deployment

Start at the scale the problem needs

A batch job, embedded library, or small HTTP service can serve many classical ML systems. Begin with volume, latency, model size, update frequency, availability, privacy boundary, and team ownership. Add orchestration only when scaling, isolation, or compliance evidence requires it.

Ask whether prediction must be online

If scores change daily and actions run in a nightly workflow, batch inference may be simpler and more reliable than an always-on API. Choose real-time serving only when fresher input changes a time-sensitive decision enough to justify the operational surface.

Define the prediction API contract

Validate a versioned raw schema and return score, decision, class meaning, model identity, and policy identity. Test missing, unknown, extra, and malformed fields, and make the response behavior explicit for abstention or partial input.

Separate liveness from readiness

Liveness means the process runs; readiness means the artifact and dependencies loaded and passed a fixture. Expose safe release metadata such as artifact checksum, schema version, build revision, and activation time.

Log for accountability without copying every input

Record status, latency, model version, score and decision distributions, and privacy-reviewed identifiers for later outcome joins. Do not log raw sensitive features by default. Confirm that a prediction can be traced without reconstructing personal data.

Implement failure behavior

Define timeout, retry, queue, abstention, and rollback behavior. A rule fallback is valuable only if validated and safe. Test slow requests, model-load failure, corrupted artifacts, realistic concurrency, and tail latency.

Shadow the candidate before it controls decisions

Mirror real requests to the candidate while the current model remains authoritative. Compare coverage, score distributions, latency, and errors. Shadow traffic observes behavior but does not measure every causal effect of changing decisions.

Use gradual release and immediate rollback

A canary lets a small, explicit share of live decisions use the candidate with guardrails. Keep the previous artifact and configuration, name the rollback owner, and rehearse the action. After promotion, verify the real endpoint, version marker, alarms, and delayed outcome joins.

Code

Minimal FastAPI server for a sklearn pipeline·python

import joblib, json
import pandas as pd
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
pipe = joblib.load("artifacts/churn_v1.joblib")
meta = json.load(open("artifacts/churn_v1.json"))

class Row(BaseModel):
    payload: dict

@app.get("/health")
def health():
    return {"ok": True, "version": meta["version"]}

@app.post("/score")
def score(row: Row):
    X = pd.DataFrame([row.payload])[meta["features"]]
    p = float(pipe.predict_proba(X)[0, 1])
    return {"prob": p, "label": int(p >= meta["threshold"]), "version": meta["version"]}

Structured prediction logging·python

import json, datetime

def log_prediction(payload, prob, label, version, latency_ms):
    print(json.dumps({
        "ts": datetime.datetime.utcnow().isoformat(),
        "version": version,
        "latency_ms": latency_ms,
        "prob": prob,
        "label": label,
        "payload": payload,
    }))