Robustness, Bias, and Safety

What can go wrong with a deployed model

Distribution shift — production data drifts away from training data. Model accuracy degrades silently.
Adversarial inputs — small, deliberate perturbations cause large prediction changes (especially in vision and audio).
Bias — model performs unevenly across demographic groups, often reflecting biases in training data.
Hallucination (LLMs) — fluent, plausible-sounding outputs that are factually wrong.
Prompt injection (LLMs) — user-supplied instructions in retrieved context override your system prompt.
Privacy leakage — model memorizes training data and emits it (rare but documented).

Warning: If you ship a model without monitoring for any of these, you ship a system that can degrade or cause harm without warning. The cost of monitoring is small compared to the cost of finding out from a customer (or regulator).

The minimal safety toolkit

Per-class metrics — never accept aggregate accuracy without per-class breakdown.
Sliced evaluation — evaluate on subsets your users actually care about (demographic, geographic, language).
Drift monitoring — track input distributions and model outputs over time; alert on shifts.
Confidence calibration — model probabilities should match observed frequencies; uncalibrated models lie about their certainty.
Adversarial robustness checks — at minimum, evaluate against a few standard attacks (FGSM, PGD for vision; jailbreak datasets for LLMs).
Red-teaming for LLMs — humans deliberately try to break your model's safety guardrails before users do.

Why this is engineering, not philosophy

Safety failures show up as customer complaints, regulatory fines, lawsuits, or news stories. They're an engineering risk you can quantify and mitigate, not a vague ideal. Treat them like you treat security vulnerabilities: scan for them, set up alerts, fix when found.

Principle: The cost of safety monitoring is paid in engineer time. The cost of skipping it is paid in customer trust, legal exposure, and on-call pages. The first cost is much smaller and predictable.

Code

Per-group accuracy, the simplest fairness check·python

import numpy as np

def per_group_metrics(y_true, y_pred, group):
    groups = np.unique(group)
    out = {}
    for g in groups:
        mask = (group == g)
        acc = (y_pred[mask] == y_true[mask]).mean()
        out[str(g)] = {"n": int(mask.sum()), "acc": float(acc)}
    return out

# Print: spot the group whose accuracy is much lower than the average.
for grp, m in per_group_metrics(y_val, preds, val_group).items():
    print(f"{grp}: n={m['n']}, acc={m['acc']:.4f}")

Calibration check via reliability diagram·python

import numpy as np
import matplotlib.pyplot as plt

def reliability(y_true, prob, n_bins=10):
    bins = np.linspace(0, 1, n_bins + 1)
    confs, accs, sizes = [], [], []
    for lo, hi in zip(bins[:-1], bins[1:]):
        mask = (prob >= lo) & (prob < hi)
        if mask.sum() == 0: continue
        confs.append(prob[mask].mean())
        accs.append(y_true[mask].mean())
        sizes.append(mask.sum())
    return np.array(confs), np.array(accs), np.array(sizes)

c, a, _ = reliability(y_val, preds_prob)
plt.plot([0, 1], [0, 1], "k--")
plt.scatter(c, a)
plt.xlabel("predicted prob"); plt.ylabel("observed accuracy"); plt.show()

What can go wrong with a deployed model

The minimal safety toolkit

Why this is engineering, not philosophy

Code

External links

Exercise

Progress

Comments 0