Logging Prompts and Responses Without Leaking

~12 min · production, logging, privacy

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

You need the data; you can't keep all of it

Logs are necessary for debugging, eval mining, and compliance. Logs of raw user input are also a liability — PII, regulated content, secrets users pasted by mistake. The discipline is logging enough to investigate without keeping more than policy allows.

What to log

Prompt version, model, sampler params (always).
Token usage per category (always).
User input — redacted if sensitive, retained per policy (1–90 days typical).
Tool calls and returns — same redaction rules.
Model output — same redaction rules.
Any verifier / filter flags on the output (always).

How to redact

Pre-log: regex / classifier strips emails, phone numbers, SSNs, credit cards.
Aggregation: keep counts and category labels long-term; raw text short-term.
Retention: tier the logs — full text 7 days, redacted 30 days, aggregated indefinitely.

Code

Pre-log redaction·python

import re

PATTERNS = {
    "email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"),
    "phone": re.compile(r"\b\d{3}-\d{3}-\d{4}\b"),
    "ssn":   re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
}

def redact(text: str) -> tuple[str, dict[str, int]]:
    counts = {}
    for name, p in PATTERNS.items():
        text, n = p.subn(f"[{name}]", text)
        if n: counts[name] = n
    return text, counts

External links

Exercise

Audit the log retention policy for one of your endpoints. Identify any field where you keep raw text longer than necessary. Propose a tiered retention schedule.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousCaching Strategies — What's Worth Caching Next →Telemetry — What to Measure in Prod

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.