Observability for LLM Workloads

What to measure

For every Claude call, capture: latency (TTFT and TTLT for streams), input/output/cache token counts, model id, feature label, stop_reason, error class (if any), tool_use names if applicable. These are the dimensions you slice on when costs spike or quality drops.

Trace the loop, not just the call

Tool loops can be 5-10 round-trips. A single trace per round is noise; a span per round inside one parent trace is signal. OpenTelemetry's parent/child span model fits naturally — wrap the loop in a span, each round becomes a child.

Sample prompts and responses

Cost spikes and quality regressions are usually caused by a specific prompt or input pattern. Periodically sample full prompt+response pairs (with PII redaction) into cold storage so you can replay later. cwkPippa's JSONL is exactly this — every event durable, every regression replayable.

Principle: If you cannot answer 'which feature spent the most yesterday and why', you have not built observability yet.

Code

Per-call structured log·python

import time, json

def call_with_telemetry(messages, *, feature: str, log_path: str):
    t0 = time.perf_counter()
    try:
        resp = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            messages=messages,
        )
        latency_ms = int((time.perf_counter() - t0) * 1000)
        with open(log_path, "a") as f:
            f.write(json.dumps({
                "feature": feature,
                "model": "claude-sonnet-4-6",
                "latency_ms": latency_ms,
                "input": resp.usage.input_tokens,
                "output": resp.usage.output_tokens,
                "cache_read": resp.usage.cache_read_input_tokens,
                "stop_reason": resp.stop_reason,
            }) + "\n")
        return resp
    except Exception as e:
        with open(log_path, "a") as f:
            f.write(json.dumps({"feature": feature, "error": type(e).__name__, "msg": str(e)}) + "\n")
        raise

OpenTelemetry trace around a tool loop·python

from opentelemetry import trace
tracer = trace.get_tracer("claude")

async def traced_loop(prompt: str):
    with tracer.start_as_current_span("claude.tool_loop") as parent:
        parent.set_attribute("prompt.length", len(prompt))
        for round_i in range(MAX_ITERS):
            with tracer.start_as_current_span(f"claude.round.{round_i}") as span:
                resp = await client.messages.create(...)
                span.set_attribute("stop_reason", resp.stop_reason)
                span.set_attribute("output_tokens", resp.usage.output_tokens)
                if resp.stop_reason != "tool_use":
                    return resp

Exercise

Add telemetry to one critical Claude code path. Capture latency, tokens, stop_reason, and feature label. Build one dashboard query that answers 'which feature spent the most yesterday'.

Hint

If your logging library is not structured (key/value), now is the time to switch — text logs cannot answer slice-and-dice questions cheaply.

Observability for LLM Workloads

What to measure

Trace the loop, not just the call

Sample prompts and responses

Code

External links

Exercise

Progress

Comments 0