JSONL Logging & Testing

Append-only JSONL logging creates a complete audit trail for debugging, replay testing, and cost analysis.

Mock Transport for Testing

Query JSONL logs with jq: jq 'select(.event == "tool_call") | {tool_name, latency_ms}' logs/agent.jsonl

Why mocked-model tests are folklore

The most common LLM testing pattern is to mock the model: mock_openai.return_value = "fake response". Tests pass forever. Production breaks anyway because the SDK changed a parameter, the prompt drifted, or the tool schema gained a required field. Mocks pin a fictional model to fictional behavior; reality moves out from under them.

Replay-based testing fixes this. Capture a real session as JSONL. In CI, replay the session events through your code (model output is mocked from the JSONL, not from a fixture). Assert on observable outcomes: final text, tool sequence, total cost. When the SDK or prompt changes, the test fails on the right line — the line that diverged from the captured truth.

Code

Append-only JSONL writer with line-level encryption·python

import json, time
from pathlib import Path
from datetime import datetime, timezone

class JSONLLogger:
    """Append-only JSONL logger for agent debugging."""
    def __init__(self, log_path):
        self.log_path = Path(log_path)
        self.log_path.parent.mkdir(parents=True, exist_ok=True)
        self._file = open(self.log_path, "a", buffering=1)  # line-buffered

    def log(self, event_type, **fields):
        record = {"ts": datetime.now(timezone.utc).isoformat(),
                  "event": event_type, **fields}
        self._file.write(json.dumps(record) + "\\n")

    def log_request(self, session_id, model, messages):
        start = time.monotonic()
        self.log("request", session_id=session_id, model=model,
                 message_count=len(messages))
        return start

    def log_response(self, session_id, start_time, finish_reason, usage, cost_usd):
        self.log("response", session_id=session_id,
                 latency_ms=round((time.monotonic() - start_time) * 1000),
                 finish_reason=finish_reason, cost_usd=round(cost_usd, 6))

Replay-based pytest fixture·python

class MockTransport(httpx.AsyncBaseTransport):
    """Return predetermined responses for deterministic testing."""
    def __init__(self, responses: dict):
        self._responses = responses  # url_path → response_body

    async def handle_async_request(self, request):
        body = self._responses.get(request.url.path, {"error": "Not found"})
        return httpx.Response(200, json=body, headers={"content-type": "application/json"})

# Use in tests:
mock = MockTransport({"/v1/chat/completions": {
    "choices": [{"message": {"content": "Test"}, "finish_reason": "stop"}],
    "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15},
}})
test_client = httpx.AsyncClient(base_url="https://api.openai.com", transport=mock)

Mock Transport for Testing

Why mocked-model tests are folklore

Code

External links

Exercise

Progress

Comments 0