API Testing for AI Services

The wrapper API is contract-bound; the model behind it isn't

An AI service has two layers to test in CI:

API layer — request/response shape, status codes, auth, rate limits. Conventional contract testing.
Model layer — semantic / behavioral correctness. Eval-style tests.

The first layer is what your downstream consumers depend on; it must be stable. The second layer is allowed to vary — within agreed bounds.

API contract tests

OpenAPI / Pydantic schema — every response validates against the documented shape.
Status code matrix — empty input → 400; unauthorized → 401; rate limit → 429; happy path → 200.
Streaming protocol — for SSE / WebSocket, assert event types, chunk separators, terminating events.
Backward compatibility — old clients keep working when fields are added; never remove fields without a deprecation window.

Mocking the model

For API contract tests, mock the underlying model. A real model call is slow, expensive, and non-deterministic. Inject a fake adapter that returns a canned response that matches the schema. The eval layer (separate jobs) handles real model behavior.

Code

API contract test with FastAPI + Pydantic·python

# tests/test_api_contract.py
from fastapi.testclient import TestClient
from myapp.main import app, get_brain
from myapp.schemas import ChatResponse

class FakeBrain:
    async def stream(self, *a, **kw):
        yield {'type': 'text', 'text': 'hello'}
        yield {'type': 'done', 'usage': {'in': 5, 'out': 1}}

app.dependency_overrides[get_brain] = lambda: FakeBrain()
client = TestClient(app)

def test_chat_returns_documented_shape():
    r = client.post('/api/chat', json={'message': 'hi'})
    assert r.status_code == 200
    ChatResponse.model_validate(r.json())     # raises if shape drifts

def test_chat_rejects_empty_input():
    r = client.post('/api/chat', json={'message': ''})
    assert r.status_code == 400

API Testing for AI Services

The wrapper API is contract-bound; the model behind it isn't

API contract tests

Mocking the model

Code

External links

Exercise

Progress

Comments 0