OpenAI Compatibility Tests

OpenAI compatibility is a spectrum

Every local engine claims OpenAI-compatible. They're all almost compatible. The divergences are predictable but not standardized:

Streaming format. All emit SSE, but exact chunk shapes differ.
Tool calling. Native Ollama gives parsed dict args; OpenAI emits JSON string args; some engines wrap differently.
Structured output. Ollama uses format field with JSON Schema. OpenAI uses response_format. vLLM uses guided_json. Same idea, different keys.
Token counting. Slightly different (or missing) token usage fields.

How to test compatibility

Don't trust the label. Run a small test suite that exercises the features you use. The list below is the canonical local-AI compatibility test:

Non-streaming chat with a simple message.
Streaming chat with the same message.
Multi-turn conversation (system + user + assistant).
Tool definition + multi-turn tool loop.
Structured output (JSON Schema).
Token usage in the response.

If all six pass on your engine, you can swap engines without app changes. If items 4 or 5 fail, expect to write engine-specific shims.

Code

Compatibility test harness·python

import httpx, json, time

def test_compat(base_url: str, model: str, label: str = ""):
    print(f"\n=== {label or base_url} ===")
    client_kwargs = {"timeout": 120.0}

    # 1. Non-streaming
    r = httpx.post(f"{base_url}/chat/completions", json={
        "model": model,
        "messages": [{"role": "user", "content": "Say 'ok' and nothing else."}],
        "stream": False,
    }, **client_kwargs)
    print(f"1. non-stream: {'PASS' if r.status_code == 200 else 'FAIL'}")

    # 2. Streaming
    with httpx.stream("POST", f"{base_url}/chat/completions", json={
        "model": model,
        "messages": [{"role": "user", "content": "Count: 1 2 3"}],
        "stream": True,
    }, timeout=None) as r2:
        got_chunks = sum(1 for line in r2.iter_lines() if line.startswith("data:"))
    print(f"2. stream:     {'PASS' if got_chunks > 1 else 'FAIL'} ({got_chunks} chunks)")

    # 3. Tool call
    tools = [{"type": "function", "function": {
        "name": "say_ok", "description": "Reply with 'ok'.",
        "parameters": {"type": "object", "properties": {}, "required": []},
    }}]
    r3 = httpx.post(f"{base_url}/chat/completions", json={
        "model": model,
        "messages": [{"role": "user", "content": "Use the say_ok tool."}],
        "tools": tools, "stream": False,
    }, **client_kwargs)
    has_tools = bool(r3.json().get("choices", [{}])[0].get("message", {}).get("tool_calls"))
    print(f"3. tool_call:  {'PASS' if has_tools else 'FAIL'}")

# Run against multiple engines
test_compat("http://localhost:11434/v1", "qwen2.5:7b",   label="Ollama")
test_compat("http://localhost:8080/v1",  "qwen-direct",  label="llama-server")
# test_compat("http://localhost:8000/v1", "Qwen/Qwen2.5-7B-Instruct", label="vLLM")

OpenAI compatibility is a spectrum

How to test compatibility

Code

External links

Exercise

Progress

Comments 0