API Wrapping & Monitoring

FastAPI in front

vLLM/TGI both expose APIs, but you usually want a thin app-layer in front of them — for auth, rate limiting, request logging, output validation, and your own business logic. FastAPI is the canonical pick.

The five things to monitor in production

Latency — P50, P95, P99 response times. Watch for regressions after model swaps.
Quality — sample outputs and run them through LLM-as-judge periodically. Catches silent quality drift.
Drift — compare current input distributions to baseline. Did your users start asking different questions?
Errors — log and alert on malformed outputs, refusals, hallucinations, schema validation failures.
Cost — GPU utilization, requests per dollar, per-customer cost. The flywheel is broken if you can't see the dollar curve.

Code

FastAPI wrapper around vLLM with structured response·python

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from openai import OpenAI
import time, logging

app = FastAPI()
log = logging.getLogger("my-ft-app")
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

class ChatRequest(BaseModel):
    message: str
    system_prompt: str = "You are a helpful assistant."

class ChatResponse(BaseModel):
    response: str
    model: str
    latency_ms: int

@app.post("/chat", response_model=ChatResponse)
async def chat(req: ChatRequest):
    start = time.perf_counter()
    try:
        result = client.chat.completions.create(
            model="my-finetuned-model",
            messages=[
                {"role": "system", "content": req.system_prompt},
                {"role": "user",   "content": req.message},
            ],
            temperature=0.7,
            max_tokens=1024,
        )
    except Exception as e:
        log.exception("upstream failure")
        raise HTTPException(502, str(e))
    latency_ms = int((time.perf_counter() - start) * 1000)
    log.info("chat ok", extra={"latency_ms": latency_ms,
                                 "prompt_len": len(req.message),
                                 "response_len": len(result.choices[0].message.content)})
    return ChatResponse(
        response=result.choices[0].message.content,
        model=result.model,
        latency_ms=latency_ms,
    )

Exercise

Stand up the FastAPI wrapper above in front of a local vLLM instance. Add structured logging for the three numbers (latency, prompt_len, response_len). Hit it with 100 mixed prompts and grep the logs to compute P50/P95 latency and median response length. This is the minimum operational floor you'd ship behind.

FastAPI in front

The five things to monitor in production

Code

External links

Exercise

Progress

Comments 0