Serving — FastAPI, TorchServe, vLLM

From .pth to HTTP endpoint

Three flavors of serving, in increasing operational sophistication:

FastAPI + uvicorn — write your own server. Maximum flexibility, minimum infrastructure. Great for prototypes, internal tools, and small-scale production.
TorchServe — managed inference server (AWS + Meta). Built-in batching, model versioning, metrics, multi-model serving. The right choice for "we have many models in production".
vLLM / TGI / SGLang — LLM-specific serving. Continuous batching, PagedAttention, optimized kernels. The right choice for serving language models.

The basics any serving setup needs

Load the model once on startup, not per request.
Set model.eval() + use torch.inference_mode() for every request.
Batch requests when latency budget allows — much better throughput.
Avoid CPU-GPU sync inside the request — pre-allocate buffers, use non_blocking transfers.
Have a health endpoint for orchestrators to know if the service is alive.
Have a metrics endpoint (latency, throughput, error rate, GPU utilization).

Code

FastAPI — the simplest HTTP server·python

# pip install fastapi uvicorn[standard]
from fastapi import FastAPI
from pydantic import BaseModel
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

app = FastAPI()

# Load once at startup
model_name = "distilbert-base-uncased-finetuned-sst-2-english"
tok = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
labels = ['NEGATIVE', 'POSITIVE']

class Req(BaseModel):
    text: str

class Resp(BaseModel):
    label: str
    score: float

@app.post("/predict", response_model=Resp)
@torch.inference_mode()
async def predict(req: Req) -> Resp:
    inputs = tok(req.text, return_tensors='pt', truncation=True, max_length=512).to(device)
    logits = model(**inputs).logits
    probs = logits.softmax(-1)[0]
    idx = int(probs.argmax())
    return Resp(label=labels[idx], score=float(probs[idx]))

@app.get('/health')
async def health(): return {'status': 'ok'}

# Run: uvicorn app:app --host 0.0.0.0 --port 8000

TorchServe — the production-grade option·python

# Bash, not Python — TorchServe is a separate process

# 1. Package model
# torch-model-archiver \
#     --model-name resnet50 \
#     --version 1.0 \
#     --model-file resnet_model.py \
#     --serialized-file resnet50.pth \
#     --handler image_classifier \
#     --export-path model_store \
#     --force

# 2. Start server
# torchserve --start --model-store model_store --models resnet50=resnet50.mar

# 3. Call API
# curl -X POST http://localhost:8080/predictions/resnet50 -T image.jpg

# Built-ins: batching, scaling, metrics, model versioning, multi-model
# Drawback: more setup vs FastAPI; opinionated about the handler interface

vLLM — the LLM-shaped option·python

# pip install vllm
# vLLM is the de facto LLM serving runtime — continuous batching, PagedAttention,
# OpenAI-compatible API.

# Start a server (CLI)
# python -m vllm.entrypoints.openai.api_server \
#     --model meta-llama/Llama-3.2-3B-Instruct \
#     --host 0.0.0.0 --port 8000

# Or programmatically
from vllm import LLM, SamplingParams

llm = LLM(model="gpt2")
sampling = SamplingParams(temperature=0.7, max_tokens=64)

prompts = ["Once upon a time", "The capital of France is"]
outputs = llm.generate(prompts, sampling)

for output in outputs:
    print(output.prompt, '→', output.outputs[0].text)

Naive request batching — when latency budget allows·python

import asyncio
import torch
from collections import defaultdict
from fastapi import FastAPI

app = FastAPI()
queue = asyncio.Queue()

async def batched_worker():
    while True:
        # Pull up to 32 requests OR wait 20ms, whichever first
        first = await queue.get()
        batch = [first]
        try:
            for _ in range(31):
                more = await asyncio.wait_for(queue.get(), timeout=0.02)
                batch.append(more)
        except asyncio.TimeoutError:
            pass

        # Run model.forward on the whole batch
        inputs = [b['input'] for b in batch]
        with torch.inference_mode():
            outputs = model(inputs)              # batched call

        # Resolve each request's future
        for b, out in zip(batch, outputs):
            b['future'].set_result(out)

@app.on_event("startup")
async def start_worker():
    asyncio.create_task(batched_worker())

@app.post("/predict")
async def predict(text: str):
    fut = asyncio.get_event_loop().create_future()
    await queue.put({'input': text, 'future': fut})
    return await fut

Serving — FastAPI, TorchServe, vLLM

From .pth to HTTP endpoint

The basics any serving setup needs

Code

External links

Exercise

Progress

Comments 0