Cloud Serving: vLLM & TGI

vLLM (the production default)

vLLM is the go-to for high-throughput LLM serving. Two key innovations:

PagedAttention: manages KV-cache memory like an OS pages virtual memory. Eliminates fragmentation, dramatically increases concurrent capacity.
Continuous batching: processes multiple requests in the same forward pass, packing them in as slots free up. Massive throughput gain over naive sequential serving.

vLLM speaks the OpenAI-compatible API out of the box, so you can point any OpenAI SDK client at it with zero code change.

vLLM with LoRA — no merge needed

vLLM can serve LoRA adapters directly on top of the base model and hot-swap between them per request. This is the right setup when you serve many fine-tuned variants on the same base (multi-tenant, per-customer specialization).

TGI (Text Generation Inference)

Hugging Face's own inference server. Particularly good for tight Hub integration. Generally similar throughput to vLLM, slightly different ergonomics.

Code

vLLM serving — OpenAI-compatible API·bash

# Install
pip install vllm

# Serve your merged model with OpenAI-compatible API
python -m vllm.entrypoints.openai.api_server \
    --model ./merged-model \
    --port 8000 \
    --tensor-parallel-size 1 \
    --max-model-len 4096

# Now use with the OpenAI SDK!
# from openai import OpenAI
# client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

vLLM with multiple LoRA adapters (no merge)·python

# Server side:
# python -m vllm.entrypoints.openai.api_server \
#     --model meta-llama/Llama-3.1-8B-Instruct \
#     --enable-lora \
#     --lora-modules medical=./adapter-medical legal=./adapter-legal

# Client side: select adapter per request via the model field
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

medical = client.chat.completions.create(
    model="medical",  # uses the medical LoRA
    messages=[{"role": "user", "content": "What is hypertension?"}],
)

legal = client.chat.completions.create(
    model="legal",    # uses the legal LoRA on the same base
    messages=[{"role": "user", "content": "What's force majeure?"}],
)

TGI via Docker (Hugging Face stack)·bash

docker run --gpus all -p 8080:80 \
    -v ./merged-model:/model \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id /model \
    --quantize bitsandbytes-fp4 \
    --max-input-tokens 2048 \
    --max-total-tokens 4096

Exercise

Set up vLLM locally serving a merged 7B model with the OpenAI-compatible API. Hit it with a 50-request load test (use aiohttp or a quick script). Measure P50, P95, P99 latency. Compare to a single-request synchronous baseline — the throughput multiplier from continuous batching should surprise you.