vLLM for Throughput

What vLLM is for

vLLM is built around continuous batching and PagedAttention. The point is: when you have many concurrent requests, vLLM packs them into batches that share GPU compute and memory across requests, getting dramatically higher throughput than a single-request engine.

When to reach for vLLM

You're serving an internal API with multiple users hitting the same model.
You're running a batch job over thousands of prompts.
You have multi-GPU NVIDIA hardware and want sharded inference across cards.

If you're a single user on a single Mac, vLLM is overkill. Ollama is cleaner.

What you give up

Apple Silicon support is partial. vLLM is NVIDIA-first. Some paths work on Mac (CPU-only, slow), but you'd never reach for vLLM on a Mac in production.
Heavier setup. Python deps, CUDA versions, model paths — more knobs than Ollama.
Different memory model. vLLM allocates a contiguous KV-cache pool up front. The trade-off is fast scheduling, but you have to size it right.

Code

Run vLLM with one model (NVIDIA host)·bash

# Install vLLM (assumes CUDA 12.x and a GPU)
pip install vllm

# Serve a HuggingFace model on port 8000 with OpenAI-compat
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192

# Test
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role":"user","content":"Hello!"}],
    "stream": false
  }'

Throughput test — many concurrent requests·python

import asyncio, httpx, time

async def hit(client: httpx.AsyncClient, i: int):
    r = await client.post("http://localhost:8000/v1/chat/completions",
                          json={"model": "Qwen/Qwen2.5-7B-Instruct",
                                "messages": [{"role": "user",
                                              "content": f"Count to 10 — request #{i}"}],
                                "stream": False, "max_tokens": 80},
                          timeout=120.0)
    return r.json()["choices"][0]["message"]["content"]

async def main():
    async with httpx.AsyncClient() as c:
        t0 = time.time()
        # 50 concurrent requests
        results = await asyncio.gather(*(hit(c, i) for i in range(50)))
        dt = time.time() - t0
        print(f"50 requests in {dt:.1f}s → {50/dt:.2f} req/s")

asyncio.run(main())

What vLLM is for

When to reach for vLLM

What you give up

Code

External links

Exercise

Progress

Comments 0