Why a Dedicated Inference Server

pipeline() is for prototyping

A pipeline() call re-tokenizes per request, allocates per request, can't share KV cache across requests, and gives you no control over batching. For one user it's fine. For ten concurrent users it's a stack of pipeline objects fighting over a single GPU.

What an inference server adds

Continuous batching — new requests join the in-flight batch every step instead of waiting for the previous one to finish.
PagedAttention / KV cache management — the engine page-faults attention tensors so your GPU memory holds many short conversations instead of a few long ones.
Streaming — SSE or token-by-token response from the very first server response.
Quantization at serve time — AWQ / GPTQ / bnb / fp8 / int4 selectable at startup.
Health, metrics, model management — /health, /metrics, /info endpoints baked in.

Two stacks: TGI and vLLM

HF's first-party server is text-generation-inference (TGI). The community standard is vLLM. They overlap; both are worth knowing. We'll cover TGI first because it composes natively with the Hub and the OpenAI-compatible router.

Code

Compare: pipeline vs TGI for 10 concurrent users·bash

# Pipeline approach: one Python process, sequential
# Throughput tops out around 2-5 req/s on a 7B model on a single GPU

# TGI approach: one TGI process, continuous batching
# Throughput on the same hardware: 30-100 req/s for the same workload

# Take this as illustrative, not prescriptive — measure on your hardware.

Spot-test concurrency with httpx·python

import httpx
import asyncio

async def hit(client, prompt):
    r = await client.post(
        "http://localhost:8080/generate",
        json={"inputs": prompt, "parameters": {"max_new_tokens": 50}},
        timeout=60,
    )
    return r.json()

async def main():
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*[hit(client, "ping") for _ in range(20)])
        print(len(results), results[0])

asyncio.run(main())

pipeline() is for prototyping

What an inference server adds

Two stacks: TGI and vLLM

Code

External links

Exercise

Progress

Comments 0