TGI: Pull, Run, Tune

~30 min · serving, tgi

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The 5-minute startup

TGI ships as a Docker image. The standard recipe: pick a model, mount a volume for weights cache, expose port 8080, set --model-id. The container pulls weights on first start (slow), then warm-starts on subsequent runs (fast).

Knobs you'll actually tune

--model-id — HF repo id or local path.
--quantize — awq | gptq | bitsandbytes | bitsandbytes-nf4 | fp8. Mix-and-match with the right model variant.
--max-concurrent-requests — how many requests can be in flight simultaneously. Default is conservative; bump for higher throughput.
--max-input-length, --max-total-tokens — pre-allocates KV cache. Higher = more GPU memory budget per request, fewer concurrent requests.
--num-shard — tensor-parallel shards across GPUs.

Endpoints that come for free

POST /generate, POST /generate_stream, POST /v1/chat/completions (OpenAI-compatible), GET /info, GET /health, GET /metrics (Prometheus).

Code

Run TGI with one command·bash

# Single-GPU 7B model
docker run --gpus all --shm-size 1g -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --quantize bitsandbytes-nf4 \
  --max-concurrent-requests 64

# In another terminal:
curl http://localhost:8080/info | python -m json.tool

TGI as an OpenAI-compatible endpoint·bash

# TGI exposes /v1/chat/completions out of the box.
curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "tgi",
    "messages": [{"role":"user","content":"Hello"}],
    "max_tokens": 50
  }'

External links

Exercise

Pull the latest TGI image. Run a 1-3B instruct model with --quantize bitsandbytes-nf4. Hit /info, /health, /metrics. Send a chat completion via the OpenAI-compatible endpoint. Then stop the container, restart it, and verify weights warm-cache from the volume.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousWhy a Dedicated Inference Server Next →vLLM: PagedAttention and Continuous Batching

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.