Health Checks, Metrics, and Restart Strategy

~22 min · serving, ops, monitoring

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

What to scrape

Both TGI and vLLM expose Prometheus metrics on a configurable port. The key series:

tgi_request_count / vllm_requests_total — throughput.
tgi_request_inference_duration — per-request latency histogram.
tgi_queue_size / vllm_num_requests_waiting — backlog. Alert when sustained > 0.
tgi_batch_current_size — live batch fullness.
GPU memory: scrape from nvidia-smi-exporter or DCGM exporter.

Health checks

GET /health on both. Returns 200 once weights are loaded. Use as the readiness probe in k8s; use GET /info to verify the running model id matches what your config thinks is live.

Restart strategy

Inference servers do leak memory occasionally on long-tailed workloads. Pin --restart=unless-stopped in Docker. In k8s use restartPolicy: Always and a liveness probe with a generous initial delay (model load can be minutes). Plan for cold-start: pre-load weights into the image or a persistent volume; never first-pull on a production node.

Code

k8s readiness + liveness probe (TGI / vLLM)·yaml

# Full pod spec elided.
spec:
  containers:
  - name: tgi
    image: ghcr.io/huggingface/text-generation-inference:latest
    args: ["--model-id", "/models/llama-3.1-8b", "--port", "80"]
    ports: [{containerPort: 80}]
    readinessProbe:
      httpGet: {path: /health, port: 80}
      initialDelaySeconds: 30
      periodSeconds: 10
    livenessProbe:
      httpGet: {path: /health, port: 80}
      initialDelaySeconds: 300   # generous: weights load is slow
      periodSeconds: 30
      failureThreshold: 3

Quick check before alerting on it·bash

# Hit the metrics endpoint and grep for the key series
curl -s http://localhost:8080/metrics | egrep 'request_count|queue_size|batch_current_size' | head -20

External links

Exercise

Add Prometheus scraping to your TGI or vLLM container. Run a soak test for 30 minutes (low constant traffic). Watch: queue size, batch size, request latency p50/p95, GPU memory. Identify which series would be your alerting candidates and at what thresholds.