Optimum: Hardware-Specific Optimization

~20 min · ops, optimum

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

What optimum is for

optimum is HF's hardware-acceleration umbrella. Each subpackage targets a vendor / runtime: optimum.onnxruntime, optimum.intel (OpenVINO, Neural Compressor), optimum.habana (Gaudi), optimum.tpu, optimum.nvidia (TensorRT-LLM). Same Auto-class shape (ORTModelForCausalLM, OVModelForCausalLM, ...) so swapping runtimes is a one-line import change.

Two patterns

Export-time optimization — ORTModelForCausalLM.from_pretrained(repo, export=True) converts and saves an optimized graph.
Run-time optimization — optimum-cli onnxruntime quantize applies INT8 dynamic quantization, etc.

When to reach for it

If your serving target is anything other than "PyTorch on a NVIDIA GPU," optimum probably has a runtime for it. Mobile / embedded / Intel CPU / TPU / Gaudi / TensorRT — all converge on optimum's shape. The cost is one extra dependency and a per-target export step.

Code

Optimum + Intel OpenVINO for CPU inference·python

# pip install "optimum[openvino]"
from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

repo = "Qwen/Qwen2.5-1.5B-Instruct"
ov = OVModelForCausalLM.from_pretrained(repo, export=True)
tok = AutoTokenizer.from_pretrained(repo)

inputs = tok("Hello", return_tensors="pt")
out = ov.generate(**inputs, max_new_tokens=20)
print(tok.decode(out[0], skip_special_tokens=True))

External links

Exercise

Pick a runtime that matches your hardware (ONNX, OpenVINO, or TensorRT-LLM if you have NVIDIA). Convert a 1B model. Run a 100-prompt benchmark. Compare per-token latency vs the same model in transformers + PyTorch on the same machine.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousAccelerate: Multi-Device, Mixed Precision, FSDP Next →TRL: RLHF, DPO, and Preference Optimization

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.