Compilation and Export Paths

From PyTorch model to deployed inference

You've trained a model. Now you need it to run somewhere — a server, a phone, an embedded device, an edge GPU. Three main paths:

torch.compile — fastest path on PyTorch-native targets. JIT-compiles your forward pass into a fused kernel. Works on the same device PyTorch supports.
ONNX — export to a vendor-neutral graph format that runs on ONNX Runtime, TensorRT, OpenVINO, CoreML, etc. Portable.
TorchScript — older PyTorch-native serialization. Mostly superseded by torch.compile for new code, still common in legacy.

Tip: For modern inference, the realistic tooling is: torch.compile on PyTorch servers, ONNX Runtime on cross-platform deployments, vLLM or TGI for LLM serving, Core ML for iOS, TensorFlow Lite for Android, llama.cpp for local quantized LLM inference.

Quantization

Reduce weight precision (FP32 → INT8 or INT4) for smaller models and faster inference, often with minimal accuracy loss. Three flavors: post-training quantization (cheap, no retraining), quantization-aware training (better accuracy, more complex), weight-only quantization (used in LLM deployment, e.g. AWQ, GPTQ).

Distillation

Train a small student model to mimic a large teacher's predictions. Common in production to reduce serving cost. The student often achieves 90-95% of teacher accuracy at 5-10% of the parameter count.

Principle: Inference is its own engineering discipline. The model that wins your eval is rarely the model that goes to production unmodified — quantization, distillation, and graph optimization usually trim it before deployment.

Code

ONNX export and Runtime inference·python

import torch
import onnxruntime as ort

model.eval()
dummy = torch.randn(1, 3, 224, 224)
torch.onnx.export(
    model, dummy, "model.onnx",
    input_names=["x"], output_names=["logits"],
    dynamic_axes={"x": {0: "batch"}, "logits": {0: "batch"}},
    opset_version=17,
)

sess = ort.InferenceSession("model.onnx",
                            providers=["CPUExecutionProvider"])
out = sess.run(None, {"x": dummy.numpy()})[0]
print(out.shape)

Quantize an LLM for local inference·bash

# Convert HF model to GGUF (llama.cpp format)
python convert-hf-to-gguf.py meta-llama/Llama-3.1-8B-Instruct

# Quantize to 4-bit
./quantize Meta-Llama-3.1-8B-Instruct.gguf model-q4_K_M.gguf q4_K_M

# Run via Ollama on Apple Silicon — Pippa's local fallback brain path
ollama create my-llama -f Modelfile
ollama run my-llama "Explain mixed precision in one paragraph"

From PyTorch model to deployed inference

Quantization

Distillation

Code

External links

Exercise

Progress

Comments 0