Admin Cheat Sheet

The knobs that actually move the needle in production

Knob	What to do	Why it works
Datatype	Prefer FP16 / BF16 weights; keep FP32 accumulation only when stability demands	Hits the matrix unit / Tensor Core path
Hidden size	Round to 64 or 128	Fills 128×128 block tile perfectly; avoids padding-fallback
Memory placement	Weights → private; activations → shared only if CPU must read	Keeps hot data in on-chip SRAM
Warm-up pass	Run one dummy inference after loading weights	Caches the library's best algorithm choice
Batch size	Push as high as memory allows for inference	GEMV → GEMM, bandwidth-bound → compute-bound
Algorithm caching	Persist cuBLASLt / autotune choices across runs	Skip 1–10s of cold-start algo search

Common face-plants & quick fixes

Symptom	Probable cause	Fix
GEMM < 40% F32 utilization	Hidden size not a multiple of 16	Pad M and/or K to 64 / 128
Throughput halves overnight	Forgot to enable reduced-precision flag after a library upgrade	Re-enable, rebuild, document the flag
Big gaps between command buffers	CPU tokenizer or I/O stalling the queue	Async pre-tokenize; keep enough work queued
Library swaps to GEMV path silently	Tiny m or k (e.g. head_dim = 32 for single-token decode)	Fuse heads or rewrite KV-cache to keep batch ≥ 8
Inference latency varies 5×	Library re-running heuristic per shape	Cache algorithm choice; sort requests by shape

Bottom line

GEMM is 90% of the math in a transformer. Feed the library shapes it loves (padded, batched, reduced-precision, hidden-size on the right tile boundary), glance at utilization counters when something slows, let BLAS do the heavy lifting. Hand-rolled kernels are for learning (this quest!) and for shapes the library genuinely doesn't cover well — not as a default.

Code

Production checklist as a Python pre-flight script·python

import torch

def preflight_gemm(model_hidden_size, batch_size, dtype):
    issues = []

    if model_hidden_size % 64 != 0:
        issues.append(
            f'hidden_size={model_hidden_size} not multiple of 64 — '
            'expect 20-40% perf loss vs padded')

    if dtype == torch.float32 and torch.cuda.is_available():
        cap = torch.cuda.get_device_capability()
        if cap >= (8, 0):  # Ampere or newer
            issues.append(
                f'FP32 on Tensor-Core-capable GPU (cc {cap}); '
                'consider FP16/BF16 for ~2× throughput')

    if batch_size < 8:
        issues.append(
            f'batch_size={batch_size} → likely GEMV path. '
            'Bandwidth-bound; pad batch if latency budget allows')

    return issues

# Example:
for issue in preflight_gemm(model_hidden_size=4099, batch_size=1, dtype=torch.float32):
    print('WARN:', issue)

Cache cuBLASLt heuristic choice across runs·python

import os
import torch

# Persist autotune choices to disk — ~5-10s saved per cold start
os.environ['CUBLASLT_LOG_FILE'] = '/tmp/cublaslt.log'
os.environ['CUBLASLT_HEURISTIC_CACHE_DIR'] = '/tmp/cublaslt_cache'

# Equivalent for PyTorch's own autotuner
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True

# Then your model's first warmup pass populates the cache; subsequent
# runs reuse it instead of re-searching.

Exercise

On any model you've actually deployed (or your favorite open-source one), audit the production-knob table item by item. Find one place where a hidden_size, batch_size, or dtype choice is leaving performance on the table. Fix it (or at least document it). Even one fix that improves a hot kernel by 20-40% is worth more than weeks of micro-tuning elsewhere — and you now have the vocabulary to explain why to your team.