C.W.K.
Stream
More →
Lesson 03 of 04 · published

Loading FLUX Without the Repo

~13 min · flux, mps, model-loading, transformers, war-story

Level 0Tool Renter
0 XP0/33 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
"The checkpoint you have is one piece of the model. The other pieces have to come from somewhere — and figuring out where is half the work."

A Checkpoint Is Not Always the Whole Model

With older image models, the file you downloaded was the whole thing: backbone, text encoder, VAE, all in one checkpoint. Transformer-era models broke that assumption too. A FLUX checkpoint shared in the community format is often transformer-only — it contains the DiT backbone and nothing else. The text encoders (a CLIP and a large T5) and the VAE are simply not in the file. Try to load it like an old all-in-one checkpoint and you get a model with no way to read a prompt.

Source the Missing Pieces Explicitly

The fix is to load the transformer from the local file and source the auxiliary components — the T5, the CLIP, the VAE — from the model's base repository definition. In practice: load the transformer with a from-single-file path, then construct the full pipeline supplying that transformer while letting the base components come from their known definitions. The local file gives you the part that's actually local; the rest is assembled around it.

Know which parts of a model are local and which are referenced. A modern model is often a transformer file plus a set of standard auxiliary components. Treat the checkpoint as one slot in a larger assembly, not the whole machine. The loader's job is to fill every slot, sourcing each from wherever it actually lives.

The First-Run Trip: Missing Tokenizer Dependencies

There's a sharp edge here that bites everyone once. The large T5 text encoder needs specific tokenizer libraries installed to function — and if they're absent, the failure doesn't say 'install these libraries.' It says something obscure about being unable to load a tokenizer or convert it. The real cause is two missing packages in the environment. This is the canonical first-run trip for this model family: the error message points at a symptom, and the fix is a dependency you didn't know you needed.

Encapsulate the family's complexity in one place. All the FLUX-specific assembly — the transformer-local load, the base-component sourcing, the tokenizer requirement — lives inside one backbone module. Callers ask for 'a FLUX backbone' and never see the assembly. The complexity is real, so you contain it behind the module boundary rather than leaking it to every caller.

MPS Is Not CUDA, and That's Usually Fine

All of this runs on Apple Silicon's Metal backend, not on the CUDA most diffusion code assumes. Most of the time the difference is invisible — the same model code runs on the Metal device. But the abstraction isn't perfect: some operations behave subtly differently, some dtype choices matter more, and the unified-memory model (from the OOM lesson) changes the memory math. The discipline is to treat the compute backend as a known variable, test on the actual device, and never assume CUDA-tuned advice transfers unchanged.

Most diffusion knowledge assumes CUDA; verify it on your actual backend. Tutorials, forum answers, and default code paths are overwhelmingly written for CUDA. On a Metal device, most of it works — but the exceptions are real and quiet. Test the actual path on the actual hardware rather than trusting that CUDA advice carries over. The 90% that transfers makes the 10% that doesn't extra surprising.

Pippa's Confession

I assumed a checkpoint was self-contained because every model I'd 'known' before was. FLUX taught me to ask, every time, 'what's actually in this file?' before assuming. And the tokenizer trip humbled me — I burned real time on an obscure error that turned out to be two missing packages. Now my first move on any new model family is to find out which pieces are in the file, which are referenced, and what the environment secretly needs. Assume nothing about completeness.

Code

Local transformer + referenced base parts·python
# A FLUX checkpoint is often TRANSFORMER-ONLY. The text encoders and VAE
# are not in the file -- they're sourced from the base model definition.
from diffusers import FluxTransformer2DModel, FluxPipeline

# 1. Load the part that's actually local: the DiT transformer.
transformer = FluxTransformer2DModel.from_single_file(local_checkpoint_path)

# 2. Assemble the full pipeline -- transformer from the local file,
#    T5 + CLIP + VAE sourced from the base model definition.
pipe = FluxPipeline.from_pretrained(
    base_model_definition,        # supplies T5-XXL, CLIP-L, VAE
    transformer=transformer,      # your local transformer slots in
)

# First-run trip: the T5 tokenizer needs sentencepiece + protobuf in the env.
# Without them the error is obscure ('cannot load/convert tokenizer'),
# and the real fix is: install those two packages.

# All of this complexity lives inside ONE backbone module.
# Callers ask for a FLUX backbone; they never see this assembly.

External links

Exercise

Take any model you've loaded from a single file and investigate: is it actually self-contained, or does the loading code quietly pull auxiliary components from somewhere else? Trace where each component comes from. If you've only ever loaded all-in-one checkpoints, find a transformer-era model's loading docs and list which pieces are local vs referenced.
Hint
The tell is in the load call: a single from_single_file that returns a complete pipeline is all-in-one; a from_single_file for one component followed by a from_pretrained that supplies the rest is the local-plus-referenced pattern.

Progress

Progress is local-only — sign in to sync across devices.
← PreviousAccept Concurrently, Run One at a TimeNext →Ground Truth and the Dev Mirror
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.