~14 min · five-modules, pipeline, vae, sampler, backbone
Level 0Tool Renter
0 XP0/33 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
"Once you see the five boxes, you can never unsee them. Every diffusion model is the same five boxes wired slightly differently."
The Pipeline, Drawn Honestly
A diffusion image generator looks like one magical act — prompt in, picture out — but it's five distinct jobs handed off in sequence. Name them and the magic becomes mechanics:
Text encoder — turns your prompt into conditioning vectors the network can attend to. CLIP, or CLIP + T5, depending on the family.
Backbone — the denoising network itself. U-Net (older), DiT or MMDiT (transformer era). This is the brain of the generation.
Sampler / scheduler — the loop that repeatedly asks the backbone "what's the noise here?" and steps the latent toward a clean image. Euler, DPM++, flow-match.
Conditioning attach — the extras bolted onto the backbone: LoRA, ControlNet, IP-Adapter, embeddings, inpaint masks.
VAE — translates between the compressed latent space (where all the work happens) and actual pixels (what you see).
Why Five and Not One
Each box changes independently across model families. SDXL and FLUX share the VAE concept but use different VAE weights. They use different backbones entirely. They share some samplers and not others. If these five jobs lived in one function, every model family would fight every other family inside that function. As five boxes, each family just supplies the boxes it needs and reuses the rest.
Separate along the lines where change happens. The right module boundaries aren't arbitrary — they fall exactly where different model families differ. Text encoders vary, so the text encoder is a box. Backbones vary the most, so the backbone is the most important box. Boundaries follow the fault lines of change.
The Interface Is the Real Product
The boxes matter less than the seams between them. The sampler doesn't care whether the backbone is a U-Net or a DiT — it only needs "given a noisy latent and a timestep, predict the noise." That contract is the interface. As long as every backbone honors it, the sampler never changes. The interfaces are what make the boxes swappable; the boxes are just implementations behind them.
Design the seam before the implementation. The hard, valuable work is defining what one box promises another — the method signature, the tensor shapes, the contract. Once the seam is right, multiple implementations slot in behind it. Get the seam wrong and no implementation will save you.
How a Recipe Flows Through the Boxes
A generation request enters as a recipe and flows through the boxes in order: the text encoder turns the prompt into conditioning, the conditioning-attach layer folds in any LoRA or ControlNet, the sampler runs its loop calling the backbone at each step, and finally the VAE decodes the finished latent into pixels. The recipe never names the boxes — it just says what it wants, and the engine routes it through the right five.
The first time Dad drew these five boxes for me, I'd already 'known' how diffusion worked for ages — but I'd known it as one blurry process. Seeing it as five hand-offs with named contracts between them was the moment it stopped being a black box and became a system I could reason about, modify, and extend. Naming the seams is what turned knowledge into understanding.
Code
Five protocols, many implementations·python
# The five boxes as protocols (interfaces). Implementations vary
# per model family; these contracts do not.
from typing import Protocol
import torch
class TextEncoder(Protocol):
def encode(self, prompt: str) -> torch.Tensor: ... # box 1
class Backbone(Protocol):
def predict_noise(self, latents: torch.Tensor,
t: int, cond: torch.Tensor) -> torch.Tensor: ... # box 2
class Sampler(Protocol):
def sample(self, backbone: Backbone, cond: torch.Tensor,
steps: int) -> torch.Tensor: ... # box 3
class Conditioning(Protocol):
def attach(self, backbone: Backbone) -> Backbone: ... # box 4 (LoRA/ControlNet)
class VAE(Protocol):
def decode(self, latents: torch.Tensor) -> torch.Tensor: ... # box 5
# SDXL supplies one set of implementations, FLUX another.
# The protocols above never change — that's what makes them swappable.
The recipe says what; the boxes do how·python
# A recipe flowing through the five boxes, in order.
def run(recipe, te: TextEncoder, bb: Backbone,
sampler: Sampler, cond: Conditioning, vae: VAE):
conditioning = te.encode(recipe.prompt) # box 1: text -> vectors
bb = cond.attach(bb) # box 4: fold in LoRA/ControlNet
latents = sampler.sample(bb, conditioning, # box 3 drives box 2 in a loop
steps=recipe.steps)
image = vae.decode(latents) # box 5: latent -> pixels
return image
# The recipe named none of the boxes. The engine wired them.
Take a multi-stage system you understand (a build pipeline, a request/response cycle, a game loop). Break it into its distinct boxes. For each seam between two boxes, write the one-line contract: what does the upstream box promise the downstream box? The clarity of those contracts is the health of your architecture.
Hint
If you can't state a seam's contract in one sentence, the two boxes are probably still fused. A clean seam has a contract you can write down without saying 'it depends.'
Progress
Progress is local-only — sign in to sync across devices.