What 'Dense' Actually Means

Definition: every token, every parameter

A dense model is a decoder-only Transformer where every token activates the entire set of feed-forward weights at every layer. When a token flows through an FFN block, it goes through all the neurons — no routing, no gating, no selective activation. If a model has 70B parameters, all 70B participate in processing every single token, every position, every batch.

Three properties that fall out of the definition

Total = active. What you see is what you compute. The parameter count on the card is the parameter count that fires.
Uniform compute per token. The word "the" costs the same as a complex code token. Capacity planning is straightforward — multiply tokens by per-token cost.
Deterministic activation path. Same input, same weights, same path through the network every time. No router decisions to debug.

What dense gives up, and what it keeps

Dense gives up the ability to scale total parameters faster than per-token compute. Doubling a dense model's parameter count roughly doubles the FLOPs per token. At frontier scale (100B+) this gets expensive fast — which is why MoE exists. But dense keeps almost everything else: simplicity, debuggability, fine-tuning ergonomics, and the entire mature serving ecosystem (vLLM, llama.cpp, MLX, TensorRT-LLM all started as dense-first stacks).

Where dense still wins in 2026

Below ~30B parameters, MoE overhead (router, load balancing, expert parallelism) often eats the efficiency gains. So small/medium open-weight models are still overwhelmingly dense — Llama 3 8B, Gemma 3 27B, Qwen3 14B/32B, Phi-4 14B, Mistral NeMo 12B. Dense is also the default for any local-deployment workload where you want predictable memory and quantization-friendly behavior.

Code

Dense FFN block in PyTorch (canonical shape)·python

import torch.nn as nn

class DenseFFN(nn.Module):
    def __init__(self, d_model, d_ffn):
        super().__init__()
        self.gate = nn.Linear(d_model, d_ffn, bias=False)
        self.up   = nn.Linear(d_model, d_ffn, bias=False)
        self.down = nn.Linear(d_ffn, d_model, bias=False)

    def forward(self, x):
        # SwiGLU-style — every token sees every weight, every time.
        return self.down(nn.functional.silu(self.gate(x)) * self.up(x))

Per-token FLOP estimate for a dense FFN layer·python

def dense_ffn_flops_per_token(d_model, d_ffn):
    # gate, up: each d_model x d_ffn
    # down:        d_ffn x d_model
    # Forward pass is 3 matmuls of those shapes.
    return 2 * (3 * d_model * d_ffn)
# Example: Llama 3 70B has d_model=8192, d_ffn=28672
# -> ~1.4 GFLOP per token per layer, just in the FFN block.

Exercise

Open the config.json of any small dense model on Hugging Face (Llama 3 8B, Qwen3 4B, Phi-4 14B). Find d_model (hidden_size) and d_ffn (intermediate_size). Compute the FFN FLOPs per token using the formula above, multiply by num_hidden_layers, and that gives you a rough dense FFN compute budget per token for that model.