Autoregressive Generation: One Token at a Time

~10 min · generation, decoding

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

A trained causal LM produces one token at a time. The procedure is:

Run the prompt through the model. Take the logits for the last position.
Sample a token from those logits (greedy, top-k, top-p, or temperature-scaled).
Append the new token to the sequence.
Repeat from step 1 until you hit an EOS token, a stop sequence, or a maximum length.

This loop is fundamentally serial. Token 5 cannot be predicted until token 4 has been chosen. Even with infinite GPUs, you cannot parallelize across the time axis during generation — only across the batch axis (multiple independent generations) and the depth axis (within a single forward pass). This single fact is what drives KV-cache, speculative decoding, and continuous batching: all are attempts to soften the impact of inherent sequentiality.

Code

Bare-bones generate loop·python

def generate(model, prompt_ids, max_new_tokens=64,
             temperature=0.7, top_p=0.9, eos_id=None):
    ids = prompt_ids
    for _ in range(max_new_tokens):
        logits = model(ids)[:, -1, :]                  # last position
        if temperature > 0:
            logits = logits / temperature
            # nucleus sampling
            sorted_logits, sorted_idx = logits.sort(descending=True)
            probs = F.softmax(sorted_logits, dim=-1)
            cumulative = probs.cumsum(dim=-1)
            mask = cumulative > top_p
            mask[..., 1:] = mask[..., :-1].clone()
            mask[..., 0] = False
            sorted_logits = sorted_logits.masked_fill(mask, float('-inf'))
            next_id = sorted_idx.gather(
                -1,
                torch.multinomial(F.softmax(sorted_logits, dim=-1), 1)
            )
        else:
            next_id = logits.argmax(-1, keepdim=True)
        ids = torch.cat([ids, next_id], dim=1)
        if eos_id is not None and next_id.item() == eos_id:
            break
    return ids

External links

Hugging Face — text generation strategies

Exercise

Implement the generate loop above without any optimization. Generate 100 tokens at temperature=0.7 from a small model. Measure tokens-per-second. Then enable model.config.use_cache and KV caching — measure again. The speedup at modest sequence lengths should be 5-10×.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousPre-Training Objectives: Next-Token vs Masked LM Next →Training Data: Trillions of Tokens

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.