Generation Configuration: Sampling, Beam, Constraints

generate() is a state machine, not a function

Under the hood, model.generate() is a token-by-token decoding loop driven by a GenerationConfig object. The config carries the strategy (do_sample, num_beams), the limits (max_new_tokens, min_new_tokens), the sampling distribution shaping (temperature, top_p, top_k, repetition_penalty), and the stopping criteria (eos_token_id, stop_strings).

Three strategies you'll actually use

Greedy (do_sample=False, num_beams=1): pick the argmax at every step. Deterministic. Often the right default for tool-calling, JSON, code completion.
Sampling (do_sample=True): nucleus + top-k + temperature. The default for chat. Tune temperature=0.7, top_p=0.9 as a starting point.
Beam search (num_beams=4): explores multiple candidate sequences. Useful for translation and summarization where there's a "single best" answer. Costly — num_beams × the work.

Stop conditions are not optional

Forgetting to set eos_token_id is the most common reason a model "loops forever." Modern instruct models often have multiple end-of-turn tokens (<|eot_id|>, <|end_of_turn|>, etc.) — pass them as a list. stop_strings matches arbitrary substrings post-decode (slower but flexible).

Code

Greedy vs sampling vs beam·python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Translate to Korean: The Hub is the registry."}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tok(prompt, return_tensors="pt").to(model.device)

# Greedy — deterministic
out_g = model.generate(**inputs, max_new_tokens=50, do_sample=False)

# Sampling — varied
out_s = model.generate(**inputs, max_new_tokens=50,
                       do_sample=True, temperature=0.7, top_p=0.9)

# Beam — exhaustive
out_b = model.generate(**inputs, max_new_tokens=50, num_beams=4, early_stopping=True)

for label, out in [("greedy", out_g), ("sample", out_s), ("beam", out_b)]:
    print(label, ":", tok.decode(out[0], skip_special_tokens=True)[-200:])

Stop on multiple EOS tokens (Llama-3 style)·python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

# Llama-3 has both eos_token and an eot (end-of-turn) marker
eos_ids = [tok.eos_token_id, tok.convert_tokens_to_ids("<|eot_id|>")]
print("stop ids:", eos_ids)

# Pass to generate(): eos_token_id=eos_ids

Exercise

On a 1-3B instruct model, run the same prompt 5 times each under: (a) greedy, (b) sample with temp=0.3, (c) sample with temp=0.9, (d) beam=4. Save the outputs. Compute character-level diversity (number of unique outputs / 5) for each strategy. Notice when greedy is desirable and when it's restrictive.