Repetition Penalty and Stop Conditions

Without help, generative models drift into repetition loops: "I think that I think that I think that..." This happens because the model trained on text where some patterns are common, and once it generates a fragment that fits a high-probability continuation pattern, it reinforces itself.

Three approaches to repetition

Repetition penalty (CTRL-style): divide logits of already-generated tokens by a penalty factor (e.g., 1.2). This makes them less likely to be picked again. Default for many open-source toolchains.
Frequency penalty (OpenAI): subtract a value proportional to the count of each token already generated. Stronger penalty for tokens that have appeared many times.
Presence penalty (OpenAI): subtract a flat value if the token has appeared at all (regardless of count). Encourages new vocabulary.

Stop conditions

Generation stops when (1) the model produces an end-of-sequence token (EOS), (2) a stop string is matched (e.g., a specific role marker like <|im_end|>), (3) the maximum number of tokens is reached, or (4) a custom logit processor halts. For chat, model-specific stop tokens are critical: Llama 3's <|eot_id|>, Mistral's </s>, GPT's <|im_end|>.

Code

Repetition penalty in PyTorch·python

def apply_repetition_penalty(logits, generated_ids, penalty=1.2):
    # logits: (B, vocab); generated_ids: (B, generated_so_far)
    for i in range(logits.size(0)):
        for tok_id in generated_ids[i].tolist():
            if logits[i, tok_id] > 0:
                logits[i, tok_id] /= penalty
            else:
                logits[i, tok_id] *= penalty
    return logits
# Hugging Face's `RepetitionPenaltyLogitsProcessor` does this efficiently.

Repetition Penalty and Stop Conditions

Three approaches to repetition

Stop conditions

Code

External links

Exercise

Progress

Comments 0