Text Generation

Generation is just next-token prediction in a loop

A causal language model does one thing: given everything so far, predict the next token. generate() wraps that single step in a loop — predict a token, append it, feed the longer sequence back in, repeat until max_length or an end-of-text token. Loading a generator is the same one-liner you've seen all track (GPT2CausalLM.from_preset(...)), and a string prompt in gives a string completion out.

The sampler is where the personality lives

Here's the part people underestimate: the model's raw output is a probability distribution over the whole vocabulary, every single step. How you pick from that distribution — the sampler — changes everything. temperature flattens or sharpens the distribution: near 0 you always grab the single most likely token (deterministic, repetitive, safe); near 1.0 you let unlikely tokens have a real shot (creative, sometimes incoherent). top_k restricts the choice to the k most likely candidates, and top_p (nucleus sampling) keeps just enough top candidates to cover probability mass p. As the Code section shows, you can pass a sampler to compile() or let generate() use the default — but knowing the knob exists is what separates a usable demo from a frustrating one.

Code

Generate text and tune the sampler·python

import keras_hub

# Load GPT-2 for text generation
gpt2 = keras_hub.models.GPT2CausalLM.from_preset("gpt2_base_en")

# Generate with various sampling strategies
output = gpt2.generate(
    "The future of AI is",
    max_length=100,
)
print(output)

# Control generation with temperature, top_k, top_p
gpt2.compile(sampler=keras_hub.samplers.TopKSampler(k=50, temperature=0.7))

Generation is just next-token prediction in a loop

The sampler is where the personality lives

Code

External links

Exercise

Progress

Comments 0