How RL Creates Reasoning Behavior — SFT, RLHF, GRPO

The training pipeline that makes a reasoning model

Modern reasoning models are produced by a multi-stage pipeline. The exact stages vary, but the canonical 2025–2026 sequence looks roughly like this:

Stage 1 — Pretraining (the base model)

Same as any other LLM: autoregressive next-token prediction on a giant corpus. This produces the raw "knows lots of things, but is not yet a useful assistant" base. Reasoning models do not require a special pretraining recipe at this stage; they typically start from the same base used for the non-reasoning sibling.

Stage 2 — Supervised fine-tuning (SFT)

The base is fine-tuned on a curated dataset of high-quality step-by-step reasoning examples. These can be human-written, distilled from a stronger model, or programmatically generated. SFT teaches the model the shape of good reasoning — when to break problems into steps, how to verify, when to backtrack.

Stage 3 — Reinforcement learning

This is where the magic happens. The model generates candidate solutions to problems with verifiable answers (math, coding, logic), and is rewarded for reaching correct answers. Various RL algorithms are used:

PPO (Proximal Policy Optimization) — standard, used in early RLHF.
GRPO (Group Relative Policy Optimization) — DeepSeek's RL recipe; eliminates the separate critic by computing advantages relative to a group of sampled responses.
DPO (Direct Preference Optimization) — simpler than PPO; learns directly from preference pairs without an explicit reward model.
RLVR (Reinforcement Learning from Verifiable Rewards) — uses programmatic verifiers (a math solver, a code interpreter) instead of a learned reward model.

Stage 4 — Optional rejection sampling and refinement

Generate many candidate responses with the RL-trained model, keep only the highest-quality ones, fine-tune again on those. This polishing step often improves the reasoning quality further with much less compute than continued RL.

The "aha moment" — emergent reasoning from pure RL

DeepSeek-R1-Zero demonstrated that you can skip the SFT stage entirely. Pure RL with outcome rewards on a base model can cause reasoning behavior — including self-verification, "wait, let me reconsider", and step-by-step decomposition — to emerge spontaneously. The model discovers reasoning strategies on its own when correctness is the only reward signal. This is one of the most surprising results in 2024–2025.

Code

GRPO advantage computation (sketch)·python

def grpo_advantage(rewards):
    # rewards: tensor of shape (group_size,) — N samples for the same prompt
    mean = rewards.mean()
    std  = rewards.std() + 1e-8
    return (rewards - mean) / std   # group-relative advantage; no critic needed

RLVR loop with a math verifier·python

def rlvr_step(model, prompts, verifier):
    rollouts = [model.generate(p) for p in prompts]
    rewards  = [1.0 if verifier(p, r) else 0.0 for p, r in zip(prompts, rollouts)]
    update_policy(model, rollouts, rewards)