GRPO and Constitutional AI: Alignment Beyond DPO

Two newer alignment recipes deserve attention: GRPO (DeepSeek's reasoning-model recipe) and Constitutional AI (Anthropic's safety recipe).

GRPO (Group Relative Policy Optimization)

Used to train DeepSeek-R1, a reasoning model competitive with OpenAI o1. The key insight: skip the reward model entirely; use rule-based rewards (does the math answer match? does the code pass tests?) and compare each generation against the group mean for the same prompt.

Sample G outputs per prompt (e.g., G=16) from the current policy.
Score each with rule-based rewards (correctness signals).
Compute the advantage: A_i = r_i − (1/G) Σ_j r_j — how much better than the group average.
Update the policy to increase the probability of above-average outputs.

No critic model needed. ~50% memory reduction vs PPO. Works exceptionally well for tasks where you can mechanically check correctness — math, code, structured output.

Constitutional AI (Anthropic)

Used by all Claude models. Replaces most human preference labeling with AI feedback guided by a written set of principles (the "constitution," ~16 in the original paper).

Red-team and self-revise. Generate harmful prompts → have the model respond → ask the model to critique its own response against the constitution → revise. Fine-tune on the revised responses.
RLAIF (RL from AI Feedback). Same as RLHF but the preference labels come from an AI (using the constitution) rather than humans. Scales beyond what human labelers can produce.

The win: alignment scales without proportionally scaling the human labeling effort, and the principles are written and inspectable rather than implicit in opaque human judgments.

Code

GRPO advantage computation·python

def grpo_advantage(rewards):
    # rewards: (G,) — rule-based scores for G generations from same prompt
    mean_r = rewards.mean()
    return rewards - mean_r           # A_i = r_i - group mean
# Then standard policy gradient with these advantages.
# No reward model, no critic, no value head.

Constitutional AI self-critique loop (sketch)·python

def constitutional_revise(model, prompt, constitution):
    response = model(prompt)
    critique_prompt = (
        f"Original prompt: {prompt}\n"
        f"Original response: {response}\n"
        f"Critique this response based on:\n{constitution}"
    )
    critique = model(critique_prompt)
    revise_prompt = (
        f"{critique_prompt}\n"
        f"Critique: {critique}\n"
        f"Now write a revised response addressing the critique."
    )
    return model(revise_prompt)
# Repeat to build a dataset; then SFT or RLAIF on (prompt, revised_response).

GRPO and Constitutional AI: Alignment Beyond DPO

GRPO (Group Relative Policy Optimization)

Constitutional AI (Anthropic)

Code

External links

Exercise

Progress

Comments 0