C.W.K.
Stream
Lesson 08 of 12 · published

Emergent Capabilities and Test-Time Compute

~10 min · emergence, test-time-compute

Level 0Token
0 XP0/94 lessons0/10 achievements
0/120 XP to next level120 XP to go0% complete

Emergent capabilities are abilities that appear sharply at a certain scale — present in large models, absent in smaller ones, with the transition not gradual but rapid.

Classic examples

  • In-context learning. GPT-3 (175B) could pick up tasks from few examples in the prompt; GPT-2 (1.5B) couldn't reliably.
  • Chain-of-thought reasoning. Above a certain scale, prompting a model to "think step by step" produces dramatically better math/reasoning answers. Below that scale, CoT prompting changes little.
  • Code generation from descriptions. Tiny models can complete code but rarely synthesize functional programs from English specs; large models do this routinely.
  • Multilingual transfer. Training mainly on English produces models that nevertheless work well across languages once parameters cross some threshold.

Test-time compute scaling

A more recent development: models like OpenAI o1, DeepSeek-R1, and Gemini 2.5 Pro thinking-mode can be allocated more inference compute per query — visible chain-of-thought, multi-attempt reasoning, internal verification. This lets a fixed model improve on hard tasks by spending more compute at inference time, breaking the assumption that "model quality = parameter count alone."

Recent research questions whether some "emergent" capabilities are truly sudden or artifacts of evaluation — discontinuities in metrics, not in the underlying ability. The cleaner conclusion: model capability is a function of scale × data × post-training × test-time compute, with all four interacting.

Code

Test-time compute knob (Anthropic API)·python
import anthropic

client = anthropic.Anthropic()

# Allocate up to 80K thinking tokens before the answer
response = client.messages.create(
    model="claude-3-7-sonnet-20250219",
    max_tokens=128_000,
    thinking={"type": "enabled", "budget_tokens": 80_000},
    messages=[{"role": "user", "content":
        "Find x such that 3^x + 5^x = 8^x."}]
)
# The same model produces dramatically better answers on hard math
# when given more thinking budget. Tune budget_tokens per task.

External links

Exercise

Take a hard reasoning prompt (e.g., a Project Euler problem). Run it through Claude 3.7 Sonnet with thinking_budget = 0, 4K, 16K, 64K. Plot answer correctness vs. thinking budget. Where does the curve flatten? What does that tell you about the right default budget for your tasks?

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.