The most cited prompt trick of the GPT-3 era
Adding "Let's think step by step" to a prompt was the original Chain-of-Thought (CoT) finding: it raised math-word-problem accuracy on small models by double digits. The technique survived into 2026 but its leverage shifted. On a frontier model, CoT helps in a narrower band — and on the wrong tasks, it hurts.
Where CoT still helps
- Multi-step problems where the steps must align (math, logic, planning, debugging traces).
- Tasks where the model has been overshooting — guessing without checking.
- Non-reasoning models that don't think internally.
Where CoT hurts
- Trivial tasks (classification, formatting). The thinking burns tokens for no quality gain.
- Reasoning models with native thinking. "Think step by step" double-thinks.
- Tasks where you need a tight, structured output. The thinking pollutes the response.
The compromise: structured CoT
Ask the model to think inside a tagged region the consumer ignores: <thinking>...</thinking> followed by <answer>...</answer>. You get the reasoning trace for debugging without polluting the output schema.