chain-of-thought

Overview

Chain-of-thought (CoT) prompting gets language models to reason through problems step by step before answering, instead of jumping straight to a final answer. The mechanism is straightforward: by generating intermediate reasoning tokens, the model gives itself more computation per problem and a scratchpad to hold partial results. On arithmetic, logic puzzles, and multi-hop questions, this reliably produces large accuracy gains over direct answering — it's one of the most replicated findings in prompting research.

The two flavors are zero-shot CoT ("Let's think step by step" appended to the prompt) and few-shot CoT (demonstrations showing worked reasoning). Few-shot CoT with well-crafted demonstrations generally outperforms zero-shot, but zero-shot is a strong, nearly-free baseline. The modern ecosystem extends the idea: self-consistency samples multiple chains and takes the majority answer; least-to-most decomposes problems into subproblems; and reasoning is now often trained in (RL on verifiable reasoning traces) rather than merely prompted.

CoT is inference-time compute made legible: you pay in tokens, you gain in accuracy, and you get a trace you can inspect. The art is knowing when the trade is worth it.

When to use

• Math word problems, logic puzzles, and any task where the answer requires combining multiple facts.
• Multi-hop question answering where evidence must be chained across sources.
• Code generation for non-trivial algorithms — reasoning about the approach before writing code.
• Debugging model errors: a visible reasoning trace shows where the model's logic went wrong.
• Any accuracy-critical task where extra inference-time tokens are an acceptable cost.
• Evaluation and red-teaming, where traces reveal failure modes that final answers hide.

Core concepts

• Reasoning as compute: each generated token is computation. CoT works because it lets the model spend tokens on intermediate steps rather than forcing the answer out of a single forward pass.
• Zero-shot CoT: appending a phrase like "Let's think step by step" or "Reason carefully before answering." Costs nothing to try; typically adds a few points on reasoning benchmarks.
• Few-shot CoT: demonstrations with worked solutions. The demonstrations teach both the reasoning style and the output format. Most effective when demonstrations cover diverse problem structures.
• Self-consistency: sample N reasoning chains at temperature > 0, take the majority final answer. Trades compute for accuracy; particularly effective when individual chains are noisy but the modal answer is right.
• Faithfulness caveat: the generated chain is a rationalization the model produces, not a guaranteed record of its internal computation. Chains can contain plausible-sounding steps that don't actually determine the answer. Don't treat them as proofs.
• Answer extraction: with CoT, parse the final answer from the trace (e.g., "Therefore, the answer is X"). Standardize the extraction format in demonstrations to make parsing reliable.
• Least-to-most: decompose a hard problem into easier subproblems, solve sequentially, building on answers. Helps when the full reasoning chain is too long to hold in one pass.
• Program-aided reasoning: offload exact computation to code (the model writes and runs programs) instead of doing arithmetic in prose. Eliminates the most common CoT failure mode — arithmetic slips.

Practical workflow

1. Baseline without CoT. Measure direct-answer accuracy first so you know what the reasoning is buying you.
2. Try zero-shot CoT. Add an explicit reasoning instruction. If it helps meaningfully, you have a cheap win.
3. Craft few-shot demonstrations. Write 4–8 worked examples with clear, correct reasoning steps in a consistent format. Include at least one example demonstrating how to handle a trick or edge case.
4. Standardize the answer format. End every demonstration with an unambiguous final-answer marker. Your parser depends on this.
5. Add self-consistency for critical tasks. Sample 5–10 chains, majority-vote the extracted answers. Measure the accuracy-per-dollar trade-off.
6. Consider program-aided variants. For math-heavy tasks, have the model write code and execute it rather than reasoning in prose — strictly better where execution is available.
7. Evaluate the chains, not just answers. Sample traces and check: are steps actually valid? Does the final answer follow from the steps? Invalid-but-lucky chains indicate the prompt needs work.

Checklist for a CoT deployment:

• Accuracy delta over direct answering measured on held-out data.
• Answer parser tested against real traces (not just the clean demonstrations).
• Token budget accounted for: CoT multiplies output tokens significantly.
• Trace sampling in production for ongoing quality monitoring.
• Decision documented on when CoT applies (routed) vs. always-on.

Common pitfalls

• Assuming faithfulness. A fluent reasoning chain can mask a wrong answer reached for the wrong reasons — or a right answer reached by luck. Verify steps independently on a sample.
• Demonstrations that leak the answer pattern. If all few-shot examples have answer "C" or follow an identical template, the model learns the template, not the reasoning.
• CoT on non-reasoning tasks. For memorization, style, or simple lookup tasks, CoT adds tokens and sometimes hurts (the model talks itself into wrong answers). Apply it where intermediate computation helps.
• Unparseable traces. Free-form reasoning without a consistent final-answer format makes automated evaluation and downstream use painful. Enforce the format.
• Ignoring cost. CoT can 5–10x output tokens. For high-volume, low-margin tasks, distill the CoT behavior into the model (train on traces) or reserve CoT for hard cases via routing.
• Prompting a weak reasoner. CoT amplifies reasoning ability but can't create it. Small or non-instruction-tuned models often produce incoherent chains — verify the base capability first.
• Arithmetic in prose. Letting the model do multi-digit arithmetic in the trace instead of using code execution. The most avoidable CoT failure.
• No routing. Applying CoT uniformly when only 20% of queries need it. Route by difficulty — a classifier or confidence heuristic — and save the tokens.