Chain of thought (CoT)

A language model produces its answer one token at a time. Without intermediate steps, a multi-part task forces it to deliver the result in one leap. With chain of thought, the model first writes out the steps, such as quantity, discount tier and subtotal, and then derives the answer from them. Each written step stays in the context and feeds the tokens that follow.

The term comes from a 2022 paper by Jason Wei and colleagues at Google Research, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” They put a few worked examples into the prompt, each with a task, a written-out solution and the answer, and saw clear gains on arithmetic, commonsense and symbolic tasks. Takeshi Kojima and colleagues then showed that adding “Let’s think step by step” is often enough. Since 2024 there have been reasoning models, trained to produce such a chain on their own before answering.

For executives the term matters for three reasons. Chains of thought cost money, because the extra tokens are billed as output, and time, because the answer arrives later. And they read like a justification even though they do not reliably show how the model actually reached its result. If you base decisions on AI output, treat the chain as a debugging aid, not as evidence.

How chain of thought works.

At every step, a language model computes how likely each possible next token is. Everything already written feeds into that computation. When the model writes out an intermediate step, it can build on it in the next one. A hard task gets broken into several easier ones.

There are two basic forms. In the original version by Wei and colleagues, the prompt contains a few complete examples, each made up of an input, a chain of thought and an answer. In zero-shot CoT, introduced by Kojima and colleagues, there are no examples; an instruction like “think step by step” is enough to trigger a chain.

In practice, a good prompt keeps the chain and the result apart, with the answer in a final line or a fixed field. That way a program can extract the answer. The authors of the original paper are explicit on one point: the chain emulates human thinking, but whether the model is actually reasoning is left open. And a correct reasoning path is not guaranteed.

Where the technique helps and where it doesn’t.

Chain of thought helps when an answer is built from several partial results: arithmetic, tiered pricing, rule checks with several conditions, logic questions and multi-step decisions. Wei and colleagues saw the largest gains on the hardest tasks in their test sets.

For tasks with a direct answer, the chain adds little. Looking up a term, translating a sentence or sorting an email into a category mostly gets you longer, pricier output. A short, clear prompt is the better choice there.

Model size matters too. In the original 2022 paper, the technique only paid off with models of around 100 billion parameters. Smaller models wrote fluent but illogical chains and did worse than without them. So only a test on your own task shows whether a chain helps.

A common extension is self-consistency, introduced by Xuezhi Wang and colleagues in 2022. The model generates several chains for the same task, and the most frequent answer wins. That raises accuracy on arithmetic tasks, and it also multiplies the cost of each request.

Reasoning models think on their own.

OpenAI o1, released in September 2024, introduced a new kind of model: reasoning models that learn in training to form a chain of thought before answering. The chain appears unprompted, and its length is adjustable: reasoning effort at OpenAI, thinking level at Google, effort levels at Anthropic.

The major vendors show users little of it (as of November 2026). OpenAI does not return reasoning tokens through its API and offers a summary on request. Anthropic returns at most a summary, written by a different model, and never the raw chain of thought; on newer models even the summary is hidden by default. Google likewise shows only thought summaries for Gemini.

Two prompting rules follow. First, telling a reasoning model to “think step by step” is redundant. OpenAI recommends giving a clear goal, constraints and the desired output format without prescribing every intermediate step. Anthropic now describes a written-out chain in the prompt only as a fallback for when thinking is turned off. Second, the classic technique stays useful wherever a model runs without reasoning, for example at high volume to save cost.

The chain is not an explanation.

A chain of thought reads like a justification, but it does not necessarily reflect how the model actually computed its answer. Miles Turpin and colleagues tested this in 2023 with GPT-3.5 and Claude 1.0. They subtly altered tasks, for instance by placing the correct answer first in every example. This steered the models toward wrong answers, yet their chains never mentioned the influence and offered plausible reasons instead. Across 13 tasks from the BIG-Bench Hard collection, accuracy dropped by as much as 36 percent.

A 2025 study by Anthropic found something similar for reasoning models. The researchers slipped the models hidden hints and checked whether the chain mentioned the hint used. Averaged across all hint types, Claude 3.7 Sonnet did so 25 percent of the time and DeepSeek R1 39 percent of the time.

The operational takeaway: the chain is a good tool for finding errors. It is not evidence that a result came about for the right reasons. Where traceability matters, as in decisions about people, check the result against rules and sources instead of the model’s account of itself.

What chain of thought costs and what drives it.

Every intermediate step consists of output tokens. Output tokens usually cost considerably more than input tokens. A request with a chain of thought therefore costs more by the length of the chain. There is no fixed ratio; it depends on the task, the model and the prompt.

With reasoning models, this line item is easy to miss. OpenAI, Anthropic and Google bill thinking tokens as output, even when they never appear in the response or show up only as a summary. Thinking tokens also take up room in the context window and delay the first line of the answer.

Vendor parameters control the effort: lower settings for simple, urgent tasks, higher ones for hard problems. Which setting pays off shows only in a comparison on your own task: the same test cases at different settings, measured for quality, cost and response time. Logging every call with its cost shows where the chain of thought earns its keep and where it only makes the bill longer.

What a good implementation looks like.

A good implementation uses chain of thought selectively, not everywhere. It runs on multi-step tasks, not on simple lookups and classifications.

The chain and the result are kept apart. The result comes in a fixed format a program can read, and it is checked against a rule, a price list or a source before it moves on. For arithmetic, an ordinary program often does that work: the model identifies the values, and the calculation itself runs in code.

The effect is measured. A fixed set of test cases shows whether the chain raises accuracy and how much it adds to cost and response time. After every model change the test runs again, because a written-out chain can become redundant when you switch to a reasoning model.

Chains, or their summaries, are logged for debugging, not handed to customers as the reason for a decision. Anyone hiring a vendor can ask about exactly these points: Where is the chain switched on, how is the result checked, and what does it cost per call?

Where the term stops working

Chain of thought improves multi-step tasks, but it does not make any step correct by default and does not prove how the model actually reached its answer. It differs from prompt chaining, which runs several separate calls in sequence, and from self-consistency, which compares several chains. Explainable AI has a different goal: making a model’s decisions traceable with its own methods instead of the model’s account of itself.