Chain-of-Thought Deep Dive

CoT Is More Than Just "Think Step by Step"

The 2022 paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" from Google Research ignited a wave of research on CoT, but most people's understanding remains at the level of adding "Let's think step by step" to the prompt. In fact, CoT embodies profound cognitive science principles—it simulates the working of human System 2 thinking (slow thinking): decomposing complex problems into a series of intermediate reasoning steps, each with clear logical support, ultimately converging to a reliable conclusion.

Why is CoT effective? From an information theory perspective, CoT increases the computational effort of the model's "thinking"—at each reasoning step, when generating tokens, the model re-attends to the input and previously generated reasoning steps, effectively performing multiple rounds of internal attention computation. This recursive attention mechanism allows the model to capture long-range dependencies that are easily overlooked in a single forward pass. In other words, CoT essentially trades more computation for stronger reasoning ability.

Four Variants of CoT

Zero-shot CoT: The simplest form, just append "Let's think step by step" to the prompt. The advantage is zero cost (no need to prepare examples), but the effect is unstable—it helps significantly on simple reasoning tasks but offers limited help on complex tasks.

Few-shot CoT: Provide 2-3 examples with complete reasoning processes. This is currently the most effective and widely used form of CoT. Key design elements: the reasoning in examples must be correct (a wrong example is worse than no example), examples should cover different reasoning modes (arithmetic, logical, common sense—one each), and the complexity of examples should match the target task.

Auto-CoT: Automatically generate reasoning chains. Select diverse example questions via clustering, and let the model automatically generate reasoning chains for each example, without manual annotation. Suitable for scenarios requiring a large number of examples, but the quality of reasoning chains needs manual review.

Tree-of-Thought (ToT): An advanced version of CoT—not just linear reasoning, but exploring multiple possible reasoning paths at each step, evaluating the quality of each path, and choosing the most promising one to continue. ToT simulates the human decision process of "brainstorm → evaluate → select" and outperforms CoT on tasks requiring search and planning.

Fine Design of Few-shot CoT

Many developers mistakenly think "just find a few examples with reasoning processes." In reality, designing Few-shot CoT is a delicate craft. Example formats must be consistent—all examples use the same structure: question → reasoning → answer. Reasoning steps should be clear and verifiable—each step should be an independent, verifiable sub-conclusion, avoiding leaps. Cover edge cases—at least one example should demonstrate a situation where halfway through reasoning, more information is needed or uncertainty arises, teaching the model to express uncertainty rather than fabricate answers.

from openai import OpenAI

client = OpenAI(api_key="your-deepseek-api-key", base_url="https://api.deepseek.com")

FEWSHOT_COT = """Please answer the question following the reasoning style of the examples below.

[Example 1 - Mathematical Reasoning]
Question: A store has a promotion: buy 3 get 1 free. Xiaoming needs to buy 20 books. What is the minimum number of books he has to pay for?
Reasoning:
1. Understand the rule: for every 3 bought, 1 is free, i.e., for every 4 books, only 3 need to be paid for.
2. Calculate how many groups of "4 books": 20 ÷ 4 = 5 groups.
3. Pay for 3 books per group: 5 × 3 = 15 books.
4. Verify: 15 paid + 5 free = 20 books ✓
Answer: He has to pay for at least 15 books.

[Example 2 - Logical Reasoning]
Question: If all A are B, and all B are C, then are all A C?
Reasoning:
1. Premise 1: All A are B — A ⊆ B
2. Premise 2: All B are C — B ⊆ C
3. By transitivity: if A ⊆ B and B ⊆ C, then A ⊆ C
4. Verification: This is a valid syllogism (Barbara form).
Answer: Yes, all A are C.

[Your Question]
Question: {question}
Reasoning:"""

question = "A pool has an inlet pipe that fills it in 3 hours and an outlet pipe that empties it in 5 hours. If both pipes are open, how many hours to fill the pool?"
response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[{"role":"user","content":FEWSHOT_COT.format(question=question)}],
    temperature=0.1
)
print(response.choices[0].message.content)

Tree-of-Thought: Beyond Linear Reasoning

Tree-of-Thought (ToT) models the reasoning process as a tree search problem. At each reasoning step, the model generates multiple possible directions, evaluates the prospects of each, and chooses the most promising to continue. If a path turns out to be a dead end, it backtracks to the previous branch point and tries other directions. This explore-evaluate-backtrack mechanism gives AI problem-solving abilities similar to human experts.

The core components of ToT include: a thought generator (generates multiple candidate thoughts), a state evaluator (evaluates the quality and prospects of current thoughts), and a search strategy (BFS/DFS/Beam Search). The choice of search strategy depends on the specific task: for solving tasks (e.g., math problems, programming problems), DFS is usually more efficient; for creative tasks (e.g., writing, planning), BFS can explore more possibilities.

class TreeOfThought:
    def __init__(self, max_depth=5, beam_width=3):
        self.max_depth = max_depth
        self.beam_width = beam_width

    def generate_thoughts(self, problem, context):
        prompt = f"Problem: {problem}\nCurrent reasoning state: {context}\n\nPlease propose {self.beam_width} different next reasoning directions. Each direction should be independent, logical, and verifiable."
        response = client.chat.completions.create(
            model="deepseek-chat",
            messages=[{"role":"user","content":prompt}], temperature=0.7)
        return response.choices[0].message.content

    def solve(self, problem):
        beams = [{"context":"","score":10}]
        for depth in range(self.max_depth):
            candidates = []
            for beam in beams:
                thoughts = self.generate_thoughts(problem, beam["context"])
                for thought in thoughts.split("\n"):
 

                   if thought.strip():
                        candidates.append({"context":beam["context"]+"\n"+thought,"score":beam["score"]+7})
            candidates.sort(key=lambda x:x["score"], reverse=True)
            beams = candidates[:self.beam_width]
        return beams[0]["context"] if beams else "No solution"

tot = TreeOfThought(max_depth=4, beam_width=3)
solution = tot.solve("Design the architecture of a distributed task scheduling system")

Self-Consistency: Let AI "Think Multiple Times"
Self-Consistency is a simple yet extremely effective CoT enhancement strategy. The core idea is to let the model perform multiple independent reasonings for the same problem (by setting temperature>0 to introduce randomness), and then vote on the multiple reasoning results—selecting the answer that appears most frequently or is most consistent. This is similar to humans' "think twice before acting"—the first thought may be biased, but after thinking multiple times and reaching a consensus, accuracy improves significantly.
Research has found that on mathematical reasoning tasks, sampling 5 times and taking the majority vote can improve accuracy from 75% (single reasoning) to over 90%. More samples yield better results, but with diminishing returns—typically 5-10 samples capture most of the benefit, and beyond 20 samples, improvement is negligible. Key parameters: temperature (recommended 0.5-0.8; too low reduces diversity, too high degrades reasoning quality), and number of samples (5-10 offers the best cost-effectiveness).
CoT Failure Modes and Countermeasures
Reasoning hallucination: The model may fabricate non-existent facts or logic during reasoning. Countermeasure: Introduce an external verification step to fact-check key assertions in the reasoning.Reasoning loops: The model keeps going in circles during reasoning and cannot progress. Countermeasure: Set a maximum number of reasoning steps (e.g., 10), and force termination with a summary when timeout occurs.Over-reasoning: Overly complex reasoning for simple problems, wasting tokens. Countermeasure: First assess problem complexity; use Zero-shot CoT for simple problems, and only use Few-shot/ToT for complex ones.Format collapse: The reasoning output does not conform to the expected format, causing downstream parsing failures. Countermeasure: Emphasize format requirements in the prompt and include few-shot negative examples of format errors.
Engineering Practice: Cost-Benefit Analysis of CoT
CoT is not a free lunch. A typical Few-shot CoT prompt may consume 3-5 times more tokens than a direct prompt (including the reasoning chains in examples and those generated by the model). In real projects, decide whether to use CoT based on the value of the task: high-value tasks (e.g., medical diagnosis assistance, legal document analysis) are worth investing more tokens for deep reasoning; low-value tasks (e.g., simple information queries) are more economical with direct prompts.
A practical strategy: classify user requests by complexity. Simple queries: answer directly; medium complexity: use Zero-shot CoT; high complexity: use Few-shot CoT + Self-Consistency. Through tiered routing, control costs while ensuring quality.
Frontier Outlook: From CoT to System 2 Reasoning
The ultimate goal of CoT is not to make the model mimic the reasoning process, but to enable the model to truly possess deep reasoning capabilities similar to human System 2. Current research frontiers include: multimodal CoT (combining images, code, and other multimodal information for reasoning), recursive CoT (the model can call itself during reasoning to solve subproblems), and self-improving CoT (the model automatically optimizes its reasoning strategy through reinforcement learning). DeepSeek-R1 is an outstanding representative of this direction—through large-scale reinforcement learning training, it enables the model to intrinsically learn chain-of-thought reasoning without manually designed CoT prompts. This marks a qualitative leap in AI reasoning ability from "prompt-induced" to "model-intrinsic." Understanding the principles of CoT is essential to truly comprehend where the reasoning capabilities of this generation of AI models come from.
Want to orchestrate this skill chain yourself?
Open in Skill Chain →