Chain of Thought Prompting: Teaching LLMs to Think Step by Step

Scenario context

Chain of Thought (CoT) prompting tells a language model to reason out loud before answering. By generating intermediate steps, the model steers itself toward correct conclusions — turning guesswork into structured reasoning. It's the difference between asking a student to "just write the answer" and asking them to "show their work."

Here is a real type of failure that large language models produce every day.

**Pro…

This walkthrough focuses on The Answer Was "11" — And It Was Wrong.

Before making a design choice, clarify:

• Here is a real type of failure that large language models produce every day.
• Prompt (no reasoning guidance): Model output: 11 That answer is wrong.
• wait — actually that is right.

Concepts in play: llm-prompt-engineering, consistency-replication, tree-data-structures, core-algorithm-patterns

Stage 1 — What Chain of Thought Prompting Actually Does Under the Hood

• Chain of Thought prompting is a technique where you instruct a language model to generate intermediate reasoning steps before producing a final answer.
• It was formally introduced by Google researchers Wei et al.
• in their 2022 paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." But CoT is not magic, and it's important to understand why it works.
• Language models are next-token predictors: given a sequence of tokens, they predict the most likely next token according to patterns learned during training.

Observable outcome: Language models are next-token predictors: given a sequence of tokens, they predict the most likely next token according to patterns learned during training.

Stage 2 — The CoT Family: Zero-Shot, Few-Shot, Auto-CoT, and Beyond

• Chain of Thought is not a single technique — it's a family of related methods.
• Understanding the landscape helps you pick the right variant for each task.
• Zero-Shot CoT: "Let's think step by step" Introduced by Kojima et al.
• (2022), Zero-Shot CoT adds no examples — just an instruction.

Q: If a train travels 60 miles per hour and needs to cover 210 miles,
   how long will the trip take?

Let's think step by step.

Q: [Example 1 question]
A: [Step 1...] [Step 2...] ... The answer is X.

Q: [Example 2 question]
A: [Step 1...] [Step 2...] ... The answer is Y.

Q: [New question]
A:

Observable outcome: (2022), Zero-Shot CoT adds no examples — just an instruction.

Stage 3 — Visualizing How Reasoning Flows Differently Across CoT Variants

• The three main CoT approaches — standard prompting, linear chain-of-thought, and Tree of Thought — differ dramatically in the path they take from question to answer.
• The diagram below makes that structural difference explicit at a glance.
• Standard prompting takes a single direct hop, which fails when multiple reasoning steps are needed.
• Linear CoT breaks the problem into a verified sequence of steps, each anchored by the previous one.

flowchart TD
    Q[Question]

    Q --> S[Standard Prompt]
    S --> A1[Final Answer - direct]

    Q --> C[Zero-Shot or Few-Shot CoT]
    C --> S1[Step 1]
    S1 --> S2[Step 2]
    S2 --> S3[Step 3]
    S3 --> A2[Final Answer]

    Q --> T[Tree of Thought]
    T --> B1[Branch A: reasoning path 1]
    T --> B2[Branch B: reasoning path 2]
    T --> B3[Branch C: reasoning path 3]
    B1 --> E1[Evaluate branch A]
    B2 --> E2[Evaluate branch B - best]
    B3 --> E3[Prune branch C]
    E2 --> A3[Final Answer]

Observable outcome: Linear CoT breaks the problem into a verified sequence of steps, each anchored by the previous one.

Stage 4 — Implementing CoT in Python: Zero-Shot, Few-Shot, and Self-Consistency

• This section shows three runnable Python patterns you can drop into any project that uses the OpenAI API.
• Each pattern builds on the previous one.
• Zero-Shot CoT Template The simplest possible implementation — append the magic phrase and let the model do the rest.
• Expected output (abbreviated): Few-Shot CoT with Three Worked Examples When the task has a consistent structure (e.g., always arithmetic word problems), few-shot examples teach the model the exact reasoning format you w…

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from environment

def zero_shot_cot(question: str) -> str:
    """Appends 'Let's think step by step' and returns the full reasoning + answer."""
    prompt = f"{question}\n\nLet's think step by step."
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a careful reasoning assistant."},
            {"role": "user", "content": prompt},
        ],
        temperature=0,  # deterministic for reproducibility
    )
    return response.choices[0].message.content

# Example usage
question = (
    "A factory produces 240 widgets per day. "
    "It operates 6 days a week. "
    "How many widgets does it produce in 4 weeks?"
)
print(zero_shot_cot(question))

Step 1: Widgets per day = 240.
Step 2: Days per week = 6, so widgets per week = 240 × 6 = 1440.
Step 3: Weeks = 4, so total = 1440 × 4 = 5760.
Answer: 5760 widgets.

Observable outcome: Expected output (abbreviated): Few-Shot CoT with Three Worked Examples When the task has a consistent structure (e.g., always arithmetic word problems), few-shot examples teach the model the exact reasoning format you w…

Stage 5 — Step-by-Step Walkthroughs: Math and Logic in Action

• To make the mechanics concrete, let's trace through two worked examples — one arithmetic and one logical deduction — showing the exact prompt and the reasoning chain the model produces.
• Worked Example 1 — Multi-step arithmetic: This example chains four sequential calculations.
• Without CoT, models frequently drop intermediate results.
• With CoT, each calculated value is pinned in the output, available for the next step.

A train departs at 08:00 and travels 3 legs:
- Leg 1: 120 km at 60 km/h
- Leg 2: 90 km at 45 km/h
- Leg 3: 150 km at 100 km/h
What time does it arrive? Let's think step by step.

Step 1: Leg 1 time = 120 ÷ 60 = 2 hours.
Step 2: Leg 2 time = 90 ÷ 45 = 2 hours.
Step 3: Leg 3 time = 150 ÷ 100 = 1.5 hours.
Step 4: Total travel = 2 + 2 + 1.5 = 5.5 hours.
Step 5: Departure 08:00 + 5.5 hours = 13:30.
Answer: 13:30

Observable outcome: With CoT, each calculated value is pinned in the output, available for the next step.

Stage 6 — When Chain of Thought Helps — and When It's a Waste of Tokens

• CoT is powerful but not universally beneficial.
• Using it blindly adds cost and latency with no accuracy gain — or sometimes actively hurts.
• CoT significantly helps when: - Multi-step arithmetic — more than two calculation steps.
• The model uses the intermediate outputs as working memory.

Observable outcome: The model uses the intermediate outputs as working memory.

End-to-end outcome

After working through the stages, you should be able to explain:

• These are the mistakes that show up most often when developers first deploy Chain of Thought prompting — and how to avoid them.
• Using CoT for simple tasks wastes tokens and confuses the model.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.