LLM Reasoning

A language model generates text based on patterns learned during training. For simple questions, it may produce an answer directly. More complex problems can require several intermediate steps.

Question: A notebook costs $40. A pen costs $15 more than the notebook. What is the total cost of two notebooks and one pen?

A direct answer requires several operations.

Step 1: Find the cost of the pen. [ 40+15=55 ]

Step 2: Find the cost of two notebooks. [ 2 x 40=80 ]

Step 3: Add the costs. [ 80+55=135 ]

The final answer is $135.

The important distinction is:

• Answer generation: Produce the final answer.
• Reasoning: Work through the relationships and intermediate steps needed to obtain the answer. Reasoning is especially useful in mathematics, programming, logical puzzles, planning, and other multistep tasks.

Why can an ordinary LLM fail at reasoning?

• Limited reasoning - An LLM might know arithmetic rules but still make mistakes when several operations must be combined.

  For example, it might correctly calculate (40+15=55) but then forget to count two notebooks.

  Knowing the individual facts doesn't guarantee that the model will combine them correctly.

• Static knowledge - A model's learned knowledge comes from its training and subsequent updates. It does not automatically know every new event or change that happens after training.

  For example, a model might know how a software library worked in an earlier version but not know about a newly released API.

  This is why external information retrieval can be useful.

• Inability to perform actions - A language model that only generates text cannot automatically execute a program, access a database, or interact with a website.

  Those capabilities require tools or an agent framework.

• Difficulty of evaluation - A final answer can be correct by chance, or it can be wrong despite containing convincing explanations.

  Therefore, we need appropriate tests to measure whether the model genuinely solves the task reliably.

Chain-of-Thought (CoT)

Chain-of-Thought (CoT) means generating intermediate reasoning steps before producing a final answer.

Instead of asking the model only for the answer, we encourage it to break a complex problem into smaller steps.

Compare the two approaches.

Direct-answer approach Question: What is the total cost?

Model: ₹135.

The answer is available, but we cannot see the intermediate calculations.

Step-by-step approach

1. The notebook costs ₹40.
2. The pen costs ₹40 + ₹15 = ₹55.
3. Two notebooks cost ₹80.
4. Total = ₹80 + ₹55 = ₹135.

The intermediate steps make the solution easier to inspect and debug.

The second approach is useful because intermediate steps can help a model organize a solution to a multistep problem.

However, producing a long explanation does not automatically mean the reasoning is correct. Each important step still needs to be checked.

Why can CoT improve performance?

Imagine a problem requires five operations.

Without intermediate steps, the model must arrive at the answer directly.

With CoT, the problem can be approached as:

$$\text{Problem} \rightarrow s_1 \rightarrow s_2 \rightarrow s_3 \rightarrow \cdots \rightarrow \text{Answer}$$

Here, \(s_i\) represents an intermediate step.

Breaking the problem into smaller parts can help the model keep track of quantities, constraints, and dependencies.

For example, in the notebook problem, explicitly calculating the pen's price first reduces the chance of confusing the pen's price with the price difference.

How is CoT prompting different from ordinary prompting?

Ordinary prompting might say:

Solve the problem.

CoT-style prompting might say:

Break the problem into smaller steps, work through the calculations, and provide the final answer.

The important idea is not a particular magic phrase. It is encouraging a multistep solution instead of an immediate answer.

What is a reasoning model?

A reasoning model is designed or trained to handle complex tasks by performing more deliberate, multistep problem-solving before answering.

A useful conceptual comparison is:

Ordinary response style	Reasoning-oriented response style
May answer immediately	May spend more computation before answering
Often suited to straightforward questions	Designed to improve performance on challenging multistep tasks
Can make mistakes on complex problems	Attempts to solve problems through intermediate steps
May have limited test-time computation	Can allocate additional computation to difficult problems

These are broad tendencies, not guarantees. An ordinary model can reason through some problems, and a reasoning model can still make mistakes.

What should we look for?

A model's writing style alone is not enough to establish its reasoning capability.

For example, compare:

Response A

The answer is 24.

Response B

First, I calculate the number of groups. Then I multiply by the number in each group. The result is 24.

Response B gives more explanation, but it isn't necessarily more reliable. The calculation must still be correct.

A better approach is to test the model on tasks that require actual problem-solving.

---

Reasoning-based benchmarks

What is a benchmark?

A benchmark is a standardized collection of tasks used to measure and compare model performance.

For reasoning models, a benchmark might contain:

• Programming problems
• Mathematical questions
• Logical reasoning tasks
• Real-world software engineering issues The model attempts the tasks, and an evaluation method measures its performance.

$$\text{Benchmark score} = \frac{\text{Successful tasks}}{\text{Total evaluated tasks}} \times100%$$

This formula applies when the benchmark uses a simple success/failure measure. Some benchmarks use other metrics.

HumanEval — evaluating code generation

HumanEval is a reasoning-related benchmark for programming. Its cited reference is Chen et al.'s work on evaluating language models trained on code. LLM_Reasoning.pdf

The basic task is: Given a programming problem, can the model generate a function that works correctly?

Suppose the task is to write a function that adds two numbers.

def add(a, b):
    return a + b

This example is easy, but actual benchmark tasks are more varied.

How is the answer evaluated?

A program can look correct but fail on certain inputs. Therefore, generated code is tested using predefined test cases.

For example:

assert add(2, 3) == 5
assert add(-1, 1) == 0
assert add(0, 0) == 0

If the implementation passes the relevant tests, it is considered successful under those tests.

Important concept: pass@k

Code-generation evaluations often use a metric called pass@k.

It measures the probability that at least one of $k$ generated solutions passes the tests.

For instance:

• pass@1: one attempt is considered.
• pass@5: five candidate solutions are considered; success means at least one passes. If we generate more candidate solutions, the chance of finding a correct one may increase. However, generating more solutions also uses more computation.

Remember: HumanEval evaluates functional code correctness, not merely whether the generated code looks convincing.

SWE-bench — resolving real-world GitHub issues

HumanEval mainly focuses on individual programming tasks. SWE-bench targets a broader software engineering workflow.

A task may require a model to:

1. Understand a reported bug or requested feature.
2. Examine the existing codebase.
3. Identify the relevant files and functions.
4. Propose code changes.
5. Produce a patch that resolves the issue. Example Imagine a repository contains a function that crashes when it receives an empty list.

The GitHub issue says: The function should return an empty result instead of crashing.

A model must understand the existing implementation, modify it appropriately, and avoid breaking other behavior. How is SWE-bench evaluated? The proposed changes are applied to the codebase and evaluated with tests.

The essential distinction is:

• HumanEval: Can the model write a function that passes tests?
• SWE-bench: Can the model solve a software issue within an existing project? This makes SWE-bench a more realistic test of software engineering ability.

AIME and HMMT — mathematical reasoning

• AIME: American Invitational Mathematics Examination.
• HMMT: Harvard-MIT Mathematics Tournament. These benchmarks involve challenging mathematics rather than just recalling definitions. Example Suppose a problem asks for an unknown integer that satisfies several conditions.

The model may need to:

1. Translate the conditions into equations.
2. Simplify the equations.
3. Identify possible values.
4. Eliminate invalid possibilities.
5. Produce the required answer.

The difficulty lies in combining the steps correctly. Why are these benchmarks useful? A model might perform well on ordinary arithmetic yet struggle with unfamiliar multistep problems.

Challenging mathematics provides a way to assess more advanced problem-solving.

However, a high mathematics benchmark score does not automatically imply strong coding, factuality, or safety performance. Each benchmark measures particular capabilities.

GSM8K — grade-school mathematical reasoning

It contains grade-school mathematical word problems.

Although the arithmetic may be simple, the model must interpret the language and determine which operations are required. Example A shop has 8 boxes. Each box contains 6 pencils. The shop sells 15 pencils. How many remain?

Step 1: Calculate the initial number of pencils. [ 8 x 6=48 ]

Step 2: Subtract the pencils sold. [ 48-15=33 ]

Final answer: 33 pencils.

This example illustrates the type of multistep arithmetic that GSM8K is designed to assess.

Comparing the mathematical benchmarks

Benchmark	Main emphasis
GSM8K	Grade-school mathematical word problems
AIME	Challenging competition mathematics
HMMT	Competition-style mathematics

The central lesson is that reasoning ability depends on the task and its difficulty. A model may perform differently across these benchmarks.

---

Reasoning-based benchmark metrics

Why do we need different metrics? Imagine two models solve programming problems.

• Model A generates one answer per problem.
• Model B generates several candidate answers per problem and selects a successful one. If we compare them without specifying how many attempts are allowed, the comparison may be misleading.

Similarly, mathematics benchmarks might require an exact numerical answer, whereas coding benchmarks can run the generated program against test cases.

Therefore, we need a clearly defined evaluation protocol.

Exact-match accuracy

For tasks with a clearly defined correct answer, we can use exact-match accuracy.

$$\text{Accuracy} =\frac{\text{Correct predictions}} {\text{Total predictions}}$$

Suppose a model answers 80 questions correctly out of 100.

$$\text{Accuracy} = \frac{80}{100} =0.8=80%$$

This is easy to interpret, but it only works when the answer can be judged reliably using the chosen correctness criterion.

For open-ended explanations, exact string matching is often unsuitable: two differently worded answers can both be correct. 15. Test-based evaluation For programming, correctness can be evaluated by running generated code against test cases.

Suppose a generated function passes 9 out of 10 tests.

Its test pass rate is: [ 9/10 x 100%=90% ]

But be careful: passing 90% of a test suite does not necessarily mean that the code is 90% correct in every real-world situation. The result depends on which tests were chosen.

A program might pass all provided tests and still fail on an untested edge case.

pass@k and multiple attempts

Suppose a model generates five candidate solutions to a coding problem:

Candidate	Result
1	Fails
2	Fails
3	Passes
4	Fails
5	Fails

Here, at least one of the five solutions passes.

• pass@1 depends on which candidate is evaluated as the single attempt.
• pass@5 is successful because at least one of the five candidates passes.

The distinction matters because a system allowed multiple attempts may solve tasks that it cannot solve reliably in one attempt.

Takeaway: Always identify the number of attempts and the success criterion when interpreting a benchmark result.

Benchmark scores must be interpreted carefully

A benchmark result is not a complete description of a model.

For example:

• A model can perform well on short mathematical problems but struggle with long proofs.
• A model can pass coding tests but produce inefficient or difficult-to-maintain code.
• A model can solve a benchmark question but fail on a slightly different version. A good evaluation therefore considers the task distribution, the scoring method, and the conditions under which the model was tested.

How can we train models to become better at reasoning?

---

Test-time scaling

Test-time scaling means allocating additional computation while a model is solving a problem, rather than relying only on a single immediate answer.

Here, "test time" means the time when the trained model is being used to answer a new question. It does not mean the model is necessarily being tested in an exam.

Consider two questions:

Question A: What is \(7+5\)?

Question B: Find an integer satisfying several mathematical constraints and prove that it is the only possible solution.

Question A usually requires little computation. Question B may require several attempts, calculations, or verification steps.

It would be inefficient to spend the same amount of reasoning effort on both questions. This motivates test-time scaling.

Training versus inference

First, distinguish two terms.

• Training - The model's parameters are updated using training data and an optimization procedure.

  Example: Using reinforcement learning to improve the model's problem-solving behavior.

• Inference - The trained model receives a new prompt and generates an answer.

  Example: Asking the model to solve an unfamiliar mathematical problem.

Test-time scaling concerns the computation used during inference.

A useful conceptual relationship is:

$$\text{More inference computation} \rightarrow \text{Potentially better problem solving}$$

The word potentially matters. Additional computation can help, but it does not guarantee correctness.

Why propose reinforcement learning?

1. Writing reasoning chains from scratch is difficult, making manually creating supervised fine-tuning (SFT) data impractical at scale.
2. We do not want to restrict the model to reasoning patterns written by humans.
3. Many problems have a natural, verifiable reward: did the model solve the problem correctly or not?

Let's unpack the reasoning behind this proposal.

A. The difficulty of creating reasoning data Supervised fine-tuning can train a model using examples of prompts paired with desired answers or reasoning traces.

But consider how difficult it would be to manually write high-quality solutions for millions of challenging mathematical and programming problems.

This creates a data-generation bottleneck.

B. Avoid restricting the model to human-written solutions Humans often solve the same problem in different ways.

For example, one person might solve an equation algebraically, while another uses a clever substitution.

Training exclusively on manually written reasoning can limit the diversity of the examples available to the model.

Reinforcement learning provides another approach: encourage successful behavior through rewards rather than requiring every desired reasoning path to be supplied in advance.

C. Correctness can be verified automatically For some tasks, we can objectively check the result.

Examples:

• Mathematics: Does the answer match the correct value?
• Programming: Do the tests pass?
• Structured reasoning: Does the output satisfy the specified constraints?

These are examples of verifiable rewards.

They allow a training system to evaluate a result without needing a human to write a detailed judgement for every attempt.

---

Reinforcement learning for reasoning

What is reinforcement learning? Reinforcement learning (RL) is a learning approach in which a model is encouraged to produce behavior that receives higher rewards.

For reasoning, the simplified idea is:

Problem
   ↓
Model generates a solution
   ↓
Evaluate the solution
   ↓
Assign reward
   ↓
Update model through RL
   ↓
Model learns to favor rewarded behavior

This process is repeated across many training examples.

The goal is to make solutions that receive good rewards more likely in future generations.

How is a reward different from a training example?

In supervised fine-tuning, we might provide a target solution:

For this prompt, produce this desired answer.

In reward-based training, we might instead provide a task and evaluate the generated solution:

This solution is correct, so it receives a positive reward.

The distinction is between explicitly supplying a desired output and evaluating the output that the model generated.

Real training systems can combine supervised fine-tuning, reward models, reinforcement learning, and other techniques.

Reward 1 — Verify that CoT is present

The conceptual objective is to encourage the model to follow the expected reasoning-output format.

For example, a training system might expect:

Reasoning section
      ↓
Final-answer section

A formatting check can determine whether the expected sections are present.

Conceptually:

$$R_{\text{format}}= \begin{cases} 1,&\text{expected format is present}\ 0,&\text{otherwise} \end{cases}$$

This is an illustrative binary reward. Why is this useful? Suppose the training objective encourages the model to provide both reasoning and a final answer.

A format reward can encourage the model to produce the required structure consistently.

Important limitation: A response can follow the required format and still contain incorrect reasoning.

For example: Reasoning: 2 + 2 = 5 Final answer: 5

The structure is present, but the mathematics is wrong.

Therefore, a formatting reward is not enough by itself.

Reward 2 — Verify that the solution is correct

For a mathematics problem with a known answer, an illustrative reward is:

$$R_{\text{correctness}}= \begin{cases} 1,&\text{answer is correct}\ 0,&\text{answer is incorrect} \end{cases}$$

For programming, correctness can be checked by running tests.

Example

Suppose the correct answer is 33.

Generated answer	Correctness reward
33	1
32	0
35	0

This reward measures correctness rather than merely checking whether the model followed the expected output format.

Combine the rewards

The two ideas address different properties:

• Reward 1 encourages the expected reasoning/output format.
• Reward 2 encourages a correct solution.

An illustrative combined reward could be:

$$R=R_{\text{format}}+R_{\text{correctness}}$$

This is a simplified teaching example; it should not be interpreted as the exact reward equation used in the cited DeepSeek-R1 system.

If both rewards are binary, the combined reward ranges from 0 to 2.

Format correct?	Answer correct?	Combined reward
No	No	0
Yes	No	1
No	Yes	1
Yes	Yes	2

The key idea is that a training system can reward multiple desired properties.

What are verifiable rewards?

A verifiable reward is a reward based on a result that can be checked against an objective criterion.

Examples:

Mathematics - Compare the final answer with the known correct answer or verify that it satisfies the problem's conditions.

Programming - Execute the generated code and check whether the test cases pass.

Structured constraints - Check whether the generated output satisfies explicit requirements.

Why is this approach powerful? A system can generate many candidate solutions and automatically distinguish successful solutions from unsuccessful ones.

That creates a scalable feedback signal for training. But there is an important limitation The reward is only as good as the verification procedure.

For example, code that passes incomplete tests may still contain bugs. A mathematics answer may match a target value even if the model arrived at it through an unreliable process.

Therefore, verifiable rewards work best when the task has a trustworthy correctness checker.

---

Controlling reasoning at inference time

A new problem: Even if a model can reason, how much reasoning should it perform for a particular prompt?

1. Dynamic budget
2. Context awareness
3. Budget forcing
4. Continuous thoughts

Why not use the same thinking budget for every prompt? Suppose a model has a budget of 2,000 tokens for reasoning.

For a simple arithmetic question, it may need only a small portion of that budget.

For a challenging proof, it may benefit from a much larger budget.

Using the same budget for all prompts creates two possible problems:

• Too much computation: The model spends time and tokens on a task that doesn't need them.
• Too little computation: The model stops before adequately solving a difficult task. The lecture's starting point is that not all prompts are equal.

Let's examine each proposed approach.

Dynamic budget

A dynamic budget means adapting the amount of computation allocated to a task.

Imagine the system uses the difficulty of a problem to determine its reasoning budget.

Simple task -> Small budget -> Example: Calculate 7 + 5.

Moderate task -> Medium budget -> Example: Solve a multistep word problem.

Difficult task -> Large budget -> Example: Solve a challenging proof or debugging task.

The objective is to balance solution quality against inference cost.

A dynamic budget can be based on a predicted difficulty, an allocated limit, or a system's strategy for deciding when more computation is useful.

Remember: Dynamic budget means the reasoning budget can vary instead of being fixed for every task.

Context awareness

Context awareness means considering the prompt and its requirements when deciding how to allocate reasoning.

For example:

Prompt A: What is the capital of France?

This usually requires a short response.

Prompt B: Compare two algorithms, prove their time complexities, and identify when each is preferable.

This may require more detailed analysis.

A context-aware approach takes the nature of the request into account rather than blindly applying one fixed reasoning strategy.

Context awareness is one of the approaches to controlling inference-time thinking.

Budget forcing

citing the paper s1: Simple test-time scaling by Muennighoff et al. (2025).

The core idea is to intervene in how long the model continues reasoning, using a specified budget or stopping strategy.

Consider two broad cases:

• A model reaches a stopping point too early.
• A model continues reasoning for longer than is necessary.

A budget-forcing strategy can be used to influence this behavior.

For example, a system might enforce a minimum amount of reasoning before allowing a final answer, or constrain how long reasoning can continue.

The exact mechanism depends on the method being implemented.

Dynamic budget versus budget forcing

These concepts are related but not identical.

Concept	Main idea
Dynamic budget	Allocate a budget that varies with the task or conditions
Budget forcing	Use an intervention or constraint to influence the model's reasoning duration

One focuses on adapting the allocation; the other focuses on enforcing or influencing budget behavior.

Important limitation

Forcing a model to generate more tokens does not guarantee that those tokens contain useful reasoning. A model can repeat itself or continue along an incorrect path.

The objective is to control useful computation, not merely maximize output length.

Continuous thoughts

The final idea is "continuous" thoughts, associated with the cited paper Training Large Language Models to Reason in a Continuous Latent Space by Hao et al. (2024).

To understand the motivation, compare two ways of representing intermediate reasoning.

A. Discrete, token-based reasoning: A conventional language model represents its generated reasoning as a sequence of tokens.

For example: Step 1 → Step 2 → Step 3 → Answer

Each step is expressed through generated text.

B. Continuous latent reasoning: A latent-space approach performs intermediate computation in learned numerical representations rather than requiring every intermediate state to be expressed as ordinary text.

Conceptually:

Input
  ↓
Latent representation
  ↓
Intermediate latent computation
  ↓
Output representation
  ↓
Final answer

The word continuous refers to the nature of the learned representation, not to an uninterrupted stream of ordinary words.

Why investigate this approach? The motivation is to explore whether a model can perform useful intermediate computation without having to express every step in natural-language tokens.

Potential advantages include reducing the need to generate lengthy textual intermediate sequences.

However, whether a particular method improves accuracy, efficiency, or interpretability must be established experimentally. Key distinction:

• CoT: Intermediate reasoning is represented as text.
• Continuous latent reasoning: Intermediate computation occurs in latent representations.

These are different approaches to representing intermediate computation.

---

Summary

┌─────────────────────────────────────────────────────────────┐
│                   1. Receive a problem                      │
│         Example: Solve a mathematical problem.              │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│             2. Generate intermediate reasoning              │
│            Use a multistep approach such as CoT.            │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│               3. Produce a candidate answer                 │
│              The model returns its solution.                │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                     4. Verify the result                    │
│   Check correctness using an answer key, tests, or another  │
│                   appropriate verifier.                     │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│              5. Use rewards during training                 │
│    Reinforcement learning can encourage successful behavior.│
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│            6. Control inference-time computation            │
│       Choose an appropriate reasoning budget for the task.  │
└─────────────────────────────────────────────────────────────┘

Concept	Main purpose
Chain-of-Thought	Encourage multistep problem-solving
Reasoning benchmarks	Measure performance on reasoning tasks
Test-time scaling	Allocate additional computation during inference
Reinforcement learning	Improve behavior using reward signals
Verifiable rewards	Provide feedback based on checkable results
Dynamic budget	Adapt the amount of computation
Context awareness	Consider task requirements when allocating computation
Budget forcing	Constrain or influence the reasoning duration
Continuous latent reasoning	Explore intermediate computation in latent representations

The key is to distinguish how reasoning is represented, how it is evaluated, how it is trained, and how much computation it receives.

1. How can a model reason? — Chain-of-Thought and intermediate computation.
2. How do we measure its ability? — Benchmarks such as HumanEval, SWE-bench, AIME, HMMT, and GSM8K.
3. How can we train better reasoning? — Reinforcement learning with rewards, including verifiable correctness rewards.
4. How can we control reasoning effort? — Dynamic budgets, context awareness, budget forcing, and continuous latent reasoning.

One final distinction is particularly important: reasoning length, reasoning correctness, and reasoning capability are not the same thing. A model may generate a long explanation and still be wrong. The goal is to improve reliable problem-solving, not simply to generate more text.