Week 6. Prerequisite: Modules 1–3 and Part 0, §§0.4–0.6.

4.0 Module map and objectives

Modules 1–3 built the harness’s skeleton: the loop, the tools, and the context. This module treats the model’s internal work — the reasoning it does between actions — and the harness’s decision about how much of that work to buy. It is the module where “thinking” becomes an engineering variable with a cost, a latency, and a measurable effect, rather than a property of the model that you either have or do not.

The module has four parts and one lab.

By the end of the module you will be able to do seven things:

1. Specify ReAct, Tree of Thoughts, Reflexion, and Self-Refine as harness configurations, not as model capabilities, and say what each adds to the loop.
2. State the empirical condition under which self-correction helps, and derive why it fails without an external signal.
3. Distinguish process from outcome supervision and explain why the former provides denser signal and resists reward hacking.
4. Compute the test-time scaling bound: the oracle ceiling $1-(1-p)^n$ and the verifier-limited asymptote, and decide when scaling is worth its cost.
5. Choose between sequential and parallel compute using a latency-cost-quality comparison.
6. Diagnose whether reasoning helps or harms a tool loop, and measure it.
7. Critique a vendor’s use of the METR time-horizon metric, naming the three caveats and the questions a rigorous reviewer must ask.

Everything is self-contained. Where Part 0 is used, the result is restated.

PART A — REASONING AND PLANNING METHODS

4.1 ReAct: interleaving reasoning and acting

4.1.1 The pattern

ReAct (Yao et al., arXiv:2210.03629, evidence tier R: independently reproduced) interleaves reasoning traces with actions. At each step, the model produces a thought (a natural-language reasoning segment), then an action (a tool call), then receives an observation (the tool result), then produces the next thought, and so on. The final answer is produced when the model decides it has enough information.

The formal structure is a trajectory

where the thought is not an action in the POMDP sense — it does not change the environment — but it conditions the next action. The model’s policy is

where $c_t$ is the context assembled from the prior thoughts, actions, and observations.

4.1.2 What ReAct adds to the loop

Module 1’s minimal harness already has a loop: model call, action, observation, repeat. ReAct adds one thing: the model is instructed to produce a reasoning trace before each action, and the trace is retained in the context. This is a harness configuration, not a model capability — the same model, with and without the ReAct instruction, behaves differently.

Three mechanisms explain why the trace helps.

Mechanism 1 — Planning before acting. The trace forces the model to articulate what it is about to do and why. This reduces impulsive actions (calling a tool without a reason) and makes the action’s rationale inspectable in the trace.

Mechanism 2 — Working memory. The trace is in the context, so it is available to the next step. A conclusion reached in thought 3 (“the error is in the parsing function”) is retained and can inform action 5, even if the intervening observations are noisy.

Mechanism 3 — Error recovery. When an action fails, the trace records what the model expected, so the model can compare expectation to outcome and revise. Without the trace, the model sees only the failure; with it, the model sees the expectation that the failure violated.

4.1.3 The harness decisions

ReAct is not a single configuration; it is a family. The harness must decide four things.

Decision 1 — Is the trace shown to the model on subsequent steps? If yes, it is working memory (Mechanism 2). If no, it is only a planning device (Mechanism 1) and the context cost is lower. The default in the literature is to show it; the default in production is often to hide it after the action, to save tokens. The choice is measurable: compare success and token cost with and without trace retention.

Decision 2 — Is the trace shown to the user? A trace can be a transparency feature (the user sees the model’s reasoning) or an internal detail. Showing it has a cost (the user may be confused by the model’s false starts) and a benefit (auditability). Module 20’s governance requirements often mandate showing it.

Decision 3 — How long can the trace be? An unbounded trace consumes context and can crowd out observations. A bounded trace (e.g., “reason in at most 100 tokens”) forces concision but may truncate a needed argument. The bound is a harness parameter.

Decision 4 — Is the trace structured? A free-text trace is flexible but hard to parse; a structured trace (e.g., a JSON object with plan, expected_outcome, confidence) is parseable but constrains the model. The structured form enables observability (Module 11) at the cost of expressiveness.

4.1.4 The limits

ReAct’s trace is generated by the same model that acts, so it inherits the model’s errors. A model that misunderstands the task will produce a trace that rationalises the misunderstanding; the trace looks coherent and is wrong. This is the rationalisation problem: the trace is a post-hoc narrative, not a faithful record of the computation. It is useful as a planning and memory device, but it is not evidence of the model’s actual reasoning, and it should not be treated as such in observability or governance.

The empirical status: ReAct improves performance on tasks that require multi-step interaction with an environment (question answering with search, decision-making in text games). It does not improve performance on tasks that are single-step (a classification, a generation), where the trace is overhead. The harness should enable it where the task requires iteration and disable it where it does not.

4.1.5 Concept check

Why is ReAct a harness configuration rather than a model capability?

Answer. Because the same model, with and without the ReAct instruction and the trace-retention policy, produces different behaviour. The harness decides whether the trace is generated (the instruction), whether it is retained (the context policy), how long it can be (the bound), and whether it is structured (the format). None of these is a property of the model’s weights; all are properties of the runtime. The model’s capability sets the ceiling on what the trace can contain; the harness sets whether the trace is produced and used.

4.2 Tree of Thoughts: search over reasoning

4.2.1 The pattern

Tree of Thoughts (ToT; Yao et al., arXiv:2305.10601, evidence tier R) replaces the linear trace of ReAct with a search over a tree of partial solutions. At each node, the model generates several candidate next steps (a branching factor $b$); a value function scores each candidate; the search selects which to expand (depth-first, breadth-first, or best-first). The process continues until a node is a complete solution that passes a check, or the budget is exhausted.

The formal structure: a tree where each node is a partial reasoning state, each edge is a generated step, and the value function $V(\text{node})$ estimates the probability that the node leads to a correct solution. The search is a best-first or beam search over this tree.

4.2.2 The value function is the bottleneck

ToT’s performance depends entirely on the value function. If the value function is accurate, the search finds good solutions efficiently; if it is inaccurate, the search explores the wrong branches and wastes compute.

Three value functions are available, and each has a different reliability.

Model self-evaluation. The model scores its own candidates (“is this step promising?”). This is cheap (one model call per node) and unreliable for the same reason self-correction is unreliable (§4.4): the model that generated the candidate is the model judging it, so it shares the candidate’s errors.

External verifier. A test suite, a type checker, a formal checker scores the candidates. This is reliable but expensive (each candidate must be executed or checked) and only available for tasks with a verifiable intermediate state.

Learned value model. A separately trained model predicts the value of a partial solution. This is the process reward model of §4.5. It is cheaper than an external verifier and more reliable than self-evaluation, but it requires training data and inherits the training distribution’s biases.

The choice of value function is the harness’s most important ToT decision. The lab for Module 4 uses an external verifier, because it is the reliable case; the lesson is that ToT without a reliable value function is expensive search over a noisy landscape.

4.2.3 The cost

ToT’s cost is $O(b^d)$ in the worst case, where $b$ is the branching factor and $d$ is the depth. With $b=5$ and $d=4$, that is 625 nodes; each node is a model call. The cost is prohibitive for most tasks, and it is justified only when:

• The task has a verifiable intermediate state (so the value function is reliable).
• The task is hard enough that linear reasoning fails (so the search finds solutions the linear trace misses).
• The value of a success justifies the cost.

For most harness tasks, ToT is overkill. The ReAct loop with a good verifier at the end is usually a better use of compute. ToT is appropriate for tasks like puzzle-solving, formal proof search, and constrained generation, where the branching is genuine and the verifier is cheap.

4.2.4 The connection to test-time scaling

ToT is a form of sequential test-time scaling with search: it spends compute to explore alternatives and select among them. The analysis of §4.6–4.7 applies: the gain over linear reasoning is bounded by the value function’s reliability, and the cost is the number of nodes expanded. The decision to use ToT is a special case of the decision to scale test-time compute, with the search structure as an additional design variable.

4.2.5 Concept check

Why does ToT’s performance depend on the value function rather than on the model’s raw reasoning capability?

Answer. Because the search explores a tree, and the value function decides which branches to explore. A capable model with a poor value function will generate good candidates in some branches and bad candidates in others, and the search will not know which is which — it will expand the wrong branches and waste compute. A less capable model with a good value function will explore the promising branches and find the solution the capable model missed. The value function is the search’s compass; without it, the search wanders.

4.3 Reflexion: learning from failure

4.3.1 The pattern

Reflexion (Shinn et al., arXiv:2303.11366, evidence tier R) adds a reflection step after a failure. When a run fails (the verifier rejects the output, or the agent decides it is stuck), the harness prompts the model to reflect: “The run failed. What went wrong, and what would you do differently?” The reflection is stored in a memory (a text buffer), and on the next attempt, the reflection is included in the context.

The formal structure: the agent’s policy is conditioned on a memory $M$ that accumulates reflections across attempts:

and $M$ is updated after each failed attempt:

4.3.2 What it adds and what it does not

What it adds. Reflexion is a form of learning without weight updates: the model does not change, but its context changes across attempts, so its behaviour can improve. It is cheap (one model call per failure) and it does not require a training pipeline. This makes it attractive for tasks where retraining is impractical.

What it does not add. The reflection is generated by the same model that failed, so it inherits the model’s understanding of the failure. If the model failed because it misunderstood the task, the reflection will rationalise the misunderstanding rather than correct it. The reflection is useful when the failure is detectable from the trajectory (the model can see what went wrong) and fixable by a different action (the model knows what it should have done). It is not useful when the failure is due to a knowledge gap or a reasoning error the model cannot see.

4.3.3 The empirical status

Reflexion improves performance on tasks with iterative feedback: coding tasks where the test suite shows the failure, decision-making tasks where the environment shows the consequence. It does not improve performance on tasks where the failure is opaque (the model cannot tell why it failed) or where the model’s understanding is the problem.

The result is a special case of the self-correction analysis in §4.4: reflection helps when there is an external signal that makes the failure legible. The verifier is the signal; the reflection is the model’s interpretation of it. Without the signal, the reflection is the model talking to itself about a failure it does not understand.

4.3.4 The harness decisions

Reflexion’s effectiveness depends on the harness’s choices.

Decision 1 — What triggers a reflection? A verifier failure is a clear trigger; a model’s self-assessment of being stuck is a noisy one. The trigger determines whether the reflection is grounded in evidence.

Decision 2 — How many reflections are retained? A buffer of all reflections grows unboundedly and can crowd out the task context. A bounded buffer (the last $k$ reflections, or a summarised version) is cheaper but may lose the relevant lesson. This is the same curation problem as the evolving playbook (Module 3, §3.7.3).

Decision 3 — Are reflections per-task or per-task-family? A per-task reflection is specific and relevant but does not transfer; a per-family reflection is general but may not apply. The choice determines whether the memory helps on a new task or only on the same task retried.

4.3.5 Concept check

Why does Reflexion help on coding tasks but not on factual question-answering tasks?

Answer. Because a coding task has an external signal (the test suite) that makes the failure legible: the model can see which test failed and why, and the reflection can identify the code change that would fix it. A factual question-answering task has no such signal: the model does not know its answer is wrong, and if it did, it would not know why. The reflection in the factual case is the model restating its answer with more confidence or less, which does not improve it. The presence of the external signal is the condition.

4.4 Self-Refine and the limits of self-correction

4.4.1 The pattern

Self-Refine (Madaan et al., arXiv:2303.17651, evidence tier R) is the simplest self-correction method: the model generates an output, critiques it (“what is wrong with this?”), and revises it based on the critique. The loop can repeat until the critique finds nothing to fix or a budget is exhausted.

The formal structure:

4.4.2 The empirical limit

The central empirical question is: does self-correction improve performance? The answer, established across several papers and summarised in the critical literature, is: it depends on whether there is an external signal.

When there is an external signal (a test suite, a checker, a ground-truth comparison), self-correction helps. The critique is grounded in evidence: “the test fails with error X” is a legible failure, and the revision can address it. This is the Reflexion case, and it works.

When there is no external signal, self-correction is unreliable. The model critiques its own output using the same knowledge and reasoning that produced the output. If the output is wrong because the model’s knowledge is wrong, the critique will not catch it — the model does not know it is wrong. Worse, the model’s confidence is not calibrated: it may revise a correct answer to an incorrect one, because the critique finds a spurious flaw.

The formal statement:

Proposition 4.4.1 (Self-correction without external signal is not guaranteed to improve). Let $y_0$ be the initial output and $y_1$ the revised output after self-critique. Without an external signal, $\Pr(\text{correct}(y_1))\ge\Pr(\text{correct}(y_0))$ is not guaranteed; in general the revision can decrease correctness.

Argument. The critique and the revision are generated by the same model $\pi_\theta$. The critique’s judgement of correctness is a function of the model’s representation, which is the same representation that produced $y_0$. If $y_0$ is incorrect because the representation is incorrect, the critique shares the error and will not flag it. If $y_0$ is correct, the critique may still find a spurious flaw (the model is not calibrated), and the revision may introduce an error. Both directions are possible, so the revision is not a monotone improvement. $\square$

This is not a theorem about all models — a model with a genuinely separate critic (a different model, a verifier, a formal checker) does not have this problem. It is a statement about self-correction, where the critic and the generator are the same.

4.4.3 The empirical picture

The literature reports mixed results, and the pattern is consistent with the proposition:

• On arithmetic and symbolic reasoning, self-correction helps modestly, because the model can check some errors (recompute a sum) even without an external tool.
• On factual knowledge, self-correction often does not help, and can hurt, because the model cannot check its facts.
• On reasoning tasks with a verifiable answer (math word problems with a known answer, code with tests), self-correction with the verifier helps substantially; without it, it is unreliable.
• On open-ended generation (essays, summaries), self-correction improves style and format but not factual accuracy, because the critique can see style but not truth.

The lesson for the harness: do not deploy self-correction as a general improvement. Deploy it where there is an external signal, and measure the delta. Where there is no signal, the correction is as likely to hurt as to help, and the cost (extra model calls) is real.

4.4.4 The harness implication

The implication is a design rule:

Design rule (self-correction). Self-correction is a verifier-gated loop, not a free-standing refinement. Enable it only when a verifier can check the output; use the verifier’s verdict as the critique’s anchor; and measure the success delta with and without the loop, with confidence intervals.

This is why the Module 4 lab asks you to reproduce a self-correction result with and without an external verifier. The expected result is that the verifier is the difference between a helpful loop and a harmful one. If your result differs — if self-correction helps without the verifier on your task — that is interesting and worth reporting, but it should be established, not assumed.

4.4.5 Concept check

A colleague says “self-correction is free improvement — the model checks its own work.” What is wrong with this claim?

Answer. It assumes the model can check its own work, which requires that the model’s checking ability be independent of its generating ability. It is not: the same weights produce both, so an error in the generation is likely to be an error in the check. The claim is true when there is an external signal (the check is grounded in evidence the model did not generate), and false when there is not. The improvement is not free — it costs extra model calls — and it is not guaranteed — it can decrease correctness. The correct claim is: “self-correction helps when a verifier anchors it, and should be measured.”

PART B — PROCESS VERSUS OUTCOME SUPERVISION

4.5 Process reward models

4.5.1 The distinction

Outcome supervision rewards the final answer: the model is trained (or selected) based on whether its final output is correct. The signal is sparse (one bit per trajectory) and delayed (it arrives at the end).

Process supervision rewards each step: the model is trained (or selected) based on whether each intermediate step is correct or promising. The signal is dense (one bit or score per step) and immediate.

The distinction is due to Lightman et al. (arXiv:2305.20050, evidence tier R: independently reproduced), who trained process reward models (PRMs) on step-level labels and compared them to outcome reward models (ORMs) on mathematical reasoning.

4.5.2 The result

The Lightman et al. result, in brief: process supervision outperforms outcome supervision on math reasoning, and the gap is largest on the hardest problems. The interpretation:

• Outcome supervision rewards a correct final answer regardless of the reasoning path. A model can reach a correct answer by a wrong path (lucky guess, compensating error), and outcome supervision reinforces the wrong path.
• Process supervision rewards correct steps. It provides a denser signal, so the model learns the reasoning, not just the answer. It also resists the “correct answer, wrong reasoning” failure.

The result has a caveat that is often missed: process supervision requires step-level labels, which are expensive to obtain. Lightman et al. used human annotators; the cost is prohibitive for most tasks. The alternative is to generate step labels automatically, which reintroduces the verifier-reliability problem (Part 0, §0.5): an automatically generated step label is only as good as the process that generated it.

4.5.3 The connection to the harness

The PRM is a value function for search (§4.2.2) and a reward for training (Module 16). Its role in the harness is:

As a search value function. In ToT or best-of-n, the PRM scores partial solutions. A reliable PRM makes the search efficient; an unreliable one makes it wander. The PRM’s reliability is the same quantity as the verifier’s reliability (TPR, FPR), and the analysis of Part 0, §0.5 applies.

As a training reward. In agentic RL, the PRM provides dense per-step rewards, which reduce the variance of the advantage estimate (Part 0, §0.6). This is the VPR/RLVP line of work (Module 16). The benefit depends on the PRM’s reliability, as before.

4.5.4 The reward-hacking risk

A PRM is a learned model, and like any learned model it can be exploited. The policy can learn to produce steps that the PRM scores highly without producing correct reasoning — the PRM’s version of reward hacking. This is the same mechanism as Part 0, §0.6, Proposition 0.6.8: the policy sharpens toward what the reward model rewards, which may be a proxy for correctness rather than correctness itself.

The defence is the same as for any verifier: adversarial evaluation (Module 17b), a held-out task distribution, and a check that the PRM’s high-scoring steps are actually correct (not just PRM-favoured). The PRM is not a substitute for a verifier; it is a cheaper, denser, less reliable signal that must be validated against the verifier.

4.5.5 Concept check

Why does process supervision resist the “correct answer, wrong reasoning” failure that outcome supervision reinforces?

Answer. Because process supervision scores each step, so a trajectory that reaches a correct answer by a wrong path is penalised at the wrong steps even though the final answer is correct. Outcome supervision sees only the final answer, which is correct, and reinforces the whole trajectory — including the wrong reasoning. The process signal separates the answer from the reasoning, so the model learns the reasoning that generalises rather than the shortcut that happened to work.

PART C — TEST-TIME SCALING

4.6 Best-of-n with verifiers

4.6.1 The setup

The harness draws $n$ independent samples from the model, $y_1,\dots,y_n\sim\pi_\theta(\cdot\mid x)$, and uses a verifier to select one. The question is: what is the probability that the selected sample is correct, as a function of $n$, the per-sample success probability $p$, and the verifier’s reliability?

4.6.2 The oracle bound

If the verifier is perfect — it accepts exactly the correct samples — then the selected sample is correct iff at least one sample is correct:

This is the oracle bound: the best that test-time scaling can do. It is the ceiling, and it is useful because it tells you whether scaling is worth trying: if $1-(1-p)^n$ is barely above $p$, the samples are too correlated or $p$ is too high for scaling to help.

Worked example. $p=0.3$, $n=10$: $1-0.7^{10}=1-0.0282=0.972$. The oracle bound is 97.2% — a huge gain over 30%. But this requires a perfect verifier. With a realistic verifier, the gain is much smaller.

4.6.3 The verifier-limited asymptote

With an imperfect verifier, the selected sample is correct with probability bounded by the verifier’s precision — the probability that an accepted sample is correct. If the verifier accepts at least one sample with probability approaching 1 as $n\to\infty$, then

where $\pi_v$ is the verifier’s precision at the operating point (Part 0, §0.5, Proposition 0.5.5):

This is the verifier-limited asymptote: no amount of sampling can exceed the verifier’s precision, because the verifier is the bottleneck on which sample is selected.

Worked example. $p=0.3$, TPR $=0.9$, FPR $=0.1$. Precision $=0.9\times0.3/(0.9\times0.3+0.1\times0.7)=0.27/(0.27+0.07)=0.794$. So the asymptote is 79.4%. The oracle bound was 97.2%; the verifier limits it to 79.4%. The 17.8-point gap is the verifier’s fault, not the model’s.

4.6.4 The finite-n behaviour

At finite $n$, the success probability is between $p$ and the asymptote. A useful approximation: the probability that at least one correct sample is generated is $1-(1-p)^n$; the verifier selects a correct one with probability approximately $\pi_v$ given that a correct one is available. So

where the fallback is the probability that the selected sample is correct when no correct sample is available — which is zero for a sound verifier (it cannot select a correct sample that does not exist). So to first order,

This approximation is conservative (it ignores the possibility that the verifier selects a correct sample when none is “available” in the oracle sense, which is impossible), and it is accurate when the verifier’s TPR is high.

Worked example, continued. $p=0.3$, $n=10$, $\pi_v=0.794$: $\Pr\approx0.972\times0.794=0.772$. Compare to the asymptote 0.794 — at $n=10$ the sampling is nearly saturated, and the gap to the asymptote is small.

4.6.5 The decision rule

Test-time scaling is worth its cost iff

where $V$ is the value of a success and $c_{\text{sample}}$ is the cost per sample. The left side is the expected value of the gain; the right side is the cost. Rearranging: scaling is worth it iff the verifier is precise enough and the samples are cheap enough.

This is the same structure as the verification-budget analysis of Part 0, §0.9, applied to sampling rather than to verification. The two are complementary: more samples increase the chance of a correct one; a better verifier increases the chance of selecting it. The optimal investment is in whichever has the higher marginal return.

4.6.6 Concept check

Why does the verifier’s precision, not its accuracy, limit test-time scaling?

Answer. Because the verifier’s job in best-of-n is to select among samples, and the selection is correct iff the selected sample is correct. Precision is the probability that an accepted sample is correct — exactly the quantity that governs selection. Accuracy (the fraction of all verdicts that are correct) mixes accept-correct and reject-incorrect, and the reject-incorrect verdicts do not affect the selection when a correct sample is available. A verifier with high accuracy but low precision (it accepts many incorrect samples) will select wrong samples even when correct ones are present, and scaling will not help.

4.7 Sequential versus parallel compute

4.7.1 The two forms

Parallel compute. $n$ independent samples, each generated from the same context, selected by a verifier. The samples do not condition on each other. Latency is the latency of one sample (if the samples are generated concurrently) or $n$ times that (if sequentially); cost is $n$ times the per-sample cost.

Sequential compute. A single trajectory that conditions each step on the previous ones: chain-of-thought reasoning, self-correction, or a multi-step plan. Latency is the sum of the step latencies; cost is the sum of the step costs.

The distinction is not just implementation: it is a difference in what the compute buys. Parallel compute buys coverage (more chances to produce a correct sample). Sequential compute buys composition (each step can use the result of the previous one).

4.7.2 The trade-off

Parallel is better when:

• The verifier is reliable (so selection works).
• The samples are diverse (so more samples cover more of the answer space).
• The task is not decomposable into dependent steps (so composition does not help).
• Latency matters and the samples can be generated concurrently.

Sequential is better when:

• The task requires composition (step $k+1$ depends on step $k$).
• The verifier is weak (so selection cannot rescue a wrong answer, and the reasoning must be right).
• The answer space is large but the reasoning path is narrow (so coverage does not help).
• Latency is less important than quality.

4.7.3 The cost comparison

Let $c_p$ be the cost per parallel sample and $c_s$ the cost per sequential step. For a budget $B$, the number of parallel samples is $n_p=B/c_p$ and the number of sequential steps is $n_s=B/c_s$. The success probability is:

• Parallel: $1-(1-p)^{n_p}$ (oracle) or the verifier-limited version.
• Sequential: $p_{\text{step}}^{n_s}$ if each step must be correct (multiplicative), or a more complex function if the steps are partially independent.

The comparison is task-specific. The lab for Module 4 is a parallel-scaling experiment; the sequential case is developed in Module 16 (agentic RL) and Module 8 (long-horizon decomposition). The principle to record now is: the two forms of compute are not interchangeable, and the choice depends on the task’s structure.

4.7.4 Concept check

A task requires four dependent steps, each of which the model gets right with probability 0.9. Is parallel or sequential compute the better investment?

Answer. It depends on whether the steps are actually dependent in the sense that a wrong early step dooms the trajectory. If they are, sequential compute with a verifier at each step (so the model can correct an early error before proceeding) is the right investment, because parallel sampling cannot compose the steps — each sample is an independent attempt at the whole trajectory, and the probability that all four steps are right in one sample is $0.9^4=0.656$. Sequential compute with per-step correction can raise each step’s success above 0.9, and the product rises accordingly. If the steps are independent (a wrong early step does not prevent a correct later one), parallel sampling is better, because it explores more combinations.

4.8 When reasoning helps or harms tool loops

4.8.1 The question

Extended reasoning — chain of thought, “thinking” tokens, long internal deliberation — is often presented as universally beneficial. It is not. In a tool loop, reasoning can help or harm, and the harness must diagnose which.

4.8.2 When reasoning helps

Planning. Before a complex action (a multi-file edit, a database migration), reasoning through the plan reduces errors. The reasoning is a form of pre-computation that the action then executes.

Interpreting ambiguous observations. When a tool result is ambiguous (an error message that could mean two things), reasoning about the possibilities helps the model choose the right next step.

Error recovery. When an action fails, reasoning about why helps the model avoid repeating the failure.

4.8.3 When reasoning harms

Overthinking simple steps. A step that requires a single tool call (read a file, run a test) does not benefit from a page of reasoning. The reasoning adds latency and cost, and it can introduce doubt that leads to a worse action.

Interfering with structured output. The “thinking” tokens can crowd out the structured output. A model that reasons at length and then has insufficient budget for the tool call may emit a truncated or malformed call. This is a token-budget interaction: the reasoning and the output share the same budget.

Rigid plans. A model that reasons out a plan and then follows it rigidly, ignoring observations that contradict the plan, is worse than a model that reasons less and adapts more. This is the commitment problem of Module 1, §1.6.3: the plan is a commitment device, and a commitment that is not reconsidered is a liability when the world changes.

Latency in interactive settings. A user waiting for a response does not benefit from the model’s extended deliberation. The latency is a cost that must be justified by the quality gain.

4.8.4 The diagnostic

The harness should measure reasoning’s effect rather than assume it. The measurement:

1. Ablate the reasoning. Run the same task set with and without the reasoning instruction (or with a shorter reasoning budget), same model, same harness otherwise.
2. Measure success, latency, and cost. Report the delta with confidence intervals (Part 0, §0.4).
3. Stratify by step complexity. The interesting result is usually that reasoning helps on complex steps and hurts on simple ones. Report the interaction.

The lab for Module 4 includes this ablation as part of the self-correction reproduction: the “with verifier” condition is a reasoning-enabled loop, and the “without verifier” condition is the same loop without the reasoning anchor.

4.8.5 Concept check

Why can reasoning interfere with structured output?

Answer. Because the reasoning and the structured output share the same token budget and the same generation stream. A model that spends 2,000 tokens reasoning has 2,000 fewer tokens available for the tool call, and if the total budget is tight, the call may be truncated or the model may rush the structured part. The interference is a budget effect, not a capability effect: the model can produce both, but not both at full length within a fixed budget. The harness can mitigate it by budgeting separately (a reasoning budget and an output budget) or by using a model that separates the two streams.

PART D — METR TIME HORIZON: A CASE STUDY IN METRIC MISUSE

4.9 The metric

4.9.1 Definition

The METR time-horizon metric (arXiv:2503.14499, evidence tier S: single-lab, methodology inspectable, with caveats) estimates the length of task — measured in human-expert completion time — at which a model succeeds with probability 0.5.

The construction:

1. Assemble a set of tasks, each labelled with the time a human expert takes to complete it.
2. Evaluate the model on each task, recording success or failure.
3. Fit a logistic model: $\Pr(\text{success}\mid t)=\sigma(\alpha+\beta\log t)$, where $t$ is the human time.
4. The 50% time horizon is the $t$ at which $\Pr(\text{success})=0.5$, i.e. $t_{50}=\exp(-\alpha/\beta)$.

4.9.2 What it measures and what it does not

It measures: the length of task at which the model is as likely to fail as to succeed, on the task distribution used.

It does not measure: the length of task the model can reliably complete, the length of task it can complete in deployment, or the length of task it can complete without a verifier. These are different quantities, and conflating them is the misuse the rest of Part D addresses.

4.10 The three caveats

4.10.1 Error bars

The 50% horizon is estimated from a finite task set, so it has a confidence interval. METR reports that the uncertainty is roughly a factor of two in each direction: a horizon estimated at 1 hour might have a confidence interval of 30 minutes to 2 hours.

The implication: a horizon of “1 hour” and a horizon of “2 hours” may not be distinguishable on the same task set. Vendor comparisons that report a horizon without its interval are reporting a point estimate as if it were precise. The rigorous reader asks for the interval, and treats overlapping intervals as no evidence of a difference.

4.10.2 Domain variation

The horizon varies by 40–100× between domains: a model’s software-engineering horizon may be 1 hour, and its computer-use horizon may be 1 minute, or vice versa. The variation is not noise; it reflects genuine differences in how the model’s capabilities map to different task structures.

The implication: a horizon is domain-specific. A vendor that reports “a 1-hour horizon” without naming the domain is reporting a number that may not apply to your domain. The rigorous reader asks: which task distribution, and what is the horizon on my distribution?

4.10.3 The 50% ≠ delegable distinction

This is the most important caveat, and the one most often missed.

A 50% success rate is the rate at which the model is as likely to fail as to succeed. For a task to be safely delegated — run without human review — the success rate must be much higher, because the cost of failure is real. The analysis of Part 0, §0.5 gives the threshold: autonomous release is rational iff $(1-\mathrm{TPR})\pi C_f<C_h$. For a verifier-free delegation, the “verifier” is the model itself, and the success rate must be high enough that the expected failure cost is acceptable.

In practice, reliability-critical tasks require 98%+ success (or a verifier that catches the failures). A model with a 50% horizon at 1 hour does not mean it can be delegated 1-hour tasks; it means that at 1 hour, half of its attempts fail. The delegable horizon — the length at which the model succeeds 98% of the time — is much shorter than the 50% horizon.

Definition 4.10.1 (Delegable horizon). The delegable horizon at reliability $r$ is the task length $t_r$ at which $\Pr(\text{success}\mid t_r)=r$. For $r=0.98$, this is the length at which the model succeeds 98% of the time. It is shorter than $t_{50}$, and the gap depends on the slope $\beta$ of the logistic fit: a steep curve has a small gap, a shallow curve a large one.

The implication: reporting the 50% horizon as “the length of task the model can do” is a category error. The 50% horizon is a capability metric; the delegable horizon is a deployment metric. They answer different questions.

4.11 The vendor misuse critique

4.11.1 The four misuses

A vendor (or a breathless press release) can misuse the metric in four ways, each corresponding to a caveat.

Misuse 1 — Reporting the point estimate as if it were precise. “Our model has a 4-hour horizon” without the interval. The interval may be 2–8 hours, and the 4-hour claim is not distinguishable from a competitor’s 3-hour claim.

Misuse 2 — Reporting a horizon from a favourable domain. “Our model has a 10-hour horizon” measured on a domain where the model happens to be strong, presented as if it applied to all domains. The domain variation is 40–100×, so the claim may be off by an order of magnitude for the reader’s domain.

Misuse 3 — Conflating the 50% horizon with the delegable horizon. “Our model can handle 4-hour tasks” when the 50% horizon is 4 hours. At 4 hours, the model fails half the time; the delegable horizon at 98% is much shorter.

Misuse 4 — Cherry-picking the task distribution. The horizon depends on the tasks used to measure it. A distribution skewed toward tasks the model is good at produces a longer horizon. Without the distribution disclosed, the number is not comparable across vendors.

4.11.2 The reviewer’s questions

A rigorous reviewer asks four questions, one per misuse:

1. What is the confidence interval on the horizon? (If not reported, the point estimate is not usable for comparison.)
2. What task distribution was used, and what is the horizon on my distribution? (If the distribution is not disclosed, the number may not apply.)
3. Is this the 50% horizon or the delegable horizon at my reliability requirement? (If 50%, the delegable horizon is shorter, and the gap must be estimated.)
4. Has the result been independently reproduced? (If not, it is a single-lab claim, and the reproduction status is S or P, not R.)

4.11.3 The constructive use

The metric is not useless; it is useful when used correctly. Its correct uses:

• As a capability trend: across model generations, the 50% horizon on a fixed task distribution is a useful measure of progress, provided the interval is reported and the distribution is held fixed.
• As a domain-specific capability measure: for a specific domain, the horizon tells you the length at which the model is at chance, which is a lower bound on its capability and an upper bound on its useful range.
• As a comparison, with intervals: two models’ horizons can be compared if the intervals are non-overlapping, on the same distribution, with the same evaluation protocol.

The metric is misused when it is treated as a deployment claim. The fix is the distinction between capability and deployment, which is the distinction between the 50% horizon and the delegable horizon.

4.11.4 Concept check

A vendor reports “a 4-hour time horizon.” What are the four questions you must ask before using this number in a deployment decision?

Answer. (1) What is the confidence interval? (2) What task distribution was used, and what is the horizon on my distribution? (3) Is this the 50% horizon or the delegable horizon at my reliability requirement? (4) Has it been independently reproduced? Without the answers, the number is a capability claim of unknown precision, unknown applicability, and unknown deployment relevance.

LAB 4 — SELF-CORRECTION WITH AND WITHOUT A VERIFIER; THE METR CRITIQUE

4.12.1 Task specification

Choose a task from your practice with two properties:

1. It has a deterministic verifier (a test suite, a checker, a golden output).
2. It has enough instances for a statistical comparison: at least 30, ideally 50.

The canonical choice is a code-editing task with a test suite, or a math word-problem task with a known answer. If you choose a different domain, the verifier must be deterministic and the instances must be numerous.

4.12.2 The conditions

Condition A — No correction (baseline). The model produces one output; the verifier scores it. No self-correction.

Condition B — Self-correction without verifier. The model produces an output, critiques it (“what is wrong with this?”), and revises it, without access to the verifier. The verifier scores the final output. The critique is generated by the same model, with no external signal.

Condition C — Self-correction with verifier. The model produces an output, the verifier runs and returns its verdict (pass/fail, and the failure message if available), and the model revises based on the verdict. The verifier scores the final output.

The three conditions share the model, the task set, the context manager (except for the critique/verdict content), and the budget (except for the extra correction step). The only variable is the presence and grounding of the correction signal.

4.12.3 The hypotheses

State before running:

• H1. Condition C outperforms Condition A, because the verifier grounds the correction.
• H2. Condition B does not outperform Condition A, and may underperform it, because the self-critique is ungrounded (Proposition 4.4.1).
• H3. The gap between C and B is larger on tasks where the verifier’s failure message is informative than on tasks where it is not.
• H4. Condition B’s variance is higher than Condition A’s, because the ungrounded critique can help or hurt depending on the task.

4.12.4 The metrics

The last two metrics are the decomposition that makes the result interpretable: the net delta is the benefit minus the harm, and the two can be large in both directions even when the net is small.

4.12.5 The METR critique

Separately, write a one-page critique of a vendor’s use of the METR time-horizon metric. The critique should:

1. Identify a specific vendor claim (real or constructed) that uses the metric.
2. Apply the four reviewer’s questions of §4.11.2.
3. State what the claim would need to include to be usable in a deployment decision.
4. Explain the difference between the 50% horizon and the delegable horizon, with the formula from Definition 4.10.1.

The critique is graded on the rigour of the questions, not on the choice of vendor.

4.12.6 Acceptance criteria

4.12.7 What good looks like

A good lab report has four properties.

It reports the decomposition, not just the net. The interesting result is usually that Condition B has both higher benefit and higher harm than Condition A, and the net is near zero. The decomposition is the finding; the net is the summary.

It interprets against the proposition. Proposition 4.4.1 predicts that Condition B does not reliably improve on A. The report should say whether the data support this, contradict it, or are ambiguous, and what the ambiguity is.

It measures the cost. Self-correction is not free. The report should state the latency and token cost of the correction step, and whether the success gain (if any) justifies it.

The METR critique is constructive. It should not just criticise; it should state what a usable claim would look like, and why the distinction between capability and deployment matters.

MODULE 4 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E4.1. A task has per-sample success $p=0.2$. The verifier has TPR $=0.95$, FPR $=0.05$. Compute the oracle bound and the verifier-limited asymptote for $n=5$ and $n=20$.

S4.1. Oracle: $n=5$: $1-0.8^5=1-0.32768=0.672$. $n=20$: $1-0.8^{20}=1-0.0115=0.988$. Precision: $0.95\times0.2/(0.95\times0.2+0.05\times0.8)=0.19/(0.19+0.04)=0.826$. Finite-n approximation: $n=5$: $0.672\times0.826=0.555$. $n=20$: $0.988\times0.826=0.816$. The asymptote is 0.826; at $n=20$ the approximation is 0.816, close to the asymptote.

E4.2. A colleague proposes scaling to $n=100$ on a task with $p=0.4$ and a verifier with precision 0.6. What is the maximum achievable success, and is the scaling worth it if a sample costs 0.01 and a success is worth 1.0?

S4.2. Asymptote $=0.6$. Oracle at $n=100$: $1-0.6^{100}\approx1$. Approximation: $1\times0.6=0.6$. The maximum achievable is 0.6 — the verifier’s precision is the bottleneck. At $n=1$, success $=0.4$ (no selection benefit). The gain is 0.2. Cost: $100\times0.01=1.0$. Value of gain: $0.2\times1.0=0.2$. Cost exceeds value: not worth it. The verifier’s precision is too low; the investment should be in the verifier, not in more samples.

E4.3. Explain why self-correction without an external signal is not guaranteed to improve, using the argument of Proposition 4.4.1.

S4.3. The critique and the revision are generated by the same model that produced the output. If the output is wrong because the model’s representation is wrong, the critique shares the error and will not flag it — the model does not know it is wrong. If the output is correct, the critique may find a spurious flaw (the model is uncalibrated) and the revision may introduce an error. Both directions are possible, so the revision is not a monotone improvement. The proposition is about self-correction specifically; a separate critic (a verifier, a different model) does not have this problem.

E4.4. A process reward model scores steps on a 0–1 scale. It is used as the value function in a ToT search with branching factor 4 and depth 3. The PRM’s precision is 0.7. What is the search’s expected success, and how does it compare to linear reasoning at $p=0.3$?

S4.4. ToT with a perfect PRM explores $4^3=64$ nodes; the oracle success is $1-(1-p)^{64}$ if the nodes are independent, which is $\approx1$. With a PRM of precision 0.7, the selected node is correct with probability at most 0.7 (the PRM’s precision bounds the selection). So the expected success is $\approx0.7$, compared to linear reasoning at 0.3. The gain is 0.4, at the cost of 64 model calls plus 64 PRM calls. Whether it is worth it depends on the value and the cost; the point is that the PRM’s precision, not the search’s breadth, is the binding constraint.

E4.5. A vendor reports a 4-hour METR horizon. The confidence interval is 2–8 hours. A competitor reports a 3-hour horizon with an interval of 1.5–6 hours. Is there evidence that the first model is better?

S4.5. No. The intervals overlap substantially. The point estimates differ by 1 hour, but the uncertainty is a factor of two in each direction, so the difference is not distinguishable. The correct conclusion is that the two models’ horizons are not measurably different on this evidence. To distinguish them, you would need a larger task set (narrower intervals) or a paired comparison on the same tasks (which would control for task difficulty).

E4.6. A vendor claims “our model can handle 8-hour tasks” based on a METR 50% horizon of 8 hours. What is the error, and what is the delegable horizon if the logistic slope is such that success drops from 0.5 to 0.98 over a factor of 4 in task length?

S4.6. The error is conflating the 50% horizon with the delegable horizon. At 8 hours, the model fails half the time. The delegable horizon at 98% is shorter. If success goes from 0.5 at $t=8$ to 0.98 at $t=8/4=2$ (a factor of 4 in length for the 0.5→0.98 change), the delegable horizon is 2 hours. The vendor’s claim overstates the delegable length by a factor of 4.

E4.7. A harness enables extended reasoning on every step. The task is a sequence of 20 simple tool calls (read a file, check a value). What is the likely effect, and how would you measure it?

S4.7. The likely effect is harm: the reasoning adds latency and cost without improving the simple actions, and it may interfere with the structured output (the reasoning tokens crowd out the tool call). The measurement: ablate the reasoning (same task set, same harness otherwise, reasoning on vs off), measure success, latency, and cost, and stratify by step complexity. The expected result is that reasoning is neutral on success, worse on latency, and worse on cost. The harness should disable reasoning for simple steps and enable it for complex ones, which requires a step-complexity classifier or a per-step reasoning budget.

E4.8. Apply the four reviewer’s questions to this claim: “Our agent has a 12-hour horizon on software engineering tasks, measured on our internal benchmark of 200 tasks.” What is missing?

S4.8. (1) Interval: not reported; the horizon’s confidence interval is missing. (2) Distribution: “internal benchmark of 200 tasks” — the distribution is not disclosed, so the horizon may not apply to the reader’s tasks; 200 tasks is also a small set, so the interval is likely wide. (3) 50% vs delegable: not stated; if it is the 50% horizon, the delegable horizon at 98% is shorter. (4) Reproduction: not stated; it is a vendor claim, so it is tier C until independently reproduced. The claim is a capability claim of unknown precision, unknown applicability, and unknown deployment relevance.

EXIT TEST FOR MODULE 4

You are ready for Module 5 when you can, without notes:

1. Specify ReAct as a harness configuration, name the four harness decisions, and explain the rationalisation problem.
2. Explain why ToT’s performance depends on the value function, and compare the three value functions.
3. Explain Reflexion’s mechanism and its condition (external signal).
4. State and argue Proposition 4.4.1, and state the design rule for self-correction.
5. Distinguish process from outcome supervision and explain why process supervision resists the “correct answer, wrong reasoning” failure.
6. Compute the oracle bound and the verifier-limited asymptote, and apply the scaling decision rule.
7. Compare parallel and sequential compute, and choose between them for a given task structure.
8. Diagnose when reasoning helps or harms a tool loop, and describe the ablation that measures it.
9. State the METR time-horizon metric, its three caveats, and the difference between the 50% horizon and the delegable horizon.
10. Apply the four reviewer’s questions to a vendor claim, and state what a usable claim would include.

End of Module 4. Say “next” to receive Module 5 in full.