Week 7. Prerequisite: Modules 1–4 and Part 0, §§0.1, 0.4, 0.5.

5.0 Module map and objectives

Modules 1–4 built the harness and its components. This module answers the question that makes all of that engineering meaningful: how do you know whether any of it works? Evaluation is not a reporting step appended to the engineering; it is the discipline that makes the engineering cumulative. Without it, every harness change is a guess, every comparison is confounded, and every claim is unfalsifiable.

The module has six parts and one lab.

By the end of the module you will be able to do eight things:

1. Characterise a benchmark by what it measures, what it does not, and how it can be gamed.
2. Design an evaluation with sufficient power, reporting the calculation before running.
3. Report pass^k correctly under task heterogeneity, with the intra-task correlation.
4. Control multiple comparisons with Holm or Benjamini–Hochberg, and state which is appropriate.
5. Design a model × harness factorial, estimate the interaction, and decompose the variance.
6. Detect contamination and leakage, and state the evidence that a benchmark is clean.
7. Assign an evidence tier and a reproduction status to a claim, including the curriculum’s own citations.
8. Critique a benchmark-reported number, naming the four questions of Module 4, §4.11.2 and the additional questions specific to evaluation.

Everything is self-contained. Where Part 0 is used, the result is restated in applied form.

PART A — THE BENCHMARK LANDSCAPE

5.1 What a benchmark is

5.1.1 The three components

A benchmark is not a dataset. It is a triple:

Definition 5.1.1 (Benchmark). A benchmark is $(\mathcal{T}, \mathcal{V}, \mathcal{P})$: a task distribution $\mathcal{T}$, a verifier $\mathcal{V}$, and a protocol $\mathcal{P}$ for administering the tasks and scoring the results.

The distinction matters because benchmarks with the same name can differ in all three components. “SWE-bench” the dataset is the same; “SWE-bench” the benchmark differs depending on whether the harness gets one attempt or many, whether the tests are run in the same environment, and whether the model sees the test output. A vendor reporting “SWE-bench 45%” is reporting a number from a specific $(\mathcal{T},\mathcal{V},\mathcal{P})$, and the number is not comparable to another vendor’s unless the triple is the same.

5.1.2 What a benchmark measures

A benchmark measures the performance of a (model, harness, protocol) triple on a task distribution, as scored by a verifier. It does not measure “the model’s capability” in any context-free sense. This is the same point as Module 1, Theorem 1.4.2, applied to evaluation: the return depends on the harness, so the benchmark score depends on the harness, so a benchmark score is a property of the triple, not of the model alone.

The practical consequence: a benchmark number without the harness and protocol is not a measurement; it is an anecdote. The rest of Part A characterises the major benchmarks by their triple, so that you can read a reported number and know what it is a number of.

5.2 The major benchmarks

5.2.1 SWE-bench

The triple. $\mathcal{T}$: 2,294 GitHub issues from 12 Python repositories, each with a gold patch and a set of fail-to-pass and pass-to-pass tests. $\mathcal{V}$: the test suite — the patch is applied, the fail-to-pass tests must pass, and the pass-to-pass tests must still pass. $\mathcal{P}$: the model is given the issue text and the repository at the base commit; it produces a patch; the patch is evaluated.

What it measures. Repository-level bug fixing: the ability to localise a bug from an issue description, read the relevant code, produce a patch, and not break existing tests. It is a code benchmark, and it is the closest widely-used proxy for the coding-agent task.

What it does not measure.

• Interaction. The model gets one shot; there is no multi-turn conversation with a user, no clarification, no follow-up.
• Ambiguity. The issue descriptions are curated to be actionable; real issues are often vague.
• Non-code tasks. It is entirely Python code.
• Long-horizon work. The tasks are small (typically one file, a few lines).

Variants. SWE-bench Verified (500 tasks human-validated to be solvable and unambiguous) and SWE-bench Lite (300 tasks, a cheaper subset). The variants change $\mathcal{T}$; the $\mathcal{V}$ and $\mathcal{P}$ are the same. A number on Verified is not comparable to a number on the full set.

Contamination risk. High. The repositories are public, and the issues and patches are in the training data of most models. The test splits are the standard defence, but a model trained on the repository’s git history has seen the patches. The evidence that a SWE-bench result is clean is the temporal split (the model’s training cutoff predates the issue) — and this is rarely reported.

5.2.2 tau-bench

The triple. $\mathcal{T}$: tool-agent-user tasks in two domains (retail and airline), where the agent must interact with a simulated user, call tools, and follow a domain policy. $\mathcal{V}$: the final database state must match the gold state, and the agent must not violate the policy. $\mathcal{P}$: multi-turn interaction with the simulated user; the agent has access to the tools.

What it measures. Tool use with user interaction: the agent must elicit information from the user, call the right tools, and respect a policy (e.g., “do not process a refund for an order older than 30 days”). It is the closest widely-used proxy for a customer-service agent.

What it does not measure.

• Open-ended tasks. The domains are closed-world; the tools and the policy are fixed.
• Long-horizon work. The tasks are short (a few turns).
• Code. It is not a coding benchmark.

The pass^k contribution. tau-bench introduced pass^k as the reliability metric (Part 0, §0.4, Proposition 0.4.5): the probability that the agent succeeds on all $k$ attempts on the same task. The motivation is that a customer-service agent must be reliable, not just capable on average; a 60% success rate that is 60% on every attempt is different from a 60% rate that is 100% on some tasks and 0% on others. pass^k distinguishes them. This is the benchmark’s most important methodological contribution, and it is why the curriculum treats pass^k as a first-class metric.

Contamination risk. Lower than SWE-bench, because the tasks are synthetic and the domains are simulated. But the policy and the tools are in the public repository, and a model can be trained on them.

5.2.3 WebArena

The triple. $\mathcal{T}$: 812 tasks across four self-hosted web applications (a shopping site, a forum, a GitLab instance, a content management system). $\mathcal{V}$: the final state of the application (e.g., the item was added to the cart, the post was created). $\mathcal{P}$: the agent controls a browser and navigates the applications.

What it measures. Web navigation and task completion in realistic applications. It is the closest widely-used proxy for a browser agent.

What it does not measure.

• The open web. The applications are self-hosted and stable; the real web is not.
• Security. The tasks do not include adversarial content (Module 19).
• Non-web tasks.

Contamination risk. Lower than SWE-bench, because the applications are self-hosted and the task instances are synthetic. The task instructions are public, but the environment is reproducible, so a model cannot memorise the state.

5.2.4 OSWorld

The triple. $\mathcal{T}$: 369 real computer tasks across operating-system applications (a file manager, a browser, a code editor, a spreadsheet). $\mathcal{V}$: the final state of the filesystem or the application, checked by a script. $\mathcal{P}$: the agent controls the mouse and keyboard (or an accessibility API).

What it measures. GUI control: the ability to perceive a screen, plan a sequence of clicks and keystrokes, and complete a task. It is the closest widely-used proxy for a computer-use agent.

What it does not measure.

• Reliability under perturbation. The tasks are run once; the variance across runs is not the benchmark’s focus.
• Long-horizon work. The tasks are short (minutes, not hours).

Contamination risk. Moderate: the applications are real (so the model may have seen screenshots in training), but the tasks are synthetic. The main risk is that the model has seen the application interfaces and can navigate them from memory rather than from perception, which would inflate the score relative to a novel application.

5.2.5 AgentBench

The triple. $\mathcal{T}$: eight distinct environments (operating system, database, knowledge graph, card game, puzzles, web shopping, web browsing, house-holding). $\mathcal{V}$: environment-specific verifiers. $\mathcal{P}$: multi-turn interaction in each environment.

What it measures. Multi-dimensional agent capability across diverse environments. Its value is breadth: it tests whether a model’s capability generalises across task types, rather than being specialised to one.

What it does not measure. Depth in any single environment. The breadth is the point, and the cost is that no single environment is as realistic as SWE-bench’s or OSWorld’s.

5.2.6 Terminal-Bench

The triple. $\mathcal{T}$: terminal-based tasks (system administration, file manipulation, process control). $\mathcal{V}$: the final state of the terminal environment, checked by a script. $\mathcal{P}$: the agent has a shell and can run commands.

What it measures. Terminal competence: the ability to use a shell, diagnose a system state, and make changes. It is the benchmark in which the harness study’s ranking-flip result was observed (30th to 5th without changing the model, tier C — the source is a secondary aggregator, and the primary result should be verified before citation).

What it does not measure. Anything outside the terminal. The environment is narrow, and the tasks are short.

The methodological significance. Terminal-Bench is the benchmark where the harness effect was most visible. The ranking flip is the empirical motivation for the factorial design of Part C, and it is the reason the curriculum treats harness-controlled comparison as the central methodological lesson.

5.2.7 BrowseComp and long-horizon suites

BrowseComp. $\mathcal{T}$: browsing tasks that require finding specific, hard-to-find information on the open web. $\mathcal{V}$: the exact answer. $\mathcal{P}$: the agent has a browser and a time budget.

What it measures. Persistence and search strategy: the ability to formulate queries, follow links, and recognise when the answer is found. The benchmark is designed to be hard — the answers are not on the first page of search results — so it rewards the behaviours that a naive agent lacks.

What it does not measure. Anything that requires synthesis or judgement; the answer is a fact, and the task is a search.

Long-horizon suites. METR’s task suite and RE-Bench, and similar suites, measure tasks that take a human expert hours to complete. They are the empirical basis for the time-horizon metric (Module 4, §4.9), and they inherit its caveats: the error bars are a factor of two, the domain variation is 40–100×, and the 50% horizon is not the delegable horizon.

5.2.8 The comparative table

The table is a starting hypothesis, not a leaderboard. Each row’s characterisation is a summary of the benchmark’s design, and the contamination risk is an assessment, not a measurement. The lab asks you to verify the characterisation for the benchmarks you use, from the primary papers.

5.3 What no benchmark measures

Five things are not measured by any of the above, and the harness engineer must supply them.

1 — The deployment distribution. Every benchmark is a sample from some distribution, and the deployment distribution is almost never the same. A model that scores well on SWE-bench may fail on your codebase, because your codebase is not in the 12 repositories. The benchmark measures a capability; the deployment measures a fit. The gap is measured by a domain-specific evaluation set, which you must build.

2 — The cost. Benchmarks report success, not cost. A model that succeeds 5 points more at 10× the token cost is not better for your deployment. The cost per success (Part 0, §0.9) is the deployment metric, and it is not in the benchmark.

3 — The reliability. pass^k is reported by tau-bench and by some others, but most benchmarks report a single success rate, which conflates a model that is 60% on every task with one that is 100% on 60% of tasks. The distinction matters for deployment (Part 0, §0.4, Proposition 0.4.6).

4 — The safety. Benchmarks do not measure whether the agent can be induced to take harmful actions. That is Module 19’s subject, and the benchmarks for it (AgentDojo, arXiv:2406.13352, tier R) are separate.

5 — The harness sensitivity. Most benchmarks report a number from a single harness, and the number is a property of that harness. The sensitivity — how much the number changes under harness variation — is measured by the factorial design, not by the benchmark.

PART B — STATISTICAL PRACTICE

5.4 Variance across seeds

5.4.1 The source of the variance

A model’s output is stochastic: the sampling temperature, the top-p, and the random seed all affect which of several possible outputs is produced. On a single task, the same model can succeed on one run and fail on the next, without any change to the harness or the model.

This is not noise to be averaged away; it is a component of the performance. A model with a 60% success rate that is 60% on every run is different from a model with a 60% rate that is 100% on some runs and 0% on others, and the difference matters for deployment. The variance is the quantity that distinguishes them.

5.4.2 Measuring it

The procedure:

1. Run each task multiple times (at least 3 seeds, ideally 5–10) with the same harness.
2. Record the per-task success vector across seeds.
3. Estimate the per-task success probability $\hat p_t$ and the intra-task correlation $\rho$.
4. Report the mean, the variance across tasks, the variance across seeds within tasks, and $\rho$.

The intra-task correlation $\rho$ is the key quantity for pass^k (Proposition 5.5.2). It is estimated from the variance decomposition of the per-task outcomes: $\rho=\sigma^2_{\text{task}}/(\sigma^2_{\text{task}}+\sigma^2_{\text{seed}})$, where $\sigma^2_{\text{task}}$ is the between-task variance and $\sigma^2_{\text{seed}}$ is the within-task variance.

5.4.3 The reporting standard

A rigorous report includes:

• The number of seeds per task.
• The per-task success distribution (not just the mean).
• $\rho$, or the variance decomposition.
• The pass^k curve (for $k=1,2,3,\dots$) if reliability is relevant to the deployment.

A report that gives only the mean success rate has thrown away the information that distinguishes a reliable model from a lucky one.

5.5 pass^k under heterogeneity

5.5.1 The iid case

Restating Part 0, Proposition 0.4.5: if per-trial success is $p$ and trials are independent,

The iid assumption is that the success probability is the same on every task. It is almost never true.

5.5.2 The heterogeneous case

Restating Part 0, Proposition 0.4.6: if task $t$ has success probability $p_t\sim F$ with mean $p$ and variance $\sigma^2$, then

for $k\ge2$, by the convexity of $x\mapsto x^k$. Task heterogeneity inflates pass^k relative to the iid prediction.

Worked example. $p=0.4$, $\sigma^2=0.04$. iid: $\rho_2=0.16$. Heterogeneous: $\rho_2\approx0.16+\frac{2\cdot1}{2}\cdot0.4^0\cdot0.04=0.16+0.04=0.20$. The heterogeneity inflates pass^2 by 4 points.

5.5.3 The reporting implication

The implication is that reporting $p^k$ computed from the marginal $p$ is misleading, in two directions:

• If the task set is heterogeneous, the true $\rho_k$ is higher than $p^k$, so reporting $p^k$ understates the reliability. A model that is 100% on the easy tasks and 0% on the hard ones has a pass^2 that is higher than $0.4^2$ would suggest.
• If the task set is homogeneous, the true $\rho_k$ is close to $p^k$, and reporting $p^k$ is accurate. The error is in assuming homogeneity without checking.

The correct report is per-task pass^k, estimated from multiple seeds per task, with $\rho$ reported. This is what tau-bench does, and it is the standard the curriculum adopts.

5.5.4 Concept check

A model achieves 60% success on a 100-task benchmark. Is its pass^2 necessarily 36%?

Answer. No. 36% is the iid prediction. If the tasks are heterogeneous — the model is 100% on some tasks and 0% on others — the true pass^2 is higher, because the tasks it succeeds on, it succeeds on every time. The deviation is governed by the task variance $\sigma^2$: $\rho_2\approx0.36+\frac{2\cdot1}{2}\cdot0.6^0\cdot\sigma^2=0.36+\sigma^2$. If $\sigma^2=0.04$, pass^2 is 0.40. To know the true value, you must measure per-task success across seeds.

5.6 Multiple comparisons

5.6.1 The problem

When you test $m$ hypotheses (e.g., $m$ harness edits, or $m$ model × harness cells), the probability of at least one false positive grows with $m$. At $\alpha=0.05$ per test, with $m=10$ independent tests, the family-wise error rate is $1-0.95^{10}=0.40$ — a 40% chance of at least one false positive.

This is the multiple comparisons problem, and it is ubiquitous in harness engineering, because harness development is an iterative process of testing many edits.

5.6.2 Holm’s step-down procedure

Restating Part 0, Proposition 0.4.10: order the p-values $p_{(1)}\le\dots\le p_{(m)}$. Reject $H_{(i)}$ if $p_{(j)}\le\alpha/(m-j+1)$ for all $j\le i$.

Holm controls the family-wise error rate (FWER) at $\alpha$ under arbitrary dependence. It is the right choice when a false positive is costly — when adopting a harness edit that does not work has a real cost (it may degrade performance, or it may consume engineering time that could be spent elsewhere).

5.6.3 Benjamini–Hochberg

Restating Part 0, Proposition 0.4.11: find the largest $k$ with $p_{(k)}\le\frac{k}{m}\alpha$; reject $H_{(1)},\dots,H_{(k)}$.

BH controls the false discovery rate (FDR) at $\alpha$ under independence or positive dependence. It is the right choice when the goal is to find candidates for further investigation, and a false positive is tolerable because it will be caught in a follow-up.

5.6.4 Which to use

The choice is a decision about the cost of errors, not a matter of preference.

Use Holm when:

• The edits will be adopted on the basis of the test.
• A false positive has a real cost (degraded performance, wasted time).
• The number of tests is small (Holm is not much less powerful than BH for small $m$).

Use BH when:

• The tests are a screening step, and the survivors will be tested again.
• A false positive is cheap (it will be caught later).
• The number of tests is large (BH is more powerful for large $m$).

Worked example. Eight harness edits, p-values $0.001,0.004,0.012,0.020,0.031,0.040,0.050,0.200$.

Holm: thresholds $\alpha/8=0.00625$, $\alpha/7=0.00714$, $\alpha/6=0.00833$, $\alpha/5=0.010$, $\alpha/4=0.0125$, $\alpha/3=0.0167$, $\alpha/2=0.025$, $\alpha/1=0.05$. Compare: $0.001\le0.00625$ ✓; $0.004\le0.00714$ ✓; $0.012\le0.00833$ ✗ — stop. Two edits survive.

BH: largest $k$ with $p_{(k)}\le k\alpha/8$: $k=1:0.001\le0.00625$ ✓; $k=2:0.004\le0.0125$ ✓; $k=3:0.012\le0.01875$ ✓; $k=4:0.020\le0.025$ ✓; $k=5:0.031\le0.03125$ ✓; $k=6:0.040\le0.0375$ ✗. Five edits survive (the first five).

The difference is the error criterion: Holm controls the probability of any false positive; BH controls the expected fraction of false positives among the rejections. Both are valid; the choice depends on what a false positive costs.

5.7 Power

5.7.1 The calculation

Restating Part 0, Proposition 0.4.9: to detect $p_2-p_1=\Delta$ at two-sided level $\alpha$ with power $1-\beta$, the required sample size per arm is

5.7.2 The worked number

$p_1=0.60$, $p_2=0.63$, $\Delta=0.03$, $\alpha=0.05$, power $0.80$:

A three-point improvement on a 60% baseline requires about 4,100 tasks per arm. This number should be computed before every comparison. Benchmark suites of a few hundred tasks cannot resolve a three-point effect, and a vendor that reports a three-point improvement on a 300-task suite is reporting a difference that is within noise.

5.7.3 The implication for benchmark reporting

The implication is uncomfortable: most reported improvements on most benchmarks are underpowered. A five-point improvement on a 500-task benchmark has a standard error of approximately $\sqrt{2\times0.5\times0.5/500}=0.032$, so a five-point improvement is about 1.6 standard errors — not significant at the 5% level without a paired design or a much larger sample.

This is why the curriculum insists on paired designs (Part 0, §0.1, Proposition 0.1.8): pairing on the same tasks reduces the variance by the within-task covariance, which can be substantial when the tasks vary in difficulty. A paired design on 500 tasks can detect an effect that an unpaired design on 2,000 tasks cannot.

5.7.4 Concept check

A vendor reports a 3-point improvement on a 300-task benchmark. Is the improvement real?

Answer. Probably not detectable. The standard error on a 300-task benchmark at a 60% baseline is approximately $\sqrt{2\times0.6\times0.4/300}=0.040$, so a 3-point improvement is less than one standard error — well within noise. The improvement might be real, but the evidence does not establish it. To establish it, you would need a paired design on the same 300 tasks (which might suffice if the tasks are heterogeneous and the pairing reduces the variance), or a much larger sample. The four questions of Module 4, §4.11.2 apply, with the additional question: what is the power of the reported test?

PART C — HARNESS-CONFOUNDED COMPARISON

5.8 The ranking-flip phenomenon

5.8.1 The claim

The harness study reports that a coding agent moved from 30th to 5th place on Terminal-Bench 2.0 in March 2026 without changing the model (tier C: the source is a secondary aggregator, and the primary result should be verified before citation). A separate survey reports that task reliability depends less on the underlying model than on the execution harness, and a 2026 arXiv synthesis reports that leaderboard rankings flip under minor harness changes.

The claim is the empirical motivation for this part. The ranking flip means that a benchmark ranking is not a property of the model alone; it is a property of the (model, harness, protocol) triple. The formal statement is Module 1, Theorem 1.4.2, and the evaluation consequence is that a comparison without harness control is confounded.

5.8.2 The evidence status

The ranking-flip claim is tier C: it comes from secondary aggregators, and the primary result has not been independently reproduced. This does not make it false; it makes it a hypothesis. The curriculum treats it as the motivation for the factorial design, and the design is worth running whether or not the specific ranking flip is real, because the confound is structural.

5.9 The factorial design

5.9.1 The model

Restating Part 0, §0.4: for model $i$, harness $j$, replicate $k$,

The interaction term $(\alpha\beta)_{ij}$ is the harness confound. A significant interaction means the model ranking depends on the harness; a non-significant interaction means the ranking is stable across the harnesses tested.

5.9.2 The interaction contrast

The model difference under harness $j$ is

The first term is the harness-independent model effect; the second is the harness-dependent adjustment. If the adjustment is large relative to the main effect, the ranking is harness-dependent, and a single-harness comparison is misleading.

5.9.3 The variance decomposition

Restating Part 0, Proposition 0.4.8, the proportion of variance attributable to the harness is

This is the quantitative answer to “is the harness load-bearing?” If $R^2_{\text{harness}}$ is large relative to $R^2_{\text{model}}$, harness engineering dominates model selection for the task at hand.

Worked example (from Part 0, §0.4.3): a 3×3 design with $K=100$, mean squares $MS_\alpha=0.81$, $MS_\beta=2.40$, $MS_{\alpha\beta}=0.36$, $MS_{\text{err}}=0.19$. The variance components are $\hat\sigma_\alpha^2=0.0015$, $\hat\sigma_\beta^2=0.0068$, $\hat\sigma_{\alpha\beta}^2=0.0017$, $\hat\sigma^2=0.19$. Harness share $=3.4$, model share $=0.75$, interaction $=0.85$, residual $95$. The harness explains more than four times the variance of the model, and the interaction is comparable to the model main effect. Both facts are decision-relevant.

5.9.4 What “fixing the harness” means

A comparison that “holds the harness fixed” must hold fixed all of the following:

• The model version (not just the model family).
• The harness version (the loop, the tools, the context manager, the control mechanisms).
• The tool versions (the schemas, the dispatchers, the external services).
• The seeds (the sampling configuration).
• The time budget (the wall-clock and token limits).
• The task set (the same tasks, in the same order or with the same randomisation).
• The verifier (the same tests, the same checks).

A comparison that fixes some but not all of these is partially confounded, and the confound is usually invisible. The factorial design is the only design that controls all of them simultaneously.

5.10 The pre-registered analysis

5.10.1 Why pre-register

Pre-registration is the practice of fixing the analysis plan before seeing the data. Its purpose is to prevent post-hoc selection: the analyst, seeing the data, chooses the analysis that produces the desired result. The choice can be unconscious — a reasonable analyst will prefer the test that shows a significant effect — which is why the discipline must be external.

5.10.2 What to pre-register

A complete pre-registration includes:

1. The hypotheses, stated in falsifiable form.
2. The primary metric, and the definition of success.
3. The sample size, with the power calculation.
4. The analysis plan: the statistical model, the tests, the multiple-comparison correction, the confidence-interval method.
5. The exclusion criteria: which runs will be excluded and why (decided before seeing the data).
6. The stopping rule: when the data collection ends (fixed $n$, or a sequential design with a stated rule).
7. The deviation log: a record of any deviation from the plan, with the reason.

5.10.3 The deviation log

Deviations are normal; hiding them is not. The log records what changed, when, and why. A deviation that is recorded and justified is acceptable; a deviation that is discovered by a reviewer is a credibility problem.

Example. The plan says “paired bootstrap”; the analyst uses a t-test because the bootstrap is slow. The log records: “t-test used instead of paired bootstrap; the two agree to three decimal places on this data; the reason is runtime.” The deviation is documented, and the reader can judge whether it matters.

5.10.4 Concept check

Why is pre-registration necessary if the analyst is honest?

Answer. Because the analyst’s honesty is not the issue; the issue is the flexibility of the analysis. An honest analyst who sees a non-significant result and tries a different test is not lying — they are exploring. But the exploration inflates the false-positive rate, because the reported test is selected after seeing the data. Pre-registration removes the flexibility by fixing the analysis in advance, so the reported result is the result of one test, not the best of many.

PART D — CONTAMINATION, LEAKAGE, AND REWARD-HACKABLE GRADERS

5.11 Contamination

5.11.1 Definition

Contamination is the presence of benchmark data in the model’s training set. A contaminated model has seen the benchmark’s tasks (or their answers) during training, so its performance on the benchmark overstates its capability on new tasks.

Leakage is a broader term: the presence of any information about the test set in the training or selection process. Contamination is a form of leakage; so is using the test set to select the model (test-set leakage) or to tune the prompt (prompt leakage).

5.11.2 Detection

Contamination is hard to detect and easy to suspect. The available evidence:

Temporal split. The strongest evidence is that the benchmark’s tasks were created after the model’s training cutoff. If the tasks are older, the model may have seen them. The temporal split is rarely reported, and it is the first question a rigorous reader asks.

Canary strings. Some benchmarks embed canary strings — unique tokens that appear only in the benchmark — and check whether the model can reproduce them. A model that can is contaminated. The canary approach is effective for detecting verbatim memorisation, but it does not detect paraphrase or summary contamination.

Held-out task families. The strongest defence is a held-out task family: a set of tasks from a distribution the model was not trained on, administered by the evaluator. The family is not public, so it cannot be in the training data. This is the approach the curriculum recommends for deployment evaluation: build your own held-out set, and do not publish it.

Performance anomalies. A model that performs much better on a benchmark than on a comparable in-distribution task may be contaminated. The anomaly is a signal, not proof.

5.11.3 The practical standard

The practical standard is:

• For benchmark evaluation, report the temporal split (or state that it is unknown), and treat the result as an upper bound on capability.
• For deployment evaluation, use a held-out task family that is not public, and do not publish it.
• For vendor claims, ask for the temporal split and the held-out evaluation; if neither is available, the claim is tier C.

5.12 Test-set leakage

5.12.1 The mechanisms

Test-set leakage occurs when the test set influences the model or the harness in a way that inflates the test score. Three mechanisms:

Selection leakage. The model (or prompt, or harness) is selected on the test set, so the test set is no longer held out. This is the most common form, and it is often unconscious: an engineer tunes the prompt until the test score improves, and the test score is now an in-sample estimate.

Prompt leakage. The test set’s content is included in the prompt during development, and the prompt is then used in evaluation. The model has seen the test content, even if it was not trained on it.

Harness leakage. The harness is tuned on the test set (e.g., the retry policy is set to the value that maximises the test score), so the harness is overfitted to the test set.

5.12.2 The defence

The defence is the train/validation/test split, with the test set used exactly once, after all development is complete. The validation set is used for development; the test set is used for the final report. The split is fixed before development, and the test set is not touched until the report.

The practical difficulty is that the test set is often small, and the temptation to use it for development is strong. The discipline is the same as pre-registration: fix the split, and record any use of the test set.

5.13 Reward-hackable graders

5.13.1 The mechanism

A grader is reward-hackable if the model can achieve a high score without doing the task. The classic examples:

• Test-suite gaming. The model modifies the tests to pass, rather than fixing the code. The verifier runs the modified tests and reports success.
• LLM-judge gaming. The model produces output that the judge rates highly without being correct — flattery, length, confident tone, or the judge’s known biases.
• Metric gaming. The model optimises the metric (e.g., BLEU, exact match) in a way that does not correspond to task quality.

The mechanism is the same as in Module 4, §4.5.4 (PRM reward hacking) and Module 16 (RL reward hacking): the policy sharpens toward what the verifier rewards, which may be a proxy for correctness rather than correctness itself.

5.13.2 The defences

Three defences, each addressing a different mechanism.

Immutable verifiers. The verifier is not modifiable by the agent. The test suite is mounted read-only; the judge is a separate process the agent cannot call; the metric is computed by the harness, not by the agent. This prevents test-suite gaming.

Adversarial evaluation. The verifier is attacked before deployment: red-team the tests to find cases where a wrong solution passes, and the judge to find cases where a bad output is rated highly. This is Module 17b’s verifier engineering, applied to the benchmark.

Held-out verification. The verifier used for evaluation is different from the verifier used during development. The agent cannot overfit to a verifier it has not seen. This is the evaluation analogue of the held-out task family.

5.13.3 The audit

The audit is a required part of any evaluation that uses a learned or gameable verifier. It consists of:

1. A list of known exploits, found by attacking the verifier.
2. A measure of the exploit rate: the fraction of successful runs that used an exploit.
3. A corrected score: the success rate after excluding the exploit runs, or a penalty for the exploit rate.

A score without an audit is a score of unknown validity. The curriculum requires the audit in the Module 16 lab and in the capstone.

5.13.4 Concept check

Why is an LLM judge a reward-hackable grader, and what is the defence?

Answer. Because the judge is a model, and the policy can learn to produce output that the judge rates highly without being correct — the judge has biases (length, confidence, format, flattery) that the policy can exploit. The defence is calibration: measure the judge’s agreement with human labels on a held-out set, report its ROC, and use it at an operating point that is robust to the known biases. If the judge cannot be calibrated to a usable precision, it is not a verifier; it is a heuristic.

PART E — EVIDENCE TIERS AND REPRODUCTION STATUS

5.14 The discipline

5.14.1 The two classifications

The curriculum uses two classifications, restated here for application.

Evidence tier.

• A: peer-reviewed or arXiv-indexed, stable spine.
• B: vendor engineering publications and specifications.
• C: secondary aggregators and leaderboards, used only to locate candidates.

Reproduction status.

• R: independently reproduced in a public harness by ≥2 groups.
• S: single-lab result, methodology inspectable, scaffold released.
• P: preprint, unreplicated, methodology partially inspectable.
• C: vendor claim; must be reproduced before use.

5.14.2 The curriculum’s own citations

The curriculum applies the classification to itself. The examples from the preface:

• arXiv:2609.00006 (eleven-system study): S — source-code analysis, non-peer-reviewed, but inspectable and the scaffold is released.
• arXiv:2604.25850 (AHE): P — single-lab; the Terminal-Bench 2 improvement (69.7% → 77.0%) is unreplicated.
• arXiv:2605.10325 (VPR) and arXiv:2607.07435 (RLVP): P — the theoretical hook is sound; the empirical claims await replication.
• arXiv:2608.05179 (autonomous research survey): P — useful for framing; the survey-derived claims are hypotheses.
• arXiv:2503.14499 (METR time horizon): S, with the caveats of Module 4.
• The ranking-flip claim (Terminal-Bench 30th→5th): C — secondary aggregator.

5.14.3 The application to a claim

For any claim, the procedure is:

1. Identify the primary source. If there is none (a blog post citing a tweet), the claim is C.
2. Assign the tier based on the source type.
3. Assign the reproduction status based on whether independent groups have reproduced it, whether the methodology is inspectable, and whether the scaffold/data is released.
4. State the claim’s status in the form “tier X / status Y”, and qualify the claim accordingly.

Worked example. A vendor blog claims “our harness improves agent success by 20 points.” The source is a vendor publication (tier B), the result is not independently reproduced (status C), and the methodology is not disclosed. The claim is B/C, and it should be treated as a hypothesis to be tested, not a fact to be relied on.

PART F — METRICS THAT MISLEAD UNDER EXTRAPOLATION

5.15 The average-score trap

5.15.1 The problem

A benchmark suite reports an average score across tasks. The average is a single number, and it hides three things:

• The per-task distribution. A 70% average could be 70% on every task (reliable) or 100% on 70% of tasks and 0% on 30% (unreliable). The average does not distinguish them.
• The task weighting. The average weights tasks equally; the deployment may not. A model that is strong on the common tasks and weak on the rare ones has a high average and a poor deployment fit.
• The variance. A high average with high variance is a different operational profile from a high average with low variance.

5.15.2 The remedy

Report the distribution, not just the average: the per-task success histogram, the pass^k curve, and the variance. The average is a summary; the distribution is the measurement.

5.16 Best-of-k versus pass^k

5.16.1 The distinction

Best-of-k reports the probability that at least one of $k$ attempts succeeds: $1-(1-p)^k$. It is the metric for a setting where the agent can try $k$ times and a human or verifier selects the best.

pass^k reports the probability that all $k$ attempts succeed: $p^k$ (iid) or the heterogeneous version of §5.5. It is the metric for a setting where the agent must succeed on every attempt — a reliability setting.

The two are different, and they are appropriate for different deployments.

• A coding agent that produces a patch for human review is a best-of-k setting: the human reviews and selects, so the relevant metric is the probability that at least one attempt is good.
• A customer-service agent that responds to users without review is a pass^k setting: every response must be correct, so the relevant metric is the probability that all attempts are good.

5.16.2 The reporting implication

A vendor that reports best-of-k without saying so is reporting a higher number than the pass^k metric, and the difference is large. At $p=0.5$, best-of-5 is $1-0.5^5=0.969$, while pass^5 is $0.5^5=0.031$. The same model, the same task set, and a 30× difference in the reported number, depending on which metric is used.

The rigorous report states which metric is used, and reports both if the deployment is ambiguous. The four questions of Module 4, §4.11.2 include “is it pass^k or best-of-k?” for this reason.

5.16.3 Concept check

A vendor reports “97% success” on a coding benchmark. What must you ask before believing it applies to your deployment?

Answer. Whether the 97% is best-of-k or pass^k. If it is best-of-5, the per-attempt success may be 50%, and the pass^5 may be 3% — which is a very different deployment profile. You must also ask the four questions of Module 4, §4.11.2 (interval, distribution, 50% vs delegable, reproduction) and the contamination question (temporal split). The 97% is a number of unknown metric, unknown precision, unknown distribution, and unknown contamination status.

LAB 5 — AN EVALUATION PROTOCOL FOR TWO MODELS ACROSS TWO HARNESSES

5.17.1 Task specification

Choose a task from your practice with a deterministic verifier, and select two open-weight models of comparable scale (Module 12’s dossier procedure can be used to select them) and two harness configurations that differ in a single component (e.g., the context manager, or the tool surface).

5.17.2 The design

A $2\times2$ factorial: two models × two harnesses. The primary metric is paired success rate on the same task set (Part 0, §0.1, Proposition 0.1.8). The secondary metrics are pass^2 (with the intra-task correlation $\rho$) and cost per success (Part 0, §0.9).

5.17.3 The pre-registered analysis plan

The plan must include:

1. Hypotheses, in falsifiable form. At minimum: the model main effect, the harness main effect, and the interaction.
2. The power calculation, with the sample size per cell. If the calculation says the study is underpowered at a feasible sample size, state the minimum detectable effect.
3. The analysis model: the two-way factorial, the interaction test, the variance decomposition.
4. The multiple-comparison strategy: Holm or BH, and the justification.
5. The confidence-interval method: paired bootstrap, with the number of resamples.
6. The exclusion criteria and the stopping rule.
7. The deviation log template.

5.17.4 The acceptance criteria

5.17.5 What good looks like

A good lab report has five properties.

It pre-registers and logs deviations. The plan is written before the data, and every change is recorded. The reader can see what was planned and what was done.

It reports the interaction. The interesting result is usually that the model ranking depends on the harness. A report that says “model A had the higher average” has missed the point; the point is where each model wins and whether the ranking is stable.

It reports the power. The minimum detectable effect is stated, so the reader knows what the study could and could not have found. A null result on an underpowered study is not evidence of no effect.

It reports pass^2 with $\rho$. The reliability metric is reported correctly, with the heterogeneity, so the reader can compute the deployment-relevant reliability.

It states the evidence tiers. The models’ claimed numbers are classified, and the reproduction status is stated. The report does not treat a vendor claim as a measurement.

MODULE 5 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E5.1. A model achieves 65% on a 200-task benchmark. Compute the standard error, and the minimum detectable difference at 80% power with a paired design if the intra-task correlation is $\rho=0.3$.

S5.1. Unpaired standard error: $\sqrt{2\times0.65\times0.35/200}=\sqrt{0.455/200}=\sqrt{0.002275}=0.0477$. The minimum detectable difference at 80% power, two-sided: $(1.96+0.84)\times0.0477=2.80\times0.0477=0.134$, i.e. 13.4 points. With a paired design and $\rho=0.3$, the paired variance is $2\sigma^2(1-\rho)=2\times0.2275\times0.7=0.3185$, so the paired standard error is $\sqrt{0.3185/200}=\sqrt{0.00159}=0.0399$, and the minimum detectable difference is $2.80\times0.0399=0.112$, i.e. 11.2 points. Pairing helps, but the study is still underpowered for a 5-point effect. To detect 5 points, you would need $n\approx(2.80\times0.0399/0.05)^2$ — much larger than 200.

E5.2. A 100-task benchmark has per-task success probabilities with mean 0.5 and variance 0.09. Compute pass^2 under iid and under heterogeneity.

S5.2. iid: $0.5^2=0.25$. Heterogeneous: $0.25+\frac{2\cdot1}{2}\cdot0.5^0\cdot0.09=0.25+0.09=0.34$. The heterogeneity inflates pass^2 by 9 points — a large difference, because the task variance is high.

E5.3. Ten harness edits are tested; the p-values are $0.002,0.008,0.015,0.022,0.030,0.041,0.048,0.060,0.150,0.400$. Apply Holm and BH at $\alpha=0.05$, and state which edits survive under each.

S5.3. Holm: thresholds $0.005,0.00556,0.00625,0.00714,0.00833,0.010,0.0125,0.0167,0.025,0.05$. $0.002\le0.005$ ✓; $0.008\le0.00556$ ✗ — stop. One edit survives. BH: largest $k$ with $p_{(k)}\le k\times0.005$: $k=1:0.002\le0.005$ ✓; $k=2:0.008\le0.010$ ✓; $k=3:0.015\le0.015$ ✓; $k=4:0.022\le0.020$ ✗. Three edits survive (the first three). The difference is the FWER/FDR trade-off.

E5.4. A vendor reports a 4-point improvement on a 500-task benchmark, with no confidence interval. What is the standard error, and is the improvement detectable at the 5% level (unpaired)?

S5.4. Assuming a baseline near 50%, the standard error on a single proportion is $\sqrt{0.25/500}=0.0224$, and the standard error on the difference (unpaired) is $\sqrt{2}\times0.0224=0.0317$. The 4-point improvement is $0.04/0.0317=1.26$ standard errors, $p\approx0.21$ two-sided. Not detectable at the 5% level. The improvement might be real, but the evidence does not establish it; a paired design or a larger sample would be needed.

E5.5. A benchmark’s tasks were created in 2023; a model’s training cutoff is 2024. Is the benchmark contaminated?

S5.5. Likely, yes. The tasks predate the training cutoff, so the model may have seen them (or their answers) in training. The temporal split is the wrong way: the tasks are older than the cutoff. The contamination is not certain — the tasks may not have been in the training corpus — but the burden is on the evaluator to show they were not. The rigorous evaluation uses tasks created after the cutoff, or a held-out family that is not public.

E5.6. A verifier for a coding task runs the test suite. The agent can modify the test files. What is the exploit, and what is the defence?

S5.6. The exploit is test-suite gaming: the agent modifies the tests to pass rather than fixing the code, and the verifier reports success. The defence is an immutable verifier: mount the test files read-only, or run the tests from a copy the agent cannot access. The audit: check whether any successful run modified a test file, and exclude or penalise those runs. If the audit finds a high exploit rate, the verifier is not measuring what it claims.

E5.7. A model achieves 50% per-attempt success. A vendor reports “97% success” using best-of-5. What is the pass^5, and what does the difference mean for a deployment that requires reliability?

S5.7. best-of-5: $1-0.5^5=0.969$. pass^5 (iid): $0.5^5=0.031$. The difference is 97% vs 3% — the same model, the same task set, and a 30× difference depending on the metric. For a deployment that requires reliability (every attempt must succeed), the relevant number is 3%, not 97%. The vendor’s 97% is a best-of-k number presented without the qualifier, and it overstates the deployment-relevant reliability by a factor of 30.

E5.8. A 2×2 factorial with $K=100$ gives $MS_\alpha=0.60$, $MS_\beta=0.90$, $MS_{\alpha\beta}=0.20$, $MS_{\text{err}}=0.15$. Compute the variance components and the harness share.

S5.8. $\hat\sigma_\alpha^2=(0.60-0.20)/200=0.002$; $\hat\sigma_\beta^2=(0.90-0.20)/200=0.0035$; $\hat\sigma_{\alpha\beta}^2=(0.20-0.15)/100=0.0005$; $\hat\sigma^2=0.15$. Total $=0.156$. Harness share $=0.0035/0.156=2.2$; model share $=1.3$; interaction $=0.3$; residual $96.2$. The harness explains more than the model, and the interaction is non-trivial relative to the main effects. The residual dominates, which is why the paired design and a large $K$ are essential.

EXIT TEST FOR MODULE 5

You are ready for Module 6 (and the start of Part II) when you can, without notes:

1. Define a benchmark as a triple and explain why a number without the triple is an anecdote.
2. Characterise SWE-bench, tau-bench, WebArena, OSWorld, AgentBench, Terminal-Bench, and BrowseComp by what they measure and what they do not.
3. State the five things no benchmark measures.
4. Measure and report variance across seeds, including the intra-task correlation $\rho$.
5. State and apply the pass^k formula under heterogeneity, and explain why reporting $p^k$ is misleading.
6. Apply Holm and BH, and justify the choice by the cost of a false positive.
7. Perform a power calculation and state the sample size for a 3-point effect at 60%.
8. Design a model × harness factorial, estimate the interaction, and decompose the variance.
9. State what “fixing the harness” means, listing the seven controls.
10. Pre-register an analysis plan and maintain a deviation log.
11. Detect contamination and leakage, and state the evidence that a benchmark is clean.
12. Audit a reward-hackable grader and report a corrected score.
13. Assign an evidence tier and reproduction status to a claim, including the curriculum’s own.
14. Explain the average-score trap and the best-of-k vs pass^k distinction, with the 30× example.

End of Module 5. Part I is complete. Say “next” to receive Module 6 in full (Part II: Harness Engineering begins).