Week 8. Part II begins. Prerequisite: Modules 1–5 and Part 0.

6.0 Module map and objectives

Part I built a harness and taught you to evaluate it. Part II is about engineering the harness — designing, instrumenting, and improving it as a system in its own right, rather than as an ad hoc collection of components that happened to work. This module is the foundation of Part II: it gives you the analytical vocabulary for dissecting a harness, the empirical anatomy from the field’s first systematic study, and — most importantly — a classification of harness components by their durability.

The durability question is the module’s intellectual core. Some harness components exist because models are currently weak; when models improve, those components should be removed. Other components exist because the deployment context imposes requirements that no model improvement can eliminate; those components are permanent. A harness engineer who cannot tell the two apart will either cling to scaffolding that has become dead weight or discard control surfaces that are load-bearing. The classification in Part D is the tool for telling them apart.

The module has four parts and one lab.

By the end of the module you will be able to do seven things:

1. Place any harness component in the ETCLOVG taxonomy and say what the placement buys you analytically.
2. Reconstruct a harness’s architecture from its source, producing a component map and a control-flow diagram.
3. State the eleven-system study’s principal findings, with their evidence status (S) and their limits.
4. Apply the three vendor principles as design heuristics, while distinguishing a heuristic from a law.
5. Classify every component of a harness as irreducible, capability-compensating, or performance-tuning, using a formal procedure.
6. Identify the five assumptions a harness makes about its model, and predict which of them a different model family would violate.
7. State the open research questions about harness convergence and overfitting, and design an experiment to answer one of them.

Everything is self-contained. Where a prior module is used, the result is restated.

PART A — THE ETCLOVG TAXONOMY

6.1 Why a taxonomy

6.1.1 The problem a taxonomy solves

A harness is a complicated object. A production coding harness has a loop, a dozen tools, a context manager with several policies, a permission system, a trace store, a test runner, and a retry policy. Without a decomposition, discussion degenerates into anecdote: “our harness is better because we added a search tool” is not a claim anyone can evaluate, because “better” and “search tool” and “our harness” are all underdefined.

A taxonomy is a partition of the object into named, exhaustive, mutually exclusive categories, chosen so that questions about the object become questions about its parts. The choice of categories is not arbitrary: it should carve the object at the joints where the engineering decisions live, so that a decision in one category can be discussed without entanglement with the others.

Module 1 gave a constitutive decomposition: the four necessary elements (loop, tool interface, context management, control mechanisms). That decomposition answers “what makes this a harness?” It is the right decomposition for defining the object. It is not fine-grained enough for engineering: “control mechanisms” bundles termination, permissions, retries, and budget limits, which are four different engineering problems with four different failure modes.

The ETCLOVG taxonomy is the finer-grained, analytical decomposition. It answers “what are the parts, in enough detail to engineer each one?” The two decompositions are complementary, not competing, and §6.3 shows how they map onto each other.

6.1.2 The status of the taxonomy

ETCLOVG is proposed in the survey literature (Li et al., Agent Harness Engineering: A Survey, cited in the curriculum’s references). It is an analytical convenience, not a discovered natural kind. Its justification is pragmatic: it is useful if it makes engineering discussions clearer. It should be revised if it stops doing so. Treat it as you would treat a good design pattern catalogue — a vocabulary, not a law.

6.2 The seven layers, defined

Each layer is defined by what it governs and by the question it answers.

Definition 6.2.1 (The ETCLOVG layers).

• E — Execution. The environment in which actions run: the sandbox, the filesystem, the network policy, the resource limits. Question it answers: where does the agent’s action take effect, and what can it reach?
• T — Tooling. The action space exposed to the model and the dispatcher that executes it: the schemas, the validation, the result normalisation. Question it answers: what can the agent do, and how is the doing translated into effect?
• C — Context. The assembly, ordering, trimming, compaction, retrieval, and formatting of the model’s input. Question it answers: what does the model see, and what does it not see?
• L — Lifecycle. The control flow: the loop, termination, retries, recovery, multi-context-window execution, handoffs. Question it answers: how does a run start, proceed, recover, and stop?
• O — Observability. The traces, metrics, logs, and the schema that makes them analysable. Question it answers: how do you know what happened, and why?
• V — Verification. The verifiers, their operating points, their calibration, and the autonomy decisions they support. Question it answers: how do you know whether the output is correct, and how much autonomy does that justify?
• G — Governance. Permissions, audit, oversight, compliance, and the policy that constrains the system. Question it answers: what is the system allowed to do, and who is accountable?

6.2.1 The layers are not a pipeline

A common misreading is that the seven layers are stages of a pipeline — execution, then tooling, then context, and so on. They are not. They are concerns, and every step of a run touches several of them simultaneously. A single tool call touches Tooling (the schema and dispatcher), Execution (the sandbox it runs in), Governance (the permission check), Observability (the trace record), and Verification (if the tool’s output is checked). The layers are orthogonal cuts, not sequential stages.

The misreading matters because it leads to an organisational error: assigning one engineer to “the context layer” and another to “the verification layer” as though they were separate subsystems. They are separate concerns that every component participates in. A context manager that does not emit traces is failing the Observability concern; a tool dispatcher that does not enforce permissions is failing the Governance concern. The layers are properties that components should have, not boxes that components live in.

6.3 Mapping the four necessary elements onto ETCLOVG

The two decompositions relate as follows.

Two observations follow.

Observation 1 — The mapping is many-to-many. Each necessary element spans several ETCLOVG layers, and each layer draws on several necessary elements. This is expected: the constitutive decomposition answers a different question from the analytical one, and a many-to-many mapping is the normal relationship between two useful decompositions of the same object.

Observation 2 — Two layers are not covered by any single necessary element: Observability and Verification. Module 1’s four elements are necessary for agency, and neither observability nor verification is necessary for agency in the strict sense — a harness could run without traces or verifiers and still be an agent. But they are necessary for engineering: a harness you cannot observe cannot be debugged, and a harness without verification cannot be safely given autonomy. This is why the ETCLOVG taxonomy is the engineering decomposition: it includes the layers that make the harness governable, not just the layers that make it an agent.

Proposition 6.3.1 (The ETCLOVG layers are necessary for engineering, not for agency). There exists a system that satisfies all four necessary elements and is an agent, but has no Observability or Verification layer. Such a system is an agent but is not engineerable: it cannot be evaluated (Module 5), debugged (Module 11), or safely given autonomy (Part 0, §0.5).

Construction. The Module 1 scaffold without trace emission and without verifier integration satisfies the four necessary elements: it has a loop, a tool interface, a context manager, and control mechanisms. It is an agent. It produces no traces, so failures cannot be attributed; it has no verifier, so success cannot be measured. It is not engineerable. $\square$

6.4 Worked mapping of a real harness

Take the Module 1 scaffold after Lab 1a and map every component.

E — Execution. The tool executors run in the harness’s process. There is no sandbox, no network policy, no resource limit beyond the wall-clock timeout. Assessment: this is the weakest layer. The harness is fine for a trusted environment and unsafe for an untrusted one (Module 7 fixes this).

T — Tooling. Three tools (read_file, write_file, run_tests), each with a typed schema and a dispatcher that validates before execution. Assessment: adequate for the task; the schemas follow the Module 2 design rules.

C — Context. Assembly (system prompt first, history chronological, instruction last), trimming (oldest-first at 8,000 tokens, recorded in Context.dropped), formatting (a single replaceable function). Assessment: the canonical layout; the trimming is crude but recorded, which is what Module 3’s lab needs.

L — Lifecycle. The loop, the four bounds (steps, tokens, tool calls, wall-clock), the loop detector, the retry policy. Assessment: the four bounds are the minimum for robustness; the loop detector is the Module 1 stall test.

O — Observability. The trace file with step index, context estimate, model output, tool calls, tool results, outcome. Assessment: present but minimal — it supports failure attribution but not the AHE-style decision observability of Module 10.

V — Verification. run_tests is the verifier, but its operating characteristics are not measured. Assessment: the verifier exists but is uncharacterised; Module 17b fixes this.

G — Governance. The permission policy is a per-tool allow-list on argument patterns, enforced in the dispatcher. Assessment: deterministic enforcement, which is the right kind; the audit log is the trace file, which is not tamper-evident (Module 20 fixes this).

The mapping is the diagnostic. It tells you where the harness is strong (T, C, L), where it is adequate (O, G), and where it is weak (E, V). The Lab 6 asks you to do the same for a production harness, where the weaknesses are less obvious and the assessment more consequential.

6.4.1 Concept check

Why is the ETCLOVG taxonomy useful if Module 1’s four elements already define a harness?

Answer. Because the four elements define the object at the granularity needed to say what makes it an agent, and the ETCLOVG layers decompose it at the granularity needed to engineer each part. “Control mechanisms” is one element for definitional purposes and three layers (Lifecycle, Governance, Verification) for engineering purposes. The two decompositions answer different questions, and a serious harness engineer needs both: the constitutive one to know what must be present, the analytical one to know what must be done well.

PART B — THE ELEVEN-SYSTEM STUDY

6.5 The study and its method

6.5.1 What it is

The study (arXiv:2609.00006, evidence tier S: source-code analysis, non-peer-reviewed, but methodology inspectable and the scaffold released) analyses eleven production coding harnesses plus a meta-harness. The eleven are Claude Code, Codex CLI, Gemini CLI, Mistral Vibe, OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, and OpenClaw. The meta-harness is an aggregate analysis across them.

6.5.2 The method

The method is source-code analysis: the authors read the harnesses’ source, identified their components, classified them against a taxonomy, and extracted the design decisions that recur. The method has strengths and limits, and both matter for reading the results.

Strengths.

• Direct evidence. The authors are reading the actual code, not press releases or blog posts. The components are what they are, not what the documentation says they are.
• Cross-system comparison. Eleven systems read by the same authors with the same taxonomy yields a comparison that a single-system case study cannot.
• Inspectability. The methodology is described, and the scaffold is released, so a reader can check the claims.

Limits.

• Non-peer-reviewed. The study is a preprint. Its claims have not been through external review.
• Single-lab. One group of authors did the analysis. Their taxonomy choices shape the results; a different group with a different taxonomy might see different patterns.
• Snapshot. The harnesses are analysed at a point in time. Harnesses change monthly; the findings are a snapshot, not a permanent description.
• Coding harnesses only. All eleven are coding harnesses. The findings may not generalise to GUI agents, research agents, or customer-service agents, whose action spaces and verification regimes differ.
• Adoption is not effectiveness. The study reports which components are adopted where; it does not measure which components make a harness better. The adoption counts are descriptive, not causal.

The evidence-status consequence. Treat the study’s findings as a well-founded hypothesis, not as established fact. The descriptive findings (which harnesses have which components) are as reliable as the source-code reading; the inferential findings (why the components are there, which are better) are hypotheses.

6.5.3 The headline descriptive findings

Three findings are worth stating up front, because the rest of the module builds on them.

Finding 1 — The four necessary elements are universal. All eleven systems contain a loop, a tool interface, a context manager, and control mechanisms. This is the empirical confirmation of Module 1’s constitutive definition: the definition was an argument from necessity, and the study finds that the systems that work all satisfy it.

Finding 2 — Skills files out-adopt MCP. Skills files are adopted in 9 of 11 harnesses; MCP in 8 of 11. The numbers are close, and the study reports adoption, not effectiveness. The finding is descriptive: skills are at least as widely used as MCP, despite being a simpler mechanism. Module 2, §2.4.4 discusses the plausible explanations (simplicity, portability, expressiveness) and the caveat (adoption is not causation).

Finding 3 — Behavioural policy is migrating into configuration. The study reports a longitudinal pattern: behaviours that were once specified in prompt prose are increasingly specified in configuration files and structured settings. This is the convergence-and-imitation finding, developed in §6.8.

6.6 The 18 design recommendations

6.6.1 Status and framing

The study extracts 18 design recommendations from the cross-system analysis. The recommendations are the study’s synthesis: patterns that recur across systems, stated as guidance. They are tier S and should be read as hypotheses with strong descriptive support.

The exact wording of each recommendation is in the primary text; a self-sufficient module should not paraphrase 18 items as though the paraphrase were the original. What the module can do — and what is more useful — is give the structure of the recommendations and a representative example from each category, so that you know what kind of guidance to expect and can read the primary text with the right schema.

6.6.2 The structure of the recommendations

The 18 recommendations fall into five categories, aligned with the ETCLOVG layers they address.

Category 1 — Execution and sandboxing (E). Recommendations about isolation, resource limits, and reproducibility. Representative example: run tools in an ephemeral sandbox with an explicit network egress policy, so that a compromised or misbehaving run cannot affect the host or reach arbitrary endpoints. Why it recurs: every production harness faces the same threat model, and the same primitives (containers, microVMs, egress rules) are available.

Category 2 — Tool surface design (T). Recommendations about schema granularity, validation, and result normalisation. Representative example: normalise every tool outcome — success, error, timeout, denial — into a single result shape, so that the context manager and the loop do not branch on outcome type. Why it recurs: the alternative (branching on outcome) is a source of silent failures that every team discovers independently.

Category 3 — Context policy (C). Recommendations about trimming, compaction, and layout. Representative example: put stable content (system prompt, schemas) first and volatile content last, which simultaneously matches the attention curve and maximises prefix-cache hit rate (Modules 3 and 6, §3.11). Why it recurs: the layout is a free improvement, and teams converge on it once they measure cost.

Category 4 — Lifecycle and control (L, G). Recommendations about termination, retry, and permission. Representative example: enforce permissions deterministically in the dispatcher, not through prompt instructions, so that the policy does not depend on the model complying. Why it recurs: the alternative is a known security failure, and the fix is cheap.

Category 5 — Observability and verification (O, V). Recommendations about tracing and verifier integration. Representative example: emit a trace record at every step with enough fields to support failure clustering and cost attribution. Why it recurs: the alternative is undebuggable, and the cost of retrofitting traces is high (Module 1, §1.2.6).

6.6.3 How to use the recommendations

The recommendations are checklists, not laws. The procedure for using them:

1. Read the primary text for the exact wording of each recommendation.
2. Classify each by the ETCLOVG layer it addresses.
3. For each, ask: does my harness do this, and if not, is the reason principled or accidental? A deviation with a principled reason (the constraint does not apply) is fine; a deviation by accident is a defect.
4. Do not adopt a recommendation without measuring its effect on your task. The recommendations are drawn from coding harnesses; their transfer to your domain is a hypothesis (the study’s limit, §6.5.2).

The fourth step is the one that matters. The recommendations are the field’s accumulated wisdom, and accumulated wisdom is a good prior, not a substitute for measurement. This is the same discipline the curriculum applies to vendor benchmarks (Module 5) and to published results (the reproduction-status tags): the recommendation is a candidate, and your harness is the test.

6.7 The 90-line scaffold

6.7.1 What it is

The study releases a minimum-viable-harness scaffold of roughly ninety lines: a complete, working harness that satisfies the four necessary elements and nothing more. It is the reference implementation of Module 1’s Definition 1.1.1, and you built its equivalent in Lab 1a.

6.7.2 Why it matters

The scaffold’s value is calibrational. It shows that the minimum is small — a harness that satisfies the constitutive definition fits in ninety lines. Everything beyond those ninety lines is an addition, and every addition is a design decision that should be justified. The scaffold makes the justification obligation visible: a 10,000-line harness is not wrong, but it is 9,900 lines of additions, and each one should have a reason.

Proposition 6.7.1 (The addition obligation). Every component beyond the minimal scaffold should be classifiable as (i) irreducible, (ii) capability-compensating, or (iii) performance-tuning (§6.13). A component that cannot be classified is either unnecessary or not yet understood, and should be investigated before it is kept.

Argument. The scaffold satisfies the constitutive definition; a component beyond it is not required for agency. It must therefore earn its place by one of the three justifications: it enforces a deployment requirement (irreducible), it compensates for a model weakness (compensating), or it improves a performance metric (tuning). A component that does none of these is doing nothing, and a component whose justification is unknown cannot be maintained correctly. The classification is the maintenance discipline. $\square$

6.7.3 The scaffold and the curriculum

The curriculum’s Lab 1a is deliberately aligned with the scaffold: same four elements, same minimality, same acceptance criteria. If you built Lab 1a from the specification, you have the scaffold. The alignment is not coincidental — it is the reason Module 1 was written the way it was. The scaffold is the reference, and the curriculum’s specification is a self-sufficient description of it.

6.8 Convergence and imitation

6.8.1 The finding

The study reports a longitudinal pattern: across the harnesses and over time, behaviours that were once specified in prompt prose are increasingly specified in configuration files, structured settings, and explicit control flow. The study calls the resulting similarity across harnesses convergence, and the mechanism by which later harnesses adopt earlier harnesses’ patterns imitation.

6.8.2 The interpretation

Three readings are available, and the study does not fully adjudicate between them.

Reading 1 — Convergence on the correct design. The harnesses are converging because there is a best way to build a harness, and the field is discovering it. The stable components (permissions, verification, observability) are stable because they are right; the volatile components (prompt scaffolds) are volatile because they are being replaced by structured equivalents.

Reading 2 — Convergence by imitation, not by evidence. The harnesses are converging because engineers copy each other, not because the copied design is better. The popularity of a pattern is a social fact, not an engineering one. On this reading, the convergence is a fashion, and the “best practices” are the practices of the most-imitated team.

Reading 3 — Convergence on a local optimum, with overfitting to the coding domain. The harnesses are converging because they all solve the same task (coding) with the same model families, and the convergence is specific to that setting. On a different task, the converged design would be wrong.

The three readings are not mutually exclusive. A component can be correct and imitated and domain-specific. The point of distinguishing them is to know what evidence would tell you which reading applies to a given component, which is the research agenda of §6.16.

6.8.3 The imitation mechanism and its risk

The imitation mechanism is observational: a team sees another team’s harness succeed, infers that a component caused the success, and adopts the component. The inference is valid only if the component is the cause, which requires a comparison that the observing team usually does not have.

Proposition 6.8.1 (Imitation is confounded by the whole harness). Observing that harness $H_2$ outperforms harness $H_1$ and adopting a component $c$ from $H_2$ does not establish that $c$ causes the improvement. The improvement may be due to any other component of $H_2$, or to the interaction of $c$ with the rest of $H_2$, or to the model $H_2$ uses. The adoption is justified only by a controlled comparison, which is Module 5’s factorial design.

Proof sketch. $H_2$ differs from $H_1$ in many components. The observed performance difference is $J(H_2)-J(H_1)$, which by Module 1, Theorem 1.4.2 depends on all the differences. Isolating $c$‘s contribution requires holding the other differences fixed, which is a controlled experiment. Without it, the attribution is a post-hoc narrative, and the adoption may import a component that does nothing (or hurts) in the new context. $\square$

This is the reason Module 5’s discipline is a prerequisite for Part II: without harness-controlled comparison, “best practice” adoption is a confounded inference, and the convergence finding is a description of fashion as much as of engineering.

6.9 The study’s limits, restated

The limits of §6.5.2 are worth restating as a checklist for reading the study’s findings.

The fifth limit is the one most often missed. “Skills files are adopted in 9 of 11 harnesses” is a fact about adoption. It is not evidence that skills files improve outcomes. The study is careful about this; readers are not always.

6.9.1 Concept check

The study reports that skills files are adopted in 9 of 11 harnesses and MCP in 8 of 11. What can you conclude, and what can you not?

Answer. You can conclude that both mechanisms are widely adopted, and that skills are at least as widespread as MCP in this sample of coding harnesses at this point in time. You cannot conclude that skills are better — the study reports adoption, not effectiveness, and it does not control for the tasks, the models, or the teams. The finding is a reason to investigate skills, not a reason to adopt them without measurement. This is the same distinction as Module 5’s evidence tiers: a tier-S descriptive finding is a well-founded hypothesis, not an established causal claim.

PART C — VENDOR GUIDANCE

6.10 Anthropic’s three principles

6.10.1 The guidance

Anthropic’s April 2026 engineering guidance (tier B: vendor publication, primary text located via the awesome-harness-engineering index) offers three principles for harness design. They are stated here as the vendor states them, with the caveats that tier-B status requires.

Principle 1 (Build on tools the model already knows). Prefer tool surfaces that resemble the interfaces the model was trained on — file operations, search, shell commands — over novel abstractions the model has not seen. The rationale: the model’s prior is a free source of competence, and a tool surface that exploits the prior requires less specification and produces fewer errors.

Principle 2 (Remove harness assumptions as capabilities improve). Harness components that compensate for a model weakness should be removed when the weakness is fixed. The rationale: scaffolding that remains after the need is gone is dead weight — it costs tokens, adds failure modes, and constrains the model’s behaviour.

Principle 3 (Set UX, cost, and safety boundaries deliberately). The harness should impose explicit, deliberate constraints on user experience (latency, verbosity), cost (budgets, routing), and safety (permissions, verification). The rationale: constraints that are not set deliberately are set accidentally, by default behaviour that no one chose.

6.10.2 The status of the principles

The three principles are tier B: they are vendor engineering publications, not peer-reviewed research. Their status is that of expert heuristics — the accumulated judgement of a team that has built production harnesses, stated as guidance.

The correct posture toward a tier-B heuristic is neither acceptance nor dismissal. It is translation and testing:

1. Translate the principle into a measurable prediction about your harness.
2. Test the prediction on your task, with the Module 5 discipline.
3. Adopt or reject on the evidence, and record the outcome.

Example translation. Principle 1 predicts: a tool surface that resembles a familiar interface (a shell command) will have a lower specification cost and a lower error rate than a novel abstraction (a custom DSL) for the same operation. The test: build both surfaces, run Lab 2’s three-surface ablation, compare interface inefficiency and error-recovery rate. The principle gives you a prior; the lab gives you the answer for your task.

6.10.3 Principle 2 is the module’s thesis

Principle 2 — remove assumptions as capabilities improve — is the practical form of the bitter-lesson tension that Part D formalises. It says: some components are temporary, and the harness should shed them. The classification of §6.13 is the tool for knowing which ones.

The principle is also the hardest to apply, because it requires two judgements that are easy to get wrong.

Judgement 1 — Is the capability actually improved? A model release that claims an improvement may not have it on your task (Module 5’s reproduction discipline). Removing a compensating component on the basis of a vendor claim, without measuring, is how harnesses regress.

Judgement 2 — Is the component actually compensating? A component that looks like scaffolding may be irreducible. A retry policy looks like compensation for model unreliability, but it is also a defence against tool stochasticity (Module 1, §0.2.9), which no model improvement fixes. Removing it because the model got better would be a mistake. This is exactly the classification problem of Part D.

6.11 Reading vendor guidance critically

6.11.1 The four questions

Vendor guidance is not a benchmark claim, so the four questions of Module 4, §4.11.2 do not apply directly. The analogous questions are:

1. What is the evidence behind the principle? Is it a measured result, a design rationale, or an assertion? A principle backed by a measurement is stronger than one backed by a rationale.
2. On what domain was it validated? A principle validated on coding tasks may not transfer to GUI or research tasks. The domain is the scope condition.
3. What is the counterexample? Every heuristic has cases where it fails. The vendor’s text may not state them; the reader must look for them.
4. What would falsify it? A principle that cannot be falsified is not a design heuristic; it is a slogan.

6.11.2 The synthesis with the study

The vendor principles and the eleven-system study are complementary sources with different evidence status: the study is S (descriptive, inspectable, single-lab), the vendor guidance is B (expert heuristic, vendor publication). Where they agree, the agreement is weak evidence for the principle (two independent sources converged). Where they disagree, the disagreement is a research question.

An example of agreement. Principle 1 (build on familiar tools) is consistent with the study’s finding that tool surfaces converge on familiar interfaces. Two sources, different methods, same conclusion: weak evidence that the principle is sound, and a good prior for your own design.

An example of a tension. Principle 2 (remove assumptions as capabilities improve) is in tension with the study’s finding that harnesses are converging — adding the same components rather than shedding them. The tension may be resolved by the distinction between compensating components (which should be shed) and irreducible ones (which converge because they are permanent). That resolution is a hypothesis, and it is the subject of Part D.

6.11.3 Concept check

Why is “the vendor says so” neither sufficient nor irrelevant?

Answer. It is not sufficient because a vendor publication is tier B — an expert heuristic, not a peer-reviewed measurement — and heuristics can be wrong, domain-specific, or motivated by the vendor’s interests. It is not irrelevant because the vendor has built production harnesses and its accumulated judgement is a genuinely useful prior, especially where it agrees with independent evidence. The correct posture is to treat it as a prior to be tested, not a fact to be applied.

PART D — COMPONENT CLASSIFICATION

6.12 The bitter-lesson tension

6.12.1 The tension

The bitter lesson in machine learning is the observation that general methods which scale with compute tend to outperform methods that encode human domain knowledge, and that this has been true across the field’s history. The lesson is “bitter” because it implies that carefully engineered domain-specific structure is often wasted effort.

Applied to harness design, the tension is:

• The harness is domain-specific structure. It encodes assumptions about how the model should behave, what tools it should use, and how context should be managed. On the bitter-lesson view, this structure is scaffolding that stronger models will make obsolete.
• The harness is also the control surface. It enforces permissions, verifies outputs, and records traces. These are not model-capability compensations; they are properties of a governed deployment. On this view, the structure is permanent.

Both views are right about different components. The tension is not a contradiction; it is a signal that “the harness” is not one thing, and that the bitter-lesson argument applies to some of its parts and not others.

6.12.2 The question the classification answers

The classification answers: for a given component, does the bitter lesson apply?

• If yes, the component is capability-compensating: it exists because the model is weak, and it should be removed when the weakness is fixed.
• If no, the component is either irreducible (it exists because the deployment requires it) or performance-tuning (it exists because it improves a metric, and the metric may persist).

The classification is the resolution of the tension: the bitter lesson applies to one class, not to the harness as a whole.

6.13 The three classes

6.13.1 Definitions

Definition 6.13.1 (Irreducible component). A component is irreducible if it enforces a requirement that is imposed by the deployment context rather than by the model’s limitations. Formally, a component is irreducible if, for every model $M$ — including a hypothetical model that solves the task perfectly — the component is still required for the deployment to be acceptable.

Examples: permission enforcement (no model, however capable, should be allowed to delete production data without a check), audit logging (the deployment requires a record of what happened), verification of irreversible actions (the deployment requires a check before the action, regardless of the model’s confidence).

Definition 6.13.2 (Capability-compensating component). A component is capability-compensating if it exists to work around a limitation of the current model, and would be unnecessary if the limitation were removed. Formally, a component is compensating if there exists a model $M^*$ such that, with $M^*$, the component is not required for the deployment to be acceptable, and the component’s presence does not improve the deployment.

Examples: few-shot exemplars that teach the model a format, prompt scaffolds that decompose a task the model cannot decompose alone, retry policies that compensate for the model’s failure to recover from its own errors (as distinct from retries for tool stochasticity, which are irreducible).

Definition 6.13.3 (Performance-tuning component). A component is performance-tuning if it improves a measured metric (success, cost, latency) without being required for acceptability, and without being a compensation for a specific model limitation. Formally, a component is tuning if its removal degrades a metric but does not make the deployment unacceptable, and its benefit is not tied to a particular model’s weakness.

Examples: compaction thresholds, parallelism settings, cache layouts, model routing. The metric they improve is real, but the deployment would be acceptable (if slower or more expensive) without them.

6.13.2 The counterfactual test

The definitions are stated in terms of hypothetical models, which makes them precise but not directly testable. The operational procedure is the counterfactual test: for each component, ask the counterfactual question, and classify by the answer.

Procedure 6.13.4 (Counterfactual classification). For a component $c$ in a harness $H$:

1. Irreducibility test. Ask: if the model were perfect — it solved the task with certainty, understood every instruction, and never made an error — would $c$ still be required for the deployment to be acceptable? If yes, $c$ is irreducible. The reason is the deployment’s requirements, not the model’s limitations.
2. Compensation test. Ask: if the model were perfect, would $c$ be unnecessary (it would neither be required nor improve anything)? If yes, $c$ is capability-compensating. The reason is a specific model limitation that the component works around.
3. Tuning test. Ask: if the model were perfect, would $c$ still improve a metric (cost, latency) without being required? If yes, $c$ is performance-tuning. The reason is an engineering trade-off, not a model limitation.

The test is a counterfactual, so it is a judgement, not a measurement. Its value is that it forces the judgement to be explicit and to be about the right question — not “does this component help?” (almost everything helps a little) but “what kind of help is it, and will it persist?”

6.13.3 The common confusions

Three classifications are commonly got wrong, and each has a diagnostic.

Confusion 1 — A retry policy classified as compensating. A retry policy looks like a workaround for model unreliability, so it is tempting to classify it as compensating. But retries also handle tool stochasticity — transient network errors, rate limits, flaky services — which no model improvement fixes. A retry policy that handles only model parse failures is compensating; one that handles tool failures is irreducible. The diagnostic: what fraction of retries are triggered by model errors vs tool errors? If the tool fraction is non-negligible, the policy is (at least partly) irreducible.

Confusion 2 — A verifier classified as tuning. A verifier looks like an engineering nicety — it slows the run down and costs compute — so it is tempting to classify it as tuning. But a verifier is what makes autonomy rational (Part 0, §0.5): without it, the deployment cannot be given autonomy, because the expected cost of missed failures exceeds the review cost. The diagnostic: does the deployment require autonomy? If yes, the verifier is irreducible, because it is the condition for autonomy.

Confusion 3 — A prompt scaffold classified as irreducible. A prompt scaffold that decomposes a task looks like part of the task specification, so it is tempting to classify it as irreducible. But if the scaffold exists because the model cannot decompose the task alone, it is compensating. The diagnostic: does the scaffold encode a requirement of the task, or a limitation of the model? A scaffold that says “first read the file, then edit it” encodes the task’s structure (irreducible only if the structure is required); a scaffold that says “think step by step” encodes a model limitation (compensating).

6.13.4 The classification is model-relative

A component’s class is not absolute; it is relative to the model and the deployment. A component that is compensating for one model may be irreducible for another (a model that cannot follow a permission instruction needs the dispatcher check; a model that can still needs it, because the check is irreducible — but a model that cannot parse a schema needs the schema simplified, which is compensating). And a component that is tuning for one deployment may be irreducible for another (a verifier is tuning for a low-stakes task where review is cheap, and irreducible for a high-stakes task where autonomy requires it).

Proposition 6.13.5 (Classification is relative to (model, deployment)). The class of a component $c$ is a function of the model $M$ and the deployment $D$: $c$ may be irreducible in $D_1$ and tuning in $D_2$, or compensating for $M_1$ and irreducible for $M_2$. The classification must therefore state its referents.

Proof. The irreducibility test quantifies over models but not over deployments: it asks whether $c$ is required for the deployment to be acceptable, and acceptability is a property of the deployment. A verifier is required for a deployment that needs autonomy (the cost of missed failures exceeds review) and not for one where every output is reviewed. The compensation test quantifies over models: it asks whether a perfect model would make $c$ unnecessary, and the answer depends on what the current model cannot do. Hence both tests are relative. $\square$

The practical consequence: a classification table is only meaningful with a stated model and deployment. “This component is compensating” is shorthand for “compensating for model $M$ in deployment $D$.”

6.14 The classification procedure

6.14.1 The steps

For each component in a harness, the procedure is:

1. Name the component precisely, and describe what it does.
2. State the model it is designed for, and the deployment it is designed for.
3. Apply the counterfactual test (Procedure 6.13.4): irreducible, compensating, or tuning.
4. Record the evidence for the classification: the fraction of retries that are model-triggered (for retry policies), the autonomy requirement (for verifiers), the task structure (for scaffolds).
5. Record the removal condition: for a compensating component, the model improvement that would justify removing it; for a tuning component, the metric trade-off that would justify removing it; for an irreducible component, the deployment change that would make it unnecessary (if any).
6. Record the removal test: the measurement that would confirm the condition. This is the pre-registration discipline applied to component maintenance.

6.14.2 Worked classification

Take the Module 1 scaffold plus its Lab 1a additions, and classify each component.

The table is illustrative. The point is not the specific classifications — reasonable people can disagree about some of them — but the structure: every component has a class, an evidence basis, and a removal condition. A harness whose components have this structure is maintainable; one whose components are a pile of “we needed this” is not.

6.14.3 The removal protocol

For a compensating component, the removal is a pre-registered experiment:

1. State the removal condition (the model improvement or measurement that justifies removal).
2. State the removal test (the comparison that would show the component is no longer needed).
3. Run the test with the Module 5 discipline: same model, same task set, component present vs absent, paired bootstrap, confidence interval.
4. Remove only if the test shows no degradation (or an improvement, if the component was also costing something).
5. Record the outcome in the component’s register, so the removal is a decision with a history, not a silent deletion.

The protocol is the practical form of Anthropic’s Principle 2: remove assumptions as capabilities improve, but only on evidence, and only with a record.

6.14.4 Concept check

A component that retries failed tool calls: is it irreducible, compensating, or tuning?

Answer. It depends on what the retries are for. If they are for model-generated malformed calls, the component is compensating — a perfect model would not generate malformed calls. If they are for tool stochasticity (network errors, rate limits, flaky services), the component is irreducible — no model improvement fixes the tool. In practice, most retry policies are mixed, and the right move is to split them: classify the model-triggered retries as compensating (with a removal condition based on the parse-failure rate) and the tool-triggered retries as irreducible (kept regardless). The classification is a diagnostic that tells you to split the policy, not a label to put on the whole.

6.15 Worked example: classifying a production harness

6.15.1 The harness

Consider a coding harness with: a loop; ten tools (read, write, edit, search, run-tests, git operations); a context manager with summarisation compaction; a permission policy with a workspace allow-list; a loop detector; a retry policy; a test-suite verifier; a trace store; and a model router that sends simple tasks to a cheap model and complex tasks to a strong model.

6.15.2 The classification

6.15.3 The implications

Three implications follow, and each is a decision the harness team should make explicitly.

Implication 1 — The compensating components are the removal candidates. Summarisation compaction, the loop detector, and the model-triggered half of the retry policy are the components that a stronger model might make unnecessary. They are the ones to watch as models improve, and the ones with a pre-registered removal test.

Implication 2 — The irreducible components are the investment candidates. The permission policy, the verifier, and the trace store are permanent. The team should invest in them — a better verifier (Module 17b), a better audit log (Module 20), a better permission model (Module 7) — rather than treat them as maintenance overhead.

Implication 3 — The tuning components are the optimisation candidates. The router is where cost engineering happens (Module 22); it is worth tuning, but its removal would not make the deployment unacceptable.

The three implications are the practical payoff of the classification: it tells the team where to look for each kind of improvement.

6.16 The research agenda

6.16.1 The open questions

The classification raises questions the curriculum cannot answer, and they are worth stating as a research agenda — the bridge to Module 14b.

Question 1 — Are the compensating components converging or diverging? The study reports convergence, but the classification predicts that compensating components should diverge as models improve (different teams shed different scaffolds at different rates) while irreducible components converge (they are permanent). A longitudinal study that classifies components and tracks their adoption would test this.

Question 2 — Does harness evolution encode general knowledge or benchmark-specific overfitting? A harness that improves on a benchmark may be overfitting to the benchmark’s task distribution. The test: does the improvement transfer to a held-out task family (Module 5, §5.11.2)? If it does, the knowledge is general; if not, it is overfit. This is the experiment Module 14b asks you to design.

Question 3 — What is the minimal harness that preserves safety properties? As capability increases, the compensating components can be shed, but the irreducible ones cannot. The minimal safe harness is the set of irreducible components, and its characterisation is a research problem: which components are irreducible for which deployments?

Question 4 — Can component classification be automated? The counterfactual test is a judgement. A tool that classified components by analysing their code — or by running the removal experiment automatically — would make the maintenance discipline cheap enough to be routine.

6.16.2 The experiment to design

Module 14b will ask you to design a longitudinal experiment for one of these questions. The design for Question 2, as an example:

1. Select a component classified as compensating.
2. Measure its effect on the development task set (Module 5’s design).
3. Measure its effect on a held-out task family the harness was not tuned on.
4. Compare the two effects. If the effect is large on the development set and small on the held-out set, the component is overfit to the development distribution; if it is large on both, the component’s benefit is general.
5. Pre-register the prediction (the classification predicts that compensating components overfit, because they are tuned to the current model’s specific weaknesses).

The design is a contribution whether the prediction holds or not: a replicated overfitting result would be a significant finding, and a failure to replicate would be evidence that the compensating components are more general than the classification assumes.

6.16.3 Concept check

Why is “which components are converging?” a research question rather than a matter of observation?

Answer. Because observation can tell you which components are adopted (the study’s finding) but not whether the adoption is convergence on a correct design or imitation of a fashion (§6.8.2). Distinguishing the two requires a longitudinal study that tracks adoption and measures effectiveness, with the Module 5 discipline — and that study has not been done. The observation is the data; the interpretation is the research question.

LAB 6 — REVERSE-ENGINEER ONE OPEN-SOURCE HARNESS

6.17.1 Task specification

Select one open-source harness from the eleven-system study’s list (or another production harness with public source): Claude Code, Codex CLI, Gemini CLI, Mistral Vibe, OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, OpenClaw. Read its source.

6.17.2 Deliverables

Document 1 — Component map. For every component, place it in the ETCLOVG taxonomy. The map should be exhaustive: every module of the harness’s source should be assigned to a layer, and a component that spans layers should be noted as such.

Document 2 — Control-flow diagram. A diagram of the run lifecycle: the loop, the tool dispatch, the context assembly, the termination conditions, the error paths. The diagram should be detailed enough that a reader could implement the loop from it.

Document 3 — Five model assumptions. Identify five assumptions the harness makes about its model, and for each, state which model family might violate it. Examples: the model emits valid JSON; the model respects a token budget; the model does not emit the same tool call indefinitely; the model’s chat template has a specific format; the model’s reasoning tokens are separated from its output. The assumptions are the harness’s implicit contract with the model, and they are where porting fails (Module 13).

Document 4 — Component classification. For every component in the map, apply Procedure 6.13.4 and produce a table like §6.14.2: component, class, evidence, removal condition / test. State the model and deployment the classification is relative to (Proposition 6.13.5).

6.17.3 Acceptance criteria

6.17.4 What good looks like

A good lab report has four properties.

It is exhaustive. Every component is placed and classified. A map with a “miscellaneous” category is a map that has not been finished; the miscellaneous components are usually the interesting ones (the retry logic, the error handling, the recovery paths).

The assumptions are specific. “The model follows instructions” is not an assumption worth listing — every harness assumes it. “The model’s tool-call output is a JSON object with a name and arguments field, and the arguments are a flat object with string values” is specific, and it is the kind of assumption that a model with nested argument schemas would violate.

The classifications are argued, not asserted. Each row has an evidence basis and a removal condition. A table of labels without evidence is a guess with a column header.

The mixed components are identified. Most real harnesses have components that are partly compensating and partly irreducible (the retry policy, the context manager). Identifying them is the diagnostic that leads to the split.

MODULE 6 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E6.1. A harness has a component that reformats the model’s output into a canonical JSON structure before parsing it. Classify it, with evidence and a removal condition.

S6.1. It depends on why the reformatting exists. If it exists because the model emits JSON with inconsistent formatting (extra whitespace, different key orders, trailing commas), it is compensating: a model with reliable structured output would not need it. If it exists because the downstream parser requires a specific canonical form regardless of the input, it is tuning (or irreducible, if the downstream requirement is a deployment constraint). The evidence: measure the reformatter’s effect on parse-failure rate. If the rate is negligible without it, the reformatter is compensating for a weakness the current model may not have; if the rate is high without it, it is doing real work. Removal condition: a model with a parse-failure rate below the threshold at which the reformatter’s cost is justified.

E6.2. The eleven-system study reports that skills files are adopted in 9 of 11 harnesses. A colleague says “we should adopt skills files.” What is wrong with the inference, and what would you do instead?

S6.2. The inference is from adoption to recommendation, which is not licensed: the study reports what is used, not what works (§6.9). The colleague’s inference is the imitation confound of Proposition 6.8.1 — observing that successful harnesses use skills does not establish that skills cause the success. Instead: (1) identify the property that makes skills attractive (simplicity, progressive disclosure, procedural expressiveness); (2) hypothesise which of your harness’s needs that property addresses; (3) build a minimal skills mechanism and measure its effect on your task, with the Module 5 discipline; (4) adopt only if the measurement supports it.

E6.3. Classify a model router (cheap model for simple tasks, strong model for complex ones) and justify the classification.

S6.3. Performance-tuning. The deployment is acceptable without it (it would be more expensive, but it would work). A perfect model would make the router unnecessary in the sense that the cheap model’s failures would vanish — but a perfect model would also make the router’s benefit vanish, because there would be no cheap-vs-strong trade-off to exploit. The router is tuning because it improves a metric (cost) without being required for acceptability, and its benefit is an engineering trade-off rather than a workaround for a specific model limitation. Removal condition: a cost regime where the routing overhead exceeds the savings (e.g., if the router’s misrouting rate is high enough that the escalation cost dominates).

E6.4. A permission policy is enforced both by a prompt instruction (“do not delete files outside the workspace”) and by a dispatcher check. Which is irreducible, and why?

S6.4. The dispatcher check is irreducible; the prompt instruction is compensating (at best) or redundant (at worst). The dispatcher check enforces the policy independently of the model’s compliance — a perfect model would still be checked, because the check is a deployment requirement, not a model limitation. The prompt instruction relies on the model’s compliance, which a perfect model would satisfy — so the instruction is compensating: it exists because the model might not comply. If the model reliably complied, the instruction would be unnecessary; the check would still be needed, because the deployment requires enforcement regardless of the model’s behaviour. The pair is the guide/sensor distinction of Module 9: the instruction is a guide, the check is enforcement.

E6.5. Apply the counterfactual test to the context formatter. Is it irreducible or compensating?

S6.5. Irreducible, but for a subtle reason. The formatter exists to present the context in the model’s expected format (the chat template, the special tokens, the role markers). A perfect model still needs its input in the format it expects — the format is a property of the model’s interface, not a workaround for its weakness. The formatter’s content may be compensating (if it adds scaffolding), but the formatter function is irreducible: every model has an input format, and the harness must produce it. The distinction is between the formatter as a mechanism (irreducible) and the formatter’s specific choices (which may be compensating or tuning). This is an example of the mixed-component case.

E6.6. The study reports convergence across harnesses. Give an observation that would distinguish “convergence on the correct design” from “convergence by imitation.”

S6.6. A longitudinal study that tracks adoption and measures effectiveness: if the components that are adopted are also the components whose removal degrades performance (measured on a held-out task family), the convergence is on the correct design. If the adopted components’ removal does not degrade performance — or if the adoption precedes any measurement of effectiveness — the convergence is imitation. The distinguishing observation is the correlation between adoption and measured effect: convergence-on-correct-design predicts a positive correlation; convergence-by-imitation predicts none.

E6.7. A harness team removes its summarisation compaction after a model upgrade, and success drops by 12 points. What went wrong, and what would you have done differently?

S6.7. The removal was made on the basis of a model upgrade, without the removal test of §6.14.3. The classification said the compaction was compensating, which was correct — it compensates for the effective-context limit — but the model upgrade did not fix the limit: the new model’s effective context was still shorter than the task’s trajectories. The removal condition was “a model with effective context ≥ the task’s longest trajectory,” and the team checked only that the model was newer, not that it met the condition. What to do differently: (1) state the removal condition precisely (a measured effective length, not a model version); (2) run the pre-registered comparison (compaction present vs absent, same model, same task set); (3) remove only if the test shows no degradation. The lesson is that “the model improved” is not the same as “the condition for removal is met.”

E6.8. You are designing a new harness for a domain where the tasks are long (hours) and the failure cost is high. Which components are irreducible, and which are compensating, and how does the deployment change the classification relative to a low-stakes coding harness?

S6.8. Irreducible: the loop and bounds (constitutive and cost control); the permission policy (high failure cost makes enforcement essential); the verifier (the autonomy decision requires it — Part 0, §0.5 — and high failure cost makes the autonomy threshold strict); the trace store (audit is required for a high-stakes deployment); checkpointing and rollback (long tasks need recovery from partial failure, and the failure cost makes the recovery essential). Compensating: summarisation compaction (still compensates for the effective-context limit, which a better model may fix); the loop detector (still compensates for stalling); the model-triggered retries (still compensate for model errors). Tuning: the router (cost optimisation), the compaction threshold. The deployment changes the classification in two ways: (1) the verifier and audit move from tuning to irreducible, because the high failure cost makes them conditions for acceptable deployment; (2) checkpointing, which might be tuning in a short-task harness, becomes irreducible, because long tasks require recovery and the failure cost makes the recovery essential. This is Proposition 6.13.5 in action: the same component can be a different class in a different deployment.

EXIT TEST FOR MODULE 6

You are ready for Module 7 when you can, without notes:

1. Define the seven ETCLOVG layers and state the question each answers.
2. Explain why the layers are concerns, not stages, and give an example of a single step touching several layers.
3. Map the four necessary elements onto ETCLOVG, and explain why Observability and Verification are engineering-necessary but not agency-necessary.
4. State the eleven-system study’s method, its three headline findings, and its five limits.
5. Characterise the 18 recommendations by category, and state how to use them without treating them as laws.
6. Explain the 90-line scaffold’s calibrational value and state the addition obligation.
7. Explain convergence and imitation, state the imitation confound, and give the observation that distinguishes convergence-on-correct-design from convergence-by-imitation.
8. State Anthropic’s three principles and the correct tier-B posture toward them.
9. State the bitter-lesson tension and explain why it is not a contradiction.
10. Define the three component classes and apply the counterfactual test to a component.
11. State the three common misclassifications and their diagnostics.
12. Explain why classification is relative to (model, deployment), and give an example of a component that changes class.
13. Apply the classification procedure to a harness and produce a table with evidence and removal conditions.
14. State the removal protocol and explain why “the model improved” is not a removal condition.

End of Module 6. Say “next” to receive Module 7 in full.