Weeks 21–22. Prerequisite: Modules 1–13, especially Module 6 (classification), Module 12 (selection), Module 13 (serving), and Part 0, §§0.4, 0.9.

14.0 Module map and objectives

Modules 1–13 built a harness around a model, then selected and served models for it. This module addresses the fact that the harness and the model are not independent: a harness tuned for one model family is, in general, not optimal for another, and the interaction between the two is often larger than either’s main effect. The module makes this quantitative — the 3×3 factorial and its variance decomposition — and then applies it to the two architectures that exploit the non-independence: capability-tiered fleets (a strong planner, cheap executors, verification-gated escalation) and distillation (the harness’s behaviour moved into a smaller model).

The module has five parts and one lab (the Part III capstone).

By the end of the module you will be able to do nine things:

1. Name the four adaptation surfaces (tool schemas, reasoning formats, compaction thresholds, retry policy) and diagnose a foreign-harness failure on each.
2. Build a co-adaptation matrix and read the adaptation cost off it.
3. Design a capability-tiered fleet (planner, executor, verifier) and compute its cost per correct task.
4. State the routing and escalation conditions, and the verification-gated escalation’s discipline.
5. Design a distillation pipeline and audit it for reward hacking.
6. Choose a sovereignty pattern (air-gapped, VPC, hybrid) and state its constraints.
7. Run a 3×3 factorial, decompose the variance, and interpret the interaction as the co-adaptation measure.
8. Design a longitudinal experiment for a component’s compensating/irreducible classification, with a pre-registered prediction.
9. State the four co-evolution research questions and the evidence that would answer each.

Everything is self-contained. Where Part 0 or a prior module is used, the result is restated.

PART A — CO-ADAPTATION

14.1 The non-independence

14.1.1 The claim

Modules 1–13 treated the harness and the model as separable: the harness is the configuration $H$, the model is the policy $\pi_\theta$, and the return is $J(\pi_\theta;\mathcal{M}_H)$ (Module 1, Theorem 1.4.2). The separability is a formal convenience; it is not an empirical independence.

Definition 14.1.1 (Co-adaptation). A harness $H$ and a model $M$ are co-adapted if $H$‘s performance is better with $M$ than with a model $M'$ of comparable capability, and $M$‘s performance is better with $H$ than with a harness $H'$ of comparable design. The co-adaptation is the interaction effect of Module 5, §5.9: the model’s ranking depends on the harness, and vice versa.

The co-adaptation is the Module 1, Theorem 1.4.2’s empirical content: the return depends on the pair $(H, M)$, not on $H$ and $M$ separately, and the interaction term is the quantitative measure.

14.1.2 The sources

Four sources, each corresponding to an adaptation surface (§14.2).

Source 1 — The tool-call format. The model is trained with a specific tool-call format (the Qwen <tool_call>, the Llama <|python_tag|>, the Hermes <tool_call>, the Mistral [TOOL_CALLS]). A harness whose parser expects a different format produces the parse failures of Module 12, §12.12 and Module 13, §13.15. The format is the most common source of co-adaptation.

Source 2 — The reasoning format. The model separates its reasoning (the thinking tags) or mixes it. A harness whose parser does not handle the format produces the truncation and the confusion of Module 13, §13.13.

Source 3 — The context’s conventions. The model’s training distribution determines the effective context (Module 3, §3.2) and the compaction’s tolerance. A harness tuned for a model with a 32k effective length will over-compact for a model with a 128k effective length, wasting the context; a harness tuned for a 128k model will under-compact for a 32k model, degrading the performance.

Source 4 — The retry and recovery policy. The model’s error-recovery behaviour determines the retry’s value. A model that recovers well from a returned error needs fewer retries; a model that repeats the error needs more. The retry policy tuned for one model is suboptimal for the other.

14.1.3 The measurement’s question

The co-adaptation is measurable: the 3×3 factorial (Part E) crosses three models with three harness configurations, and the interaction term is the co-adaptation’s magnitude. The module’s lab is the measurement, and the rest of Part A is the vocabulary for interpreting it.

14.2 The four adaptation surfaces

14.2.1 The surfaces

Definition 14.2.1 (Adaptation surfaces). The four surfaces on which a harness must adapt to a model family:

1. The tool schemas. The tool-call format, the argument encoding, the parallel-call support, the schema’s granularity.
2. The reasoning formats. The reasoning’s separation, the reasoning’s length’s budget, the stop tokens.
3. The compaction thresholds. The trimming’s threshold, the summarisation’s policy, the effective context’s assumption.
4. The retry policy. The retry’s count, the backoff, the error’s handling, the recovery’s prompt.

The four surfaces are the harness’s model-specific parameters: they are the settings that must be re-tuned when the model changes. The surfaces are the Module 13, §13.16’s format tests’ object (surface 1 and 2) plus Module 3’s compaction (surface 3) and Module 1’s retry (surface 4).

14.2.2 The surface-by-surface adaptation

Surface 1 — The tool schemas. The adaptation: the parser’s format (the model’s native template or the grammar), the argument’s type (the object vs the string), the parallel-call handling (the model’s multi-call support), the schema’s granularity (the model’s tool-selection ability). The test: the four format tests (Module 13, §13.16.1), the parse-failure rate (Module 12, §12.12).

Surface 2 — The reasoning formats. The adaptation: the parser’s reasoning extraction (the separated vs the mixed), the reasoning’s budget (the model’s expected reasoning length), the stop tokens (the model’s turn’s end). The test: the reasoning test and the stop-token test (Module 13, §13.13.4, §13.14.3).

Surface 3 — The compaction thresholds. The adaptation: the trimming’s threshold (the model’s effective context, measured per Module 3, §3.2), the summarisation’s policy (the model’s tolerance for lossy summaries), the context layout (the model’s attention’s curve). The test: the Module 3 lab’s projection error and compaction inefficiency.

Surface 4 — The retry policy. The adaptation: the retry’s count (the model’s recovery’s rate), the error’s message (the model’s sensitivity to the message’s content), the recovery’s prompt (the model’s instruction-following). The test: the error-recovery rate (Module 2’s lab).

14.2.3 The adaptation’s order

The surfaces have an order: the tool schemas and the reasoning formats are hard constraints (a model whose format is not parsed cannot run at all), and the compaction and the retry are soft parameters (the model runs, but suboptimally). The order is the adaptation’s sequence: fix the hard constraints first (the format tests), then tune the soft parameters (the lab’s 3×3).

14.3 The foreign-harness failure taxonomy

14.3.1 The failures

When a harness built for model A runs model B, the failures fall into a taxonomy, and the taxonomy is the adaptation’s checklist.

14.3.2 The diagnosis

The diagnosis’s procedure:

1. Run the format tests (Module 13, §13.16.1). A failure is a surface-1 or surface-2 issue, and the test’s name identifies it.
2. Measure the effective context (Module 3, §3.2.3). A gap between the harness’s assumption and the model’s measurement is a surface-3 issue.
3. Measure the error-recovery rate (Module 2’s lab). A low rate is a surface-4 issue.
4. Measure the parse-failure rate (Module 12, §12.12). A high rate is a surface-1 issue, even if the format test passes (the test may not cover the model’s edge cases).

The procedure’s order is the hard-constraints-first: the format tests, then the context, then the retry. The order is the adaptation’s cost’s order (Module 12, §12.16.4): the cheap checks first.

14.4 The co-adaptation matrix

14.4.1 The matrix

Definition 14.4.1 (Co-adaptation matrix). For a set of models $M_1,\dots,M_I$ and a set of harness configurations $H_1,\dots,H_J$, the co-adaptation matrix is the $I\times J$ table of the pair’s performance:

The matrix’s interaction is the deviation from the additive model: $A_{ij} - (\hat\mu + \hat\alpha_i + \hat\beta_j)$, which is the $(\alpha\beta)_{ij}$ of the factorial (Part E).

14.4.2 The adaptation cost

Definition 14.4.2 (Adaptation cost). For model $M_i$, the adaptation cost relative to the best harness is

where $j^*(i)$ is the harness the deployment actually uses. The cost is the performance the deployment leaves on the table by not using the model’s best harness.

Worked example. A 3×3 matrix with models $M_1,M_2,M_3$ and harnesses $H_1$ (tuned for $M_1$), $H_2$ (tuned for $M_2$), $H_3$ (tuned for $M_3$):

The matrix’s diagonal is each model’s best harness — the co-adaptation is real: each model performs best with its own harness. The adaptation costs: $M_1$‘s cost if the deployment uses $H_2$ is $0.72-0.65=0.07$; $M_2$‘s cost if the deployment uses $H_1$ is $0.74-0.61=0.13$; $M_3$‘s cost if the deployment uses $H_1$ is $0.70-0.58=0.12$.

The lesson: the harness’s choice is worth 7–13 points, which is comparable to a model upgrade. A deployment that selects a model without re-tuning the harness leaves that on the table.

14.5 The adaptation’s cost-benefit

14.5.1 The decision

The adaptation is an investment: the engineering time to re-tune the four surfaces, versus the performance gain. The decision:

Proposition 14.5.1 (The adaptation’s decision). Adapt the harness to model $M_i$ iff

where $c_i$ is the adaptation cost (Definition 14.4.2), $V$ is the value of a success, and $T_{\text{adapt}}$ is the engineering time’s cost.

Argument. The adaptation’s benefit is the performance gain $c_i$ times the value $V$; the cost is the engineering time $T_{\text{adapt}}$. The adaptation is worth it iff the benefit exceeds the cost. The comparison is the same as Module 2’s grammar-constrained decision (Module 2, §2.1.3), applied to the harness’s tuning. $\square$

14.5.2 The worked decision

The adaptation cost $c_i=0.10$, the value $V=1.0$, and the engineering time’s cost $T_{\text{adapt}}=0.06$ (in the same units). The adaptation’s benefit is $0.10$, which exceeds the cost $0.06$: adapt. If the adaptation cost were $0.03$, the benefit would be below the cost: do not adapt (the gain does not justify the time).

14.5.3 The surface’s priority

The adaptation’s investment should be allocated by the surface’s marginal benefit per unit of effort:

• Surface 1 (the tool schemas) is usually the highest priority: the parse failure’s cost is the largest (Module 12, Proposition 12.12.1’s $1/(1-p)$ premium), and the fix (the template, the parser) is usually cheap.
• Surface 2 (the reasoning formats) is next: the truncation and the leak are the silent failures, and the fix (the stop tokens, the parser) is cheap.
• Surface 3 (the compaction) is the third: the fix (the threshold, the layout) is cheap to measure but requires the $L_{\text{eff}}$‘s measurement (Module 3’s lab).
• Surface 4 (the retry) is the fourth: the fix (the count, the message) is cheap, but the effect is usually smaller.

The order is the cost’s and the effect’s, and it is the same order as the diagnosis’s (§14.3.2).

14.6 Concept check

A harness tuned for model A is used with model B, and the success rate drops by 8 points. Is this a model capability problem or a co-adaptation problem?

Answer. It is probably a co-adaptation problem, but the diagnosis must confirm it. The 8-point drop could be (1) the model B’s capability (a genuine difference), (2) the co-adaptation (the harness’s surfaces are mis-tuned for B), or (3) both. The diagnosis: run the format tests (a failure is co-adaptation), measure the effective context (a gap is co-adaptation), and measure the error-recovery rate (a low rate is co-adaptation). If the tests pass and the rates are comparable, the drop is the model’s capability. The 3×3 factorial (Part E) is the confirmation: if $M_B$ with its own harness recovers the 8 points, the drop is co-adaptation; if it does not, the drop is the model’s.

PART B — CAPABILITY-TIERED ARCHITECTURES

14.7 The planner–executor split

14.7.1 The architecture

A capability-tiered architecture assigns different models to different roles:

Definition 14.7.1 (The tiered architecture). A tiered architecture has three roles:

• The planner: the strong model, which decomposes the task, chooses the strategy, and reviews the result. It runs a few times per task.
• The executor: the cheap model, which performs the steps (the tool calls, the edits, the retrievals). It runs many times per task.
• The verifier: the model or the deterministic checker, which validates the executor’s output. It runs once per step.

The split’s rationale: the planner’s work is reasoning-heavy (the decomposition, the strategy), and the executor’s work is format-heavy (the tool calls, the edits). The strong model’s advantage is the reasoning; the cheap model’s advantage is the cost. The split assigns each role to the model that is best at it per unit cost.

14.7.2 The cost model

Proposition 14.7.2 (The tiered cost). For a task with $n$ executor steps, the tiered architecture’s cost is

where $C_p$ is the planner’s cost per task, $C_e$ is the executor’s cost per step, and $C_v$ is the verifier’s cost per step. The single-model alternative’s cost is $C_{\text{single}} = (n+1)\,C_s$, where $C_s$ is the strong model’s cost per step.

The tiered architecture is cheaper iff $C_p+n(C_e+C_v) < (n+1)C_s$, which is satisfied when $C_e+C_v \ll C_s$ and the planner’s cost is amortised over the steps.

Worked example. $C_s=0.10$, $C_e=0.02$, $C_v=0.005$, $C_p=0.10$, $n=20$. Tiered: $0.10+20\times0.025=0.10+0.50=0.60$. Single: $21\times0.10=2.10$. The tiered is 3.5× cheaper.

14.7.3 The success model

The tiered architecture’s success is the product of the roles’ successes, if each role must succeed:

where $p_p$ is the planner’s success (the decomposition’s correctness), $p_e$ is the executor’s per-step success, and $p_v$ is the verifier’s correction (the probability the verifier catches an executor’s error and the planner’s revision fixes it).

The product’s form is the reliability’s cost: the executor’s per-step success $p_e$ must be high, because the $n$-th power of a per-step success decays fast. At $p_e=0.95$ and $n=20$: $0.95^{20}=0.358$. At $p_e=0.99$: $0.99^{20}=0.818$. The executor’s reliability is the tiered architecture’s binding constraint.

14.7.4 The verification-gated escalation

The verifier’s role is not just to catch the executor’s errors; it is to decide when to escalate to the strong model. The escalation’s rule:

Definition 14.7.3 (Verification-gated escalation). The verifier’s rejection triggers the escalation to the planner (or the strong model), which either fixes the step or re-decomposes the task. The escalation’s frequency is the verifier’s rejection rate, and its cost is the strong model’s cost per escalation.

The escalation’s condition is the Module 11, §11.17.3’s cascade’s condition, applied to the step: escalate iff $C_e < p_e^{\text{ver}} C_s$, where $p_e^{\text{ver}}$ is the probability the executor produces a verifier-accepted step. The escalation is the cascade, at the step’s granularity.

14.8 The routing

14.8.1 The routing’s problem

The routing problem (Module 11, §11.17.1) is: for each task, which model (or role) should handle it? The routing’s criterion is the Module 11, §11.17.2’s: the pair that maximises the expected value net of cost:

14.8.2 The routing’s features

The routing’s features are the task’s properties that predict which model will succeed:

• The task’s type (a bug fix, a feature, a refactor).
• The task’s complexity (the number of files, the number of steps).
• The task’s context (the repository’s size, the tests’ availability).
• The task’s history (the model’s past performance on the type).

The features are the contextual bandit’s context, and the routing’s policy is the bandit’s. The exploration-exploitation trade-off is the routing’s learning: the router must explore the models on the new task types (to learn their performance) and exploit the known ones (to maximise the reward).

14.8.3 The router’s evaluation

The router’s evaluation is the Module 5’s: the routing policy’s performance vs the best single model’s, on the held-out tasks. The comparison is the routing’s value: a router that beats the best single model is a router whose features are informative; a router that matches it is a router whose features are not (the single model is the best).

14.9 The cascade

14.9.1 The mechanism

Restating Part 0, §0.9, Proposition 0.9.5 and Module 11, §11.17.3: the cascade runs the cheap model first, and escalates to the strong model on the verifier’s rejection.

Definition 14.9.1 (The cascade’s cost). The cascade’s expected cost is

where $p_c^{\text{ver}}$ is the cheap model’s verifier-accepted rate, and the escalation’s condition is $C_c < p_c^{\text{ver}} C_s$.

14.9.2 The cascade’s accuracy

The cascade’s accuracy is the precision of the composed verifier (Module 11, §11.22.2): the accepted outputs are the cheap model’s correct ones plus the strong model’s correct ones, and the wrong outputs are the ones the verifier accepted falsely. The cascade’s accuracy is bounded by the verifier’s precision, and the verifier’s precision is the cascade’s design’s input.

14.9.3 The cascade’s latency

The cascade’s latency is the cheap model’s latency plus the escalation’s probability times the strong model’s latency. For a task with $p_c^{\text{ver}}=0.7$, the expected latency is $0.3$ of the strong model’s, plus the cheap model’s. The cascade’s latency is lower than the strong model’s alone, because most tasks are handled by the cheap model.

14.10 The tier assignment

14.10.1 The roles’ requirements

Each role has a requirement, and the requirement determines the model.

The planner’s requirement: the reasoning quality (the decomposition, the strategy, the review). The strong model is the choice, and its cost is amortised over the task’s steps.

The executor’s requirement: the tool-call reliability (the format’s compliance, the schema’s accuracy). The mid-tier model with the native tool template (Module 12, §12.14.2) is the choice, and the parse-failure rate is the selection’s input.

The verifier’s requirement: the precision (Module 7, §7.8). The deterministic verifier (the test suite, the type checker) is the first choice; the LLM judge is the fallback, with its calibration (Module 17b).

14.10.2 The assignment’s procedure

Procedure 14.10.1 (The tier assignment).

1. List the roles (the planner, the executor, the verifier).
2. For each role, list the candidates (the models, the verifiers).
3. For each candidate, measure the role’s metric (the planner’s success, the executor’s parse-failure rate, the verifier’s precision).
4. For each candidate, compute the cost per correct task (Part 0, §0.5).
5. Choose the candidate with the lowest cost per correct task, subject to the role’s requirement.

The procedure is the Module 12’s dossier, applied to the roles. The measurement is the same: the harness’s own, not the vendor’s.

14.10.3 The worked assignment

The assignment: the mid model as the planner (the cost per correct task is lower, and the requirement is the reasoning’s sufficiency, which the mid model meets on the simple tasks); the small model as the executor (the cost per correct task is lower, but the parse rate is high, and the requirement is the reliability, so the mid model is the choice if the reliability’s requirement binds); the test suite as the verifier (the deterministic verifier is the choice).

The lesson: the assignment is the cost per correct task’s, not the capability’s. The strong model’s role is the planner only if the planner’s reasoning’s requirement is high enough that the mid model’s cost advantage is outweighed by its success’s loss.

14.11 The fleet’s cost model

14.11.1 The three terms

The fleet’s cost is the sum of the three roles’ costs, plus the routing’s and the escalation’s overheads:

where $n_r$ is the role’s calls per task, $C_r$ is the role’s cost per call, and the routing and the escalation are the overheads.

14.11.2 The fleet’s cost per correct task

The fleet’s cost per correct task is the total cost divided by the success:

where $p_{\text{fleet}}$ is the fleet’s success (the product of the roles’ successes, plus the verifier’s correction).

14.11.3 The fleet’s Pareto frontier

The fleet’s configurations are the (model, role) assignments, and the frontier is the set of the non-dominated assignments (Module 11, §11.18.1). The deployment’s choice is the frontier’s point that meets the accuracy’s requirement, and the frontier’s measurement is the fleet’s lab.

14.12 Concept check

A tiered architecture has a strong planner and a weak executor. The executor’s per-step success is 0.92, and the task has 15 steps. What is the executor’s contribution to the success, and what would you do?

Answer. The executor’s contribution is $0.92^{15}=0.286$ — the executor succeeds on all 15 steps only 28.6% of the time. The 0.92 per-step success is not high enough for a 15-step task: the compounding is the tiered architecture’s binding constraint (Proposition 14.7.2’s product). The action: (1) raise the executor’s per-step success (a better executor, a stronger model, the grammar-constrained decoding to eliminate the parse failures); (2) add the verifier-gated escalation (the verifier catches the executor’s errors, and the planner fixes them); (3) decompose the task into fewer steps (the planner’s decomposition’s granularity). The verifier is the cheapest fix if the errors are detectable; the executor’s upgrade is the fix if the errors are not.

PART C — DISTILLATION INTO SMALLER MODELS

14.13 The pipeline

14.13.1 The mechanism

Harness-Zero (Module 10, §10.11, tier P) distils the harness’s behaviour into a smaller model: the harness’s trajectories are the demonstrations, and the student is trained on them.

Definition 14.13.1 (The distillation pipeline).

1. Collect the demonstrations. The teacher (the strong model + the harness) runs on the task set, and the trajectories are recorded (the prompts, the outputs, the tool calls, the results).
2. Filter the demonstrations. The trajectories are filtered by the verifier (the successful ones are kept), and the reward-hacking audit (the exploits are removed, Module 5, §5.13).
3. Train the student. The student is trained (SFT) on the filtered trajectories, with the format’s target (the tool-call format, the reasoning format).
4. Evaluate the student. The student’s pass^k (Module 5, §5.5), the parse-failure rate (Module 12, §12.12), and the cost per correct task (Part 0, §0.5).
5. Iterate. The failing trajectories are the next round’s data, and the loop repeats.

14.13.2 What distils

The distillation transfers three things, and each has a different reliability.

The format. The tool-call’s syntax, the reasoning’s format, the stop tokens. The format is the most reliably distilled (it is a pattern, and the SFT learns it well). The format’s distillation is the harness’s compensating components’ removal (Module 6, §6.13.2): the student learns the format, and the harness’s parser’s scaffolding can be reduced.

The behaviour. The strategy (the decomposition, the tool’s selection, the error’s recovery). The behaviour is partially distilled (the student learns the common patterns, and it fails on the rare ones). The behaviour’s distillation is the harness’s compensating components’ partial removal.

The knowledge. The task’s domain (the code’s conventions, the tool’s semantics). The knowledge is the least reliably distilled (the student’s capacity is smaller, and the knowledge is compressed lossily). The knowledge’s distillation is the model’s capability, not the harness’s.

The three things’ distinction is the distillation’s scope: the format’s distillation is the harness’s win; the behaviour’s is a partial win; the knowledge’s is the model’s upgrade, not the harness’s.

14.14 The reward-hacking risk

14.14.1 The mechanism

The distillation’s demonstrations are the teacher’s successful trajectories, and the success is measured by the verifier. If the teacher has a verifier exploit (Module 5, §5.13), the successful trajectories include the exploits, and the student learns them.

Proposition 14.14.1 (The distillation amplifies the exploit). If the teacher’s successful trajectories include a fraction $e$ of verifier exploits, the student learns the exploit’s pattern, and the student’s exploit rate is at least $e$ (and often higher, because the SFT reinforces the pattern). The distillation’s success rate is inflated by the exploit’s rate, and the student’s true performance is lower.

Argument. The SFT’s objective is the likelihood of the demonstrations; the exploit’s trajectories are in the demonstrations, so the student’s likelihood of the exploit’s pattern is increased. The exploit’s pattern is a short, high-reward sequence, and the SFT’s gradient favours it. The student’s exploit rate is therefore at least the demonstrations’ rate, and the amplification occurs because the pattern is reinforced across the demonstrations. $\square$

14.14.2 The audit

The audit is the Module 5, §5.13.3’s: the exploit list, the exploit rate, the corrected score. The distillation’s audit is the teacher’s audit (the exploits are found in the teacher’s trajectories) and the student’s audit (the exploits are found in the student’s outputs). The student’s audit is the one that matters: a student that learned the exploit is a student that is not deployable.

The audit’s procedure:

1. Run the teacher on the task set, and find the exploits (the successful trajectories that used a verifier hack).
2. Filter the demonstrations: remove the exploits’ trajectories.
3. Train the student on the filtered set.
4. Run the student on the same task set, and find the exploits.
5. Compare the teacher’s and the student’s exploit rates; the student’s rate should be lower (the filter removed the teacher’s exploits), and a higher rate is the amplification’s signal.

14.15 The evaluation

14.15.1 The metrics

The student’s evaluation is the Module 5’s, with the distillation’s additions:

14.15.2 The comparison’s design

The comparison is the paired design (Module 5, §5.4): the same tasks, the same harness, the teacher and the student. The distillation is worth it iff the student’s cost per correct task is lower than the teacher’s, and the student’s pass^k is acceptable for the deployment (Module 4, §4.10.3’s delegable horizon).

14.15.3 The failure’s modes

Failure 1 — The student’s capability is too low. The student’s success is below the deployment’s requirement, and the distillation is not worth it. Diagnosis: the student’s pass^k’s distribution.

Failure 2 — The student’s format is wrong. The student learned the content but not the format (the tool-call syntax), and the parse-failure rate is high. Diagnosis: the parse-failure rate.

Failure 3 — The student’s exploit rate is high. The student learned the teacher’s exploit, and the success is inflated. Diagnosis: the audit.

Failure 4 — The student’s reliability is low. The student’s pass^k is low even when the success is high, because the student is inconsistent. Diagnosis: the pass^k’s heterogeneity (Module 5, Proposition 0.4.6).

14.16 Concept check

A distillation’s student has a 5-point higher success rate than the teacher on the held-out set, but the audit finds that 20% of the student’s successes are exploits. What is the true comparison, and what would you do?

Answer. The student’s true success is the reported success minus the exploits’ contribution: if 20% of the successes are exploits, the true success is $0.8\times$ the reported (approximately). The 5-point gain is likely the exploit’s inflation, not a real improvement. The action: (1) filter the exploits from the demonstrations and retrain; (2) re-audit; (3) re-evaluate. The lesson: the distillation’s success is only as good as the audit, and an unaudited distillation is a distillation of unknown validity (Module 5, §5.13.3).

PART D — DATA SOVEREIGNTY AND AIR-GAPPED DEPLOYMENT

14.17 The drivers

14.17.1 The four drivers

Driver 1 — The regulation. The data protection’s laws (the GDPR, the sectoral rules), the data localisation’s requirements, the cross-border transfer’s restrictions. The regulation determines whether the data can leave the jurisdiction, and the model’s inference is a data’s processing.

Driver 2 — The IP. The code, the designs, the trade secrets. The inference’s prompt contains the IP, and sending it to a third-party API is a disclosure to the third party (the API’s terms may permit the training, or the retention). The IP’s protection requires the model to run inside the organisation’s boundary.

Driver 3 — The latency. The API’s network latency, the rate limits, the availability. A local model’s latency is the GPU’s, and the availability is the organisation’s.

Driver 4 — The cost. The API’s per-token price vs the local GPU’s cost. At a high volume, the local model is cheaper (the GPU’s utilisation); at a low volume, the API is cheaper (no capital).

14.17.2 The decision

The sovereignty’s decision is the boundary’s: the data’s sensitivity, the regulation’s requirement, and the organisation’s risk’s tolerance. A high-sensitivity deployment (the personal data, the trade secrets) requires the boundary; a low-sensitivity deployment can use the API.

14.18 The deployment patterns

14.18.1 The three patterns

Pattern 1 — The full air-gap. The model, the harness, the tools, the data, and the observability are all on the organisation’s infrastructure, with no network egress (Module 7, §7.5). The model is open-weight, served locally (Module 13’s stack). The updates are manual (a new checkpoint is vetted and deployed).

Pattern 2 — The VPC. The model is in the cloud provider’s VPC, the data is in the VPC, and the egress is controlled. The model may be the provider’s managed service or a self-hosted open-weight model on the VPC’s GPUs. The provider’s data-processing agreement is the regulatory’s instrument.

Pattern 3 — The hybrid. The sensitive data is on-prem, and the model is in the VPC with a data-processing agreement and the data’s minimisation (the sensitive data is redacted or tokenised before the inference). The hybrid is the compromise when the full air-gap is too expensive and the API is too risky.

14.18.2 The patterns’ comparison

14.19 The open-weight advantage

14.19.1 The argument

The open-weight model is the only option that supports the full air-gap: the weights can be run on the organisation’s GPUs, and the API’s model cannot (the inference is the provider’s). The open-weight model’s licensing (Module 12, §12.7) is the check: a Class 1 or a Class 2 licence permits the local deployment; a Class 3 does not (the non-commercial’s restriction).

14.19.2 The constraints

The open-weight’s local deployment has four constraints.

Constraint 1 — The model’s updates. The API’s model is updated by the provider; the local model is updated by the organisation. The update’s process: the new checkpoint’s vetting (the licence, the format tests, the evaluation), the deployment’s canary (Module 22), the rollback. The update’s cadence is the organisation’s, and the lag is the risk (a vulnerability’s fix is delayed).

Constraint 2 — The licensing. The per-checkpoint rule (Module 12, §12.7.1): the air-gap does not change the licence, and the local deployment must comply with the conditions (the acceptable-use, the attribution, the revenue’s cap).

Constraint 3 — The observability. The trace store is local (Module 11, §11.6), and the analysis’s tooling is the organisation’s. The observability’s cost is the storage’s and the analysis’s, and it is the air-gap’s overhead.

Constraint 4 — The support. The API’s model has the provider’s support; the local model has the community’s (or the vendor’s commercial support). The support’s gap is the operational risk, and the organisation’s team must have the capability (the serving’s, the debugging’s).

14.20 The air-gap’s engineering

14.20.1 The serving

The serving is the Module 13’s stack, on the organisation’s GPUs. The model’s choice is the Module 12’s dossier, with the hardware’s BOM’s fit and the licence’s verdict. The engine’s choice is the Module 13’s, with the air-gap’s constraint (the engine’s dependencies must be installable without the internet, and the container images must be mirrored).

14.20.2 The data’s flow

The data’s flow is the boundary’s: the data enters the harness, the model processes it, and the trace records it. The trace’s storage is local, and the retention’s policy (Module 11, §11.6.2) is the data’s minimisation’s instrument. The egress’s policy (Module 7, §7.5) is the boundary’s enforcement, and the policy’s default-deny is the air-gap’s rule.

14.20.3 The updates

The update’s process is the canary’s (Module 22): the new checkpoint is deployed to a small fraction, the metrics are compared (Module 5), and the full rollout is the canary’s confirmation. The rollback is the previous checkpoint’s restore, and the trace’s version field (Module 11, §11.1.3) is the attribution’s key.

14.21 Concept check

A deployment is fully air-gapped, and the model is a Class 2 licence with a revenue cap. The organisation’s revenue crosses the cap. What happens, and what would you do?

Answer. The licence’s condition is violated: the deployment is no longer permitted under the Class 2 licence, and the air-gap does not change that (the licence is the weights file’s, not the deployment’s). The options: (1) a commercial licence from the model’s provider (the revenue’s cap’s negotiation); (2) a switch to a Class 1 model (the Apache 2.0 or MIT), with the dossier’s re-evaluation; (3) the deployment’s reduction (the revenue’s attribution is the legal’s question). The lesson: the sovereignty’s pattern does not exempt the licence’s conditions, and the revenue’s cap is a monitoring alert (Module 12, §12.8.2), not a one-time check.

PART E — THE 3×3 FACTORIAL

14.22 The design

14.22.1 The factors

The 3×3 factorial crosses three models with three harness configurations.

Definition 14.22.1 (The 3×3 design). The factors:

• The model factor $A$: three models $M_1,M_2,M_3$ (from the Module 12’s families, comparable in scale).
• The harness factor $B$: three configurations $H_1,H_2,H_3$, each tuned for one of the models (the “native” harness of $M_j$).
• The replicates $K$: the tasks (or the seeds) per cell, at least 50 for the Module 5’s power.

The cell $(i,j)$ is the model $M_i$ evaluated with the harness $H_j$. The design is fully crossed: every model is evaluated with every harness, on the same task set.

14.22.2 The hypotheses

• H1 (the interaction). The interaction term $(\alpha\beta)_{ij}$ is significant: the models’ rankings depend on the harness.
• H2 (the native advantage). Each model’s best harness is its native harness: $A_{ii} > A_{ij}$ for $j\neq i$.
• H3 (the harness’s share). The harness’s variance component $\sigma^2_\beta$ is comparable to or larger than the model’s $\sigma^2_\alpha$.
• H4 (the interaction’s dominance). The interaction’s variance component $\sigma^2_{\alpha\beta}$ is the largest structured component (larger than $\sigma^2_\alpha$ and $\sigma^2_\beta$).
• H5 (the adaptation’s cost). The adaptation cost (the off-diagonal’s drop) is larger for the models with the more distinctive formats.

14.23 The variance decomposition

14.23.1 The model

Restating Part 0, §0.4, Definition 0.4.2 and Proposition 0.4.8:

The variance components:

14.23.2 The interpretation

The variance components’ shares:

The shares answer the module’s questions:

• The model’s share $R^2_\alpha$: how much the model’s choice matters.
• The harness’s share $R^2_\beta$: how much the harness’s choice matters.
• The interaction’s share $R^2_{\alpha\beta}$: how much the co-adaptation matters.
• The residual’s share $R^2_{\text{res}}$: how much the task’s variance dominates.

14.23.3 The worked decomposition

The mean squares (a 3×3 design, $K=100$):

• $MS_\alpha = 2.40$
• $MS_\beta = 1.80$
• $MS_{\alpha\beta} = 0.90$
• $MS_{\text{err}} = 0.20$

The components:

The total: $0.005+0.003+0.007+0.20 = 0.215$.

The shares:

• The model: $0.005/0.215 = 2.33$
• The harness: $0.003/0.215 = 1.40$
• The interaction: $0.007/0.215 = 3.26$
• The residual: $0.20/0.215 = 93.02$

The interpretation. The interaction is the largest structured component: the co-adaptation matters more than either the model’s main effect or the harness’s main effect. The finding is the module’s thesis: a deployment that selects a model and a harness independently (without the co-adaptation) leaves the interaction’s variance on the table. The residual dominates (the task’s variance), which is why the paired design and the large $K$ are essential (Module 5, §5.4).

14.23.4 The interaction’s significance

The interaction’s F-test:

with the degrees of freedom $(I-1)(J-1)=4$ and $IJ(K-1)=891$. The p-value is $<0.01$ (the F-distribution’s tail), so the interaction is significant: the co-adaptation is real, not the noise.

14.24 The adaptation’s cost, measured

14.24.1 The definition, restated

The adaptation cost (Definition 14.4.2) is $c_i=\max_j A_{ij}-A_{i,j^*(i)}$. The matrix’s diagonal is the native harness, and the off-diagonal’s drop is the cost.

14.24.2 The worked matrix, with the decomposition

The matrix of §14.4.2:

The model’s main effects: $M_2$ (0.670) > $M_1$ (0.657) > $M_3$ (0.637). The model’s ranking: $M_2 > M_1 > M_3$.

The harness’s main effects: $H_2$ (0.673) > $H_3$ (0.653) > $H_1$ (0.637). The harness’s ranking: $H_2 > H_3 > H_1$.

The interaction: the cell’s deviation from the additive prediction. For $(M_1,H_1)$: the prediction is $0.654 + (0.657-0.654) + (0.637-0.654) = 0.640$; the observed is 0.72; the deviation is $+0.08$. The native cells are all positive (the co-adaptation’s advantage); the foreign cells are all negative.

The lesson: the main effects’ ranking ($M_2$ best, $H_2$ best) is misleading without the interaction: $M_1$ with $H_1$ (0.72) beats $M_2$ with $H_2$ (0.74? no — 0.74 is the best). Actually $M_2,H_2$ is the best cell (0.74), and the main effects’ ranking happens to match. But $M_1,H_1$ (0.72) beats $M_2$ with $H_1$ (0.61) — the ranking flips when the harness changes: with $H_1$, $M_1 > M_2$ (0.72 vs 0.61); with $H_2$, $M_2 > M_1$ (0.74 vs 0.65). The ranking’s flip is the interaction’s practical content, and it is the Module 5, §5.8’s “harness-confounded comparison” made concrete.

14.25 The reporting

14.25.1 The report’s structure

The lab’s report has five sections.

1. The design. The models, the harnesses, the tasks, the $K$, the power’s calculation.
2. The matrix. The $3\times3$ table of the pair’s performance, with the confidence intervals.
3. The decomposition. The variance components, the shares, the F-test.
4. The interaction’s analysis. The ranking’s flips, the native advantage, the adaptation’s costs.
5. The decision memo. The (model, harness) pair for the deployment, with the revision condition.

14.25.2 The pre-registration

The hypotheses (H1–H5) are stated before the data, the analysis plan (the ANOVA, the decomposition, the paired bootstrap for the cell’s CIs) is fixed, and the deviations are logged (Module 5, §5.10).

14.26 Concept check

The interaction’s share is 3.26% and the residual’s is 93%. Is the interaction important, given that the residual dominates?

Answer. Yes. The residual’s dominance means the task-to-task variance is the largest source, which is why the paired design and the large $K$ are needed to detect any effect. The interaction’s 3.26% is the structured variance’s largest component, and it is the decision-relevant one: the model’s and the harness’s choices are the deployment’s controllable variables, and the interaction says they must be chosen together. A deployment that ignores the interaction and selects the model and the harness independently loses the interaction’s variance, which is larger than either main effect’s. The 3.26% is small in absolute terms, and it is the largest actionable term.

MODULE 14b — THE CO-EVOLUTION RESEARCH AGENDA

14.27 The four questions

14.27.1 The questions

Question 1 — Convergence or divergence? Are the harnesses converging on a canonical architecture, or diverging into task-specific families? The eleven-system study reports convergence (Module 6, §6.8), and the classification (Module 6, §6.13) predicts that the compensating components should diverge (the teams shed different scaffolds at different rates) while the irreducible components converge (they are permanent).

Question 2 — General knowledge or benchmark overfitting? Does the harness’s evolution encode general engineering knowledge (transferable across task families), or does it overfit to the benchmark it was tuned on? The test: the improvement’s transfer to a held-out family (Module 5, §5.11.2).

Question 3 — The minimal safe harness? As the capability increases, what is the minimal harness that preserves the safety properties (the permissions, the audit, the verification) while shedding the compensating scaffolds?

Question 4 — The automation of the classification? Can the compensating/irreducible classification (Module 6, §6.13) be automated, so that the removal’s maintenance is cheap enough to be routine?

14.27.2 The evidence’s state

Each question’s current evidence:

• Q1: the eleven-system study’s adoption counts (tier S), and the convergence’s three readings (the correct design, the fashion, the local optimum) are unresolved (Module 6, §6.8.2).
• Q2: the AHE’s Terminal-Bench result (tier P), which does not report the held-out transfer.
• Q3: no direct evidence; the classification is a hypothesis.
• Q4: no direct evidence; the classification is manual.

The four questions are the curriculum’s open research agenda, and Module 14b’s lab is the design of the experiment for one of them.

14.28 The longitudinal experiment

14.28.1 The design

The experiment for Q2 (the general/overfit question) and Q3 (the minimal safe harness) is a longitudinal design: the same component’s effect is measured across two capability tiers (or two model generations).

Definition 14.28.1 (The longitudinal design).

1. Select a component from the Module 6’s classification, with its predicted class (compensating or irreducible).
2. Measure the component’s effect on the current model (the tier 1), with the Module 5’s design.
3. Measure the component’s effect on the stronger model (the tier 2), with the same design.
4. Compare the effects: the compensating component’s effect should shrink (the stronger model needs less scaffolding); the irreducible component’s effect should persist (the deployment’s requirement is unchanged).
5. Test the prediction with the pre-registered analysis.

14.28.2 The prediction

The classification predicts:

• A compensating component’s effect (e.g., the compaction threshold, the tool schema’s simplification) shrinks on the stronger model: the stronger model’s effective context is larger, and its tool-selection ability is better, so the scaffold’s benefit is smaller.
• An irreducible component’s effect (e.g., the permission policy, the verifier) persists: the deployment’s requirement is the same, so the component’s benefit is the same.

Definition 14.28.2 (The prediction’s form). For a compensating component $c$, the prediction is

for a stated shrinkage $\alpha<1$ (e.g., $\alpha=0.5$). For an irreducible component, the prediction is

for a stated tolerance $\epsilon$.

14.28.3 The pre-registration

The experiment’s pre-registration (Module 5, §5.10):

1. The component and its predicted class.
2. The metric and the effect’s definition.
3. The two tiers (the models, the capability’s difference).
4. The power’s calculation for each tier’s effect.
5. The prediction (the shrinkage $\alpha$, the tolerance $\epsilon$).
6. The analysis plan (the paired comparison, the interaction’s test).
7. The deviation’s log.

14.29 The experiment’s threats

14.29.1 The four threats

Threat 1 — The tiers’ confounding. The two models may differ in more than the capability (the format, the training’s data, the licence). The effect’s difference may be the format’s, not the capability’s. Control: the format tests (Module 13) for both, and the format’s effect’s isolation.

Threat 2 — The task set’s drift. The two tiers may be evaluated on different task sets (the stronger model’s tasks are harder). Control: the same task set, or the stratified sampling with the matched tasks.

Threat 3 — The effect’s smallness. The compensating component’s effect may be small even on the tier 1, so the shrinkage is undetectable. Control: the power’s calculation, and the larger $K$.

Threat 4 — The model’s non-stationarity. The “tier 2” may be a different checkpoint of the same model, not a capability’s tier. Control: the capability’s measurement (the benchmark’s, the task’s), not the model’s version.

14.29.2 The threats’ mitigation

The threats are the experiment’s validity’s conditions. The design must state each threat and its mitigation, and the report must discuss the residual threats (the ones the design cannot control).

14.30 The worked experiment’s design

14.30.1 The component

The component: the compaction threshold (Module 3, §3.4’s truncation’s threshold). The predicted class: compensating (a stronger model’s effective context is larger, so the threshold’s importance shrinks).

14.30.2 The design

• The tiers: the tier 1 (a mid-tier model) and the tier 2 (a frontier model), same family if possible.
• The conditions: the compaction threshold at 8k vs 32k, on both tiers.
• The metric: the success rate on a long-horizon task set (the trajectories longer than 8k).
• The prediction: the threshold’s effect on the tier 1 is $\Delta_1$, and the effect on the tier 2 is $\Delta_2<\alpha\Delta_1$ with $\alpha=0.5$.

14.30.3 The analysis

The interaction’s test: the model (tier) × the condition (the threshold) factorial (a 2×2), with the interaction’s term as the shrinkage’s measure. The significance’s test is the same as the Module 5, §5.9’s.

14.30.4 The interpretation

If the shrinkage holds (the interaction is significant, and $\Delta_2<\alpha\Delta_1$), the compensating hypothesis is supported, and the removal condition (Module 6, §6.14.3) is closer: the threshold’s benefit is shrinking as the capability grows, and the threshold is a candidate for removal on the tier 2.

If the shrinkage does not hold (the effect persists), the classification is wrong or the tier 2’s effective context is not larger than the tier 1’s (the capability’s improvement is in the reasoning, not the context). The finding is a correction to the classification, and it is the experiment’s value whether the prediction holds or not.

14.31 Concept check

Why is the longitudinal experiment’s prediction a test of the Module 6 classification, not just of the component?

Answer. Because the classification (Module 6, §6.13) is a counterfactual claim: a compensating component “would be unnecessary with a perfect model”, and an irreducible component “is required for every model”. The longitudinal experiment is the empirical test of the counterfactual: the tier 2 is a step toward the perfect model, and the component’s effect’s shrinkage (or persistence) is the evidence. A prediction that holds is the classification’s support; a prediction that fails is the classification’s correction. The experiment’s value is the classification’s falsifiability, which is what makes it a scientific claim rather than a taxonomy.

LAB 14 — THE 3×3 FACTORIAL (PART III CAPSTONE)

14.32.1 Task specification

Select three open-weight models (from the Module 12’s families) and three harness configurations (each tuned for one of the models). Cross them in a 3×3 factorial, on a fixed task set with a deterministic verifier, with at least 50 tasks (or the power’s calculation’s minimum).

14.32.2 The harness configurations

• $H_1$: the harness tuned for $M_1$ (the tool schema’s format, the reasoning’s parser, the compaction’s threshold, the retry’s policy, all matching $M_1$‘s conventions).
• $H_2$: the harness tuned for $M_2$.
• $H_3$: the harness tuned for $M_3$.

The three configurations differ only in the four adaptation surfaces (Module 14, §14.2): the loop, the tools’ set, the verifier, and the task set are the same.

14.32.3 The design

• The cells: $3\times3=9$, each with $\ge50$ tasks.
• The controls: the same model’s version, the same harness’s loop, the same tools, the same verifier, the same task set, the same budget.
• The seeds: the model’s sampling’s parameters, the same across the cells (or the seeds’ variation reported).

14.32.4 The metrics

14.32.5 The hypotheses

The five hypotheses of §14.22.2 (H1–H5), pre-registered.

14.32.6 The acceptance criteria

14.32.7 What good looks like

A good lab report has five properties.

The matrix is the finding. The 3×3 table is the report’s centerpiece, and the interaction’s reading (the diagonal’s advantage, the ranking’s flips) is the analysis.

The decomposition is the quantitative answer. The variance components’ shares are the answer to “is the harness load-bearing?” and “is the co-adaptation real?” — with the numbers, not the adjectives.

The interaction is the decision’s content. The report does not say “the models differ”; it says “the models’ ranking depends on the harness, and the interaction is the largest structured component”. That is the decision-relevant finding.

The parse-failure rate explains the cells. A cell’s low success is often the parse-failure rate’s, not the model’s capability’s. The report decomposes the cell’s success into the parse’s and the capability’s contributions.

The decision is the pair’s. The report recommends a (model, harness) pair, with the evidence and the revision condition. A report that recommends a model or a harness separately has missed the module’s thesis.

LAB 14b — THE LONGITUDINAL EXPERIMENT’S DESIGN

14.33.1 Task specification

Take one component from the Module 6’s classification and design a longitudinal experiment to test its predicted class.

14.33.2 The design’s fields

1. The component and its predicted class (compensating or irreducible), with the Module 6’s counterfactual’s reasoning.
2. The two tiers (the models, the capability’s difference, the format’s compatibility).
3. The conditions (the component’s presence vs absence, or the parameter’s two values).
4. The metric and the effect’s definition.
5. The power’s calculation for each tier.
6. The prediction (the shrinkage $\alpha$ or the tolerance $\epsilon$).
7. The analysis plan (the model × the condition factorial, the interaction’s test, the paired comparison).
8. The threats to validity (§14.29.1) and the mitigations.
9. The deviation’s log template.
10. The decision rule: what the result would mean for the component’s classification and its removal.

14.33.3 The acceptance criteria

14.33.4 What good looks like

A good lab report has four properties.

The component is specific. “The compaction threshold” is a component; “the context management” is not. The specificity is what makes the experiment runnable.

The prediction is falsifiable. The shrinkage $\alpha$ is a number, and the experiment can show it wrong. A prediction that “the effect may shrink” is not falsifiable, and it is not a prediction.

The threats are named. The tiers’ confounding (the format, the training’s data) is the experiment’s biggest threat, and a design that does not name it has not thought about the validity.

The decision rule is explicit. The result’s meaning for the classification is stated before the data: “if the shrinkage holds, the classification is supported, and the removal condition is closer; if not, the classification is corrected.” The explicitness is the pre-registration’s discipline.

MODULE 14 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E14.1. A co-adaptation matrix shows $M_1$: 0.72 (its harness) / 0.65 (foreign); $M_2$: 0.61 (foreign) / 0.74 (its harness). The deployment uses $H_1$. What is $M_2$‘s adaptation cost, and should the deployment adapt?

S14.1. $M_2$’s adaptation cost is $0.74-0.61=0.13$ (the gap between its native harness and $H_1$). The adaptation’s benefit is $0.13V$; the cost is $T_{\text{adapt}}$. The decision depends on $V$ and $T_{\text{adapt}}$: if $V=1$ and $T_{\text{adapt}}=0.06$, the benefit (0.13) exceeds the cost (0.06), and the deployment should adapt. If $T_{\text{adapt}}=0.20$, it should not. The lesson: the adaptation cost is the decision’s input, and the 13-point gap is comparable to a model upgrade (Module 12’s selection).

E14.2. A tiered architecture has $p_p=0.90$, $p_e=0.95$, and $n=10$ steps. What is the success before the verifier’s correction? If the verifier’s correction raises the failure’s detection to 0.8, what is the corrected success (approximately)?

S14.2. Before: $0.90\times0.95^{10}=0.90\times0.599=0.539$. The verifier’s correction: the executor’s failures (the 1-0.599 = 0.401 of the runs) are detected at 0.8, and the planner fixes them with its own success. The corrected success is approximately $0.539 + 0.401\times0.8\times0.9=0.539+0.289=0.828$. The lesson: the verifier’s correction is the tiered architecture’s reliability’s recovery, and without it the executor’s compounding is the binding constraint.

E14.3. A distillation’s teacher has a 10% exploit rate, and the filter removes the exploits. The student’s reported success is 0.75, and its audit finds a 2% exploit rate. What is the student’s true success, and is the distillation worth it?

S14.3. The true success is approximately $0.75\times(1-0.02)=0.735$ (the exploits’ 2% of the successes are not real). The comparison is the teacher’s true success (0.70 × (1-0.10) = 0.63, if the teacher’s reported is 0.70) vs the student’s 0.735 — the student is better. But the 2% exploit rate is non-zero, and the filter did not eliminate it: the student learned a residual exploit. The action: re-audit, strengthen the filter (the teacher’s demonstrations may contain a subtler exploit), and re-evaluate. The lesson: the distillation’s true success is the audited success, and an unaudited comparison is not a comparison.

E14.4. A 3×3 factorial has $MS_\alpha=1.20$, $MS_\beta=0.60$, $MS_{\alpha\beta}=0.30$, $MS_{\text{err}}=0.10$, $K=100$. Compute the variance components and the interaction’s share.

S14.4. $\hat\sigma^2_\alpha=(1.20-0.30)/300=0.003$; $\hat\sigma^2_\beta=(0.60-0.30)/300=0.001$; $\hat\sigma^2_{\alpha\beta}=(0.30-0.10)/100=0.002$; $\hat\sigma^2=0.10$. Total $=0.106$. The model 2.83%, the harness 0.94%, the interaction 1.89%, the residual 94.34%. The interaction is twice the harness’s main effect, and the model’s main effect is the largest structured term. The lesson: the interaction’s size relative to the main effects is the co-adaptation’s measure, and the ratio (1.89/0.94 = 2.0) says the co-adaptation matters more than the harness’s choice.

E14.5. An air-gapped deployment uses a Class 2 model with a revenue cap, and the organisation’s revenue approaches the cap. What is the risk, and what would you do?

S14.5. The risk is the licence’s violation: the revenue’s cross of the cap makes the deployment non-permitted (Module 12, §12.7.2). The action: (1) monitor the revenue against the cap (the alert, not the one-time check); (2) prepare the migration (the Class 1 model’s dossier, Module 12); (3) negotiate the commercial licence (the provider’s terms). The lesson: the sovereignty’s pattern does not exempt the licence, and the revenue’s cap is an operational constraint, not a deployment’s footnote.

E14.6. A longitudinal experiment’s prediction is that the compaction threshold’s effect shrinks by 50% on the tier 2. The tier 1’s effect is 12 points, and the tier 2’s is 8 points. Is the prediction confirmed?

S14.6. The predicted effect is $0.5\times12=6$ points; the observed is 8 points, which is above the predicted shrinkage (the effect shrank by 33%, not 50%). The prediction’s shrinkage is not confirmed at the stated $\alpha$. The finding: the compaction threshold is partly compensating (the effect shrank) but not as much as predicted (the tier 2’s effective context is not as large as the prediction assumed). The classification is refined: the threshold is compensating, and its removal condition (the effective context’s growth) is further away than the prediction’s. The lesson: the prediction’s failure is a refinement, not a refutation, and the magnitude’s analysis is the finding.

E14.7. A routing policy beats the best single model by 4 points on the held-out tasks. Is the routing worth it?

S14.7. It depends on the routing’s cost: the router’s inference’s cost, the misrouting’s cost, and the engineering’s cost. If the router’s cost is 0.005 per task and the value of a success is 1.0, the 4-point gain is 0.04 per task, which exceeds the cost (0.005): the routing is worth it. If the router’s cost is 0.06, the routing is not worth it (the gain is below the cost). The lesson: the routing’s value is the net of the gain and the cost, and the 4-point gain is not the decision’s answer without the cost.

E14.8. A fleet’s tier assignment gives the strong model the planner’s role and the small model the executor’s role. The small model’s parse-failure rate is 15%. What is the executor’s cost per correct step, and what would you do?

S14.8. The cost per correct step is $C_e/(1-p_{\text{parse}})=C_e/0.85=1.18\,C_e$, an 18% premium. If the mid model’s parse rate is 2% and its cost is $2\,C_e$, the mid model’s cost per correct step is $2C_e/0.98=2.04\,C_e$, which is higher than the small model’s $1.18\,C_e$ — so the small model is cheaper per correct step, despite its parse rate. But the reliability’s requirement (the compounding, §14.7.3) may bind: at $p_e=0.85$ and $n=20$, the success is $0.85^{20}=0.039$, which is unusable. The lesson: the executor’s choice is the cost per correct step and the compounding’s requirement, and the parse rate’s premium is the second, not the first.

EXIT TEST FOR MODULE 14

You are ready for Module 15 when you can, without notes:

1. Define co-adaptation and name the four sources.
2. State the four adaptation surfaces and the order of the adaptation.
3. State the foreign-harness failure taxonomy and the diagnosis’s procedure.
4. Define the co-adaptation matrix and the adaptation cost.
5. Apply the adaptation’s decision and the surfaces’ priority.
6. Describe the tiered architecture, its cost model, and its success model.
7. Explain the executor’s reliability’s compounding and the verification-gated escalation.
8. State the routing’s criterion and the cascade’s cost and accuracy.
9. Assign the roles by the cost per correct task, subject to the requirement.
10. Describe the distillation pipeline, what distils, and the reward-hacking risk.
11. State the audit’s procedure and the distillation’s failure modes.
12. State the sovereignty’s four drivers and the three patterns.
13. Explain the open-weight advantage and the local deployment’s constraints.
14. Design a 3×3 factorial, decompose the variance, and interpret the interaction.
15. State the five hypotheses and the adaptation cost’s measurement.
16. State the four co-evolution research questions.
17. Design a longitudinal experiment with a pre-registered prediction.
18. State the four threats to validity and their mitigations.

End of Module 14. Part III is complete. Say “next” to receive Module 15 in full (Part IV: Training and Adaptation of Agents begins).