Lesson 4.3.1 — Systematic Experimentation and Ablations

Week 38

Prerequisite lessons: 4.2.4 (safety and recovery), 4.2.3 (reasoning layer), 4.2.2 (policy layer), 4.2.1 (perception stack integration), 4.1.3 (evaluation design and held-out test suites), 3.1.4 (VLA fine-tuning).

Prior results used in this lesson. From 4.2.4: the safety case’s residuals and the fault-injection suite’s four metrics, restated in §L.D.3. From 4.2.3: the six strata and the deferral rate, restated in §L.D.1. From 4.2.2: the per-tier attribution and its distribution, restated in §L.D.2. From 4.2.1: the perception evidence table and the 95th percentile’s age, restated in §L.C.3. From 4.1.3: the 400-episode suite, its strata, and the held-out discipline, restated in §L.A.3. From 3.1.4: the decision conjunction and the four-ablation protocol of §L.E.3, restated in §L.B.1. Each is restated at the point of use.

Hands forward. The experiment registry, the factorial ablation’s design, the statistical power’s arithmetic, and the confound’s detection. Lesson 4.3.2 (diagnosing sim-to-real failures) uses the registry as its measurement’s protocol. Lesson 4.3.3 (long-horizon robustness) uses the power’s arithmetic as its window’s sizing. Module 4.4 (the master’s defence) uses the registry as the evidence index.

The running case in one sentence. The arm, gripper, eye-in-hand camera, table, cube, bin, divider, and three distractors are unchanged, and the seven subsystems are assembled into the full-stack agent; this lesson asks how to design, execute, and analyze an experiment that tells you which subsystem contributes to the acceptance’s metric, and by how much.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) state the experiment’s four elements and the registry’s six fields; (2) state causal inference’s three conditions and give a failure of each; (3) distinguish the one-at-a-time ablation from the factorial’s and state each one’s blindness; (4) compute a $2^{k}$ factorial’s conditions and a fractional design’s resolution from its generators; (5) compute the power’s sample size for a given effect’s size and state the seed variance’s role; (6) apply the Bonferroni and Holm corrections to a multi-ablation’s set; (7) state the interaction’s interpretation and the main effect’s; (8) write the logging contract and its four channels; (9) diagnose a confound from its pattern; (10) assemble the running case’s ablation registry with its arithmetic and its analysis’s plan.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 3.1.4 §L.E.3 the four-ablation protocol isolated the language path, the visual path, the motor precision, and the latency by holding the other three fixed. What is the protocol’s structural assumption, and what happens if it fails?

Answer. The assumption is the main effect’s separability — each ablation’s effect is measured with the others fixed, and the effects are additive. If the effects interact, the measurement is the marginal effect at one configuration rather than the general one, and the interaction’s term is unmeasured. Used in §L.B.1 to motivate the factorial’s design.

Q2. In 4.2.2 §L.D.1 the per-tier attribution was the earliest tier whose error exceeded its tolerance. What is the attribution’s relationship to the ablation’s, and why are they different?

Answer. The attribution localizes one failure’s origin from a single trace; the ablation measures a subsystem’s contribution across many episodes. The attribution’s output is the failure’s tier, and the ablation’s output is that tier’s effect’s magnitude — two different questions about the same subsystems. Used in §L.D.2 to design the ablation’s logging.

Q3. In 4.1.3 §L.C.3 the test set’s size was the maximum over the metrics’ decision resolutions. What does the ablation’s design add to that arithmetic?

Answer. The ablation multiplies the test set by the conditions’ count, and the power’s arithmetic replaces the resolution’s. The 3-point resolution’s 384 episodes for a two-group comparison becomes a per-condition requirement that the power analysis’s effect’s size determines. Used in §L.C.1 to compute the ablation’s episodes.

Q4. In 4.2.4 §L.D.2 the fault-injection suite’s four metrics included the detection rate and the recovery success rate. Which of those is an ablation’s metric rather than a fault injection’s?

Answer. Neither, strictly; the fault injection measures the architecture’s response to a specific fault, and the ablation measures a subsystem’s contribution to the nominal metric. The two share the arithmetic, and the ablation’s design borrows the injection’s per-condition structure. Used in §L.D.3 to distinguish the two evidence roles.

Part A — The experiment’s logic

§L.A.1 — The four elements

An experiment is a procedure that produces evidence about a causal claim. Its four elements are the claim’s structure.

The hypothesis. A statement of the form “subsystem $X$‘s presence raises the acceptance’s metric by at least $\delta$”. The hypothesis’s specificity is the design’s: a hypothesis that is not specific cannot be falsified, and an unfalsifiable hypothesis is not an experiment’s.

The treatment. The condition that the hypothesis varies — the subsystem’s presence or absence. The treatment’s operationalization is concrete: the subsystem’s presence is the assembled system, and its absence is the system with the subsystem replaced by a fixed baseline, not with the subsystem removed and nothing put in its place.

The control. The condition the hypothesis compares against — the baseline system. The control’s validity is the comparison’s: control and treatment must differ only in the treatment’s variable, and any other difference is a confound.

The randomization. The assignment of episodes to treatments. The randomization’s purpose is the confounds’ balance: the episodes’ uncontrolled differences — object poses, operators, batches — are distributed randomly across the treatments, so their effects cancel in the comparison.

The four elements’ failure modes. A vague hypothesis is fishing; a treatment that removes the subsystem and its retraining is the confound; a control that differs in the camera’s calibration is spurious; a randomization that is not random is systematic.

Self-explanation 1. Why is “remove the subsystem” a confound rather than a treatment?

Answer. Because the removal changes two things at once: the subsystem’s contribution and the remaining subsystems’ distribution. The others were trained with the subsystem present, and their behaviour differs without it. The two changes are inseparable in a naive removal, and the treatment’s replacement — a fixed baseline, with the others retrained or held — is what isolates the contribution.

§L.A.2 — The registry

The experiment registry is the pre-registration’s record, and its six fields are the hypothesis’s, the treatment’s, the control’s, the randomization’s, the metric’s, and the analysis’s.

The hypothesis’s field. The claim’s statement with its effect-size threshold $\delta$. For the running case, “the world model’s planner, compared to the classical planner, raises the acceptance’s success rate by at least 5 points”.

The treatment’s field. The treatment’s operationalization with its replacement’s description. “The world model’s planner is replaced by the 2.3.2 classical planner; perception, reasoning, control, and safety are the assembled versions”.

The control’s field. The baseline system’s description. “The assembled full-stack agent with the 3.2.3 planner”.

The randomization’s field. The episodes’ assignment’s procedure. “The 400-episode suite’s episodes are paired — each episode runs in both conditions — and the order within a pair is randomized by seed 20260601”.

The metric’s field. The acceptance metric’s definition. “The 4.1.1 §L.C.1 success rate, with its lower 95% bound”.

The analysis’s field. The statistical test with its $\alpha$, its power, and its multiple-comparison correction. “The two-proportion test at $\alpha = 0.05$, power 0.80, Holm’s correction across the seven ablations, and the decision on the CI’s lower bound against a 3-point threshold”.

The registry’s purpose is the falsifiability’s record: a hypothesis stated before the data is the a priori test’s; a hypothesis stated after is the post hoc test’s, and the post hoc test’s evidence is weaker because the space of hypotheses examined is not known.

The registry’s discipline: written before the first episode runs, its fields not edited after the data. An edit is an amendment, with the reason and the date documented; the amendment is the analysis’s transparency record.

§L.A.3 — Causal inference’s three conditions

A causal claim’s validity requires three conditions, each with a failure mode.

Temporal precedence. The cause precedes the effect; the treatment’s variation must precede the metric’s measurement. A failure: the subsystem’s removal is measured on the episodes the subsystem trained on, so the metric is the training record’s, not the effect’s.

Mechanism. The cause produces the effect through a specified pathway; the treatment’s variation must change the mechanism’s variables, and the effect must be that mechanism’s consequence. A failure: the ablation “removes” the world model but the reasoning layer re-queries to compensate, so the metric’s change is the compensation’s, not the world model’s.

Counterfactual. The effect is the difference between the treatment’s outcome and the same episodes’ outcome under the control. A failure: treatment and control run on different episodes, so the difference is the episodes’ distribution’s, not the treatment’s. The 3.1.3 §L.D.2 paired design is the counterfactual’s mechanism.

The three conditions together are the causal claim’s validity. The registry’s six fields are their operationalization: the hypothesis’s field is the temporal condition’s, the treatment’s is the mechanism’s, and the randomization’s is the counterfactual’s.

Concept check. An ablation’s metric falls 4 points when the world model is replaced, and the registry’s mechanism field reads “the world model’s planner is more accurate”. Which condition is under-specified, and what is the fix?

Answer. The mechanism’s. “More accurate” is not a specified pathway, and the fall could be the reasoning layer’s compensation rather than the planner’s accuracy. The fix is a specified pathway: “the world model planner’s subgoals fall within the 4.2.2 task tier’s tolerance on 92% of episodes, against the classical planner’s 85%” — the mechanism’s operational measure. [displaces M5]

Worked-example ladder.

WE-L.A.a (fully worked). A registry hypothesis reads “residual RL raises the success rate by 5 points”. The treatment is the residual’s removal, the control is the assembled system, and the randomization is the 400-episode suite’s pairing. Compute the analysis’s effect-size threshold and state whether the hypothesis is falsifiable at the suite’s resolution.

Solution. The suite’s 400-episode two-group comparison has variance term $v = p_1(1-p_1) + p_2(1-p_2) = 0.92 \times 0.08 + 0.89 \times 0.11 = 0.1715$, so the standard error is $\sqrt{0.1715/400} = 0.0207$. The minimum detectable effect at 80% power and two-sided 5% is $(1.96 + 0.84) \times 0.0207 = 0.058$ — 5.8 points. The hypothesis’s 5-point threshold is below the 5.8-point floor, so the hypothesis is not falsifiable at the suite’s resolution. The suite must rise to about 538 episodes per condition for a 5-point effect, or the threshold must rise to 6 points.

WE-L.A.b (completion). The suite is expanded to 800 episodes. Recompute the detectable effect and decide.

Completion. The standard error is $\sqrt{0.1715/800} = 0.0146$; the detectable effect is $2.80 \times 0.0146 = 0.041$, or 4.1 points. The hypothesis’s 5 points now exceed the 4.1-point floor, so the hypothesis is falsifiable.

WE-L.A.c (faded). Write the registry’s mechanism field for the world model’s ablation with its operational measure.

Faded step. “The world model planner’s subgoal pose error is 1.2 cm against the classical planner’s 2.1 cm on the held-out episodes, and the task tier’s tolerance is 1 cm.”

Completion. The mechanism’s operational measure is that error comparison at the task tier’s tolerance; the mechanism’s pathway is the error’s propagation to the acceptance metric.

WE-L.A.d (contrastive near-miss). A team’s ablation randomization reads “episodes are run in the order they were collected”. Treatment and control therefore run on different episodes, and the metric’s difference is the episodes’ distribution’s. Why it fails: the counterfactual condition is violated, and the temporal drift — the batches’ environmental conditions — is the confound. The fix: the paired design, each episode in both conditions, with the pair’s order randomized.

Comparative table: the registry’s six fields.

Design principle P57.1. Write the registry before the data and specify the mechanism’s pathway; the six fields are causal inference’s operationalization, and the pairing is the counterfactual’s.

Consolidation checkpoint 1

(a) Three capabilities. You can state the experiment’s four elements and their failure modes. You can write the registry’s six fields. You can state causal inference’s three conditions and give a failure of each.

(b) Transfer probe. The registry’s effect-size is 2 points and the suite’s 400-episode detectable effect is 5.8 points. What is the diagnosis and the fix? Answer: the hypothesis is not falsifiable at this resolution. From $n = 2.8^2 \times 0.1715 / 0.02^2 = 3{,}361$, the suite would have to reach 3,361 episodes per condition for a 2-point effect — a strong argument for raising $\delta$ instead.

(c) Remediation rule. If you cannot state why a naive removal is a confound, re-read §L.A.1 before Part B.

Part B — The ablation’s designs

§L.B.1 — One-at-a-time and its blindness

The one-at-a-time (OAT) ablation removes each subsystem from the assembled system, one at a time, and measures the metric’s change. Its structure is $k$ ablations for $k$ subsystems, so $k$ conditions beyond the control’s.

The OAT’s strength is the conditions’ count — 8 for the running case’s 7 subsystems plus the control — and the interpretation’s simplicity: each subsystem’s effect is measured with the others present.

The OAT’s blindness is the interaction’s. Its measurement of subsystem $X$‘s effect is the marginal effect at the configuration “all other subsystems present”. If the effect depends on the others’ state — the world model contributes more when perception’s pose is noisy, less when it is precise — the measurement is one point on the interaction’s surface, not the general effect.

The blindness’s consequence. The OAT cannot answer the composition’s question: “is the composition’s benefit greater than the sum of its parts?” That is the interaction’s question, and the interaction requires the factorial’s design. The 3.1.4 §L.E.3 four-ablation protocol was an OAT, and its findings are therefore marginal effects at the assembled configuration.

The arithmetic of the blindness, which the lab makes concrete. Suppose the truth is that perception’s main effect is 6 points and its interaction with the world model is −2 points. The OAT reads perception’s effect as 4 points when the world model is present and 6 points when it is absent — and it never reads 8. A team that reports “the OAT says perception is worth 6 points” has reported one row of a two-row table and called it the effect.

The OAT’s role. The OAT is the screening design: $k$ conditions identify the subsystems whose effects are large enough to be worth the factorial’s cost, and the factorial is the confirmation design. The OAT’s screening threshold is the power’s minimum detectable effect, and subsystems below it are not worth the factorial’s.

§L.B.2 — The factorial’s design

The factorial design crosses the subsystems’ presences and absences, producing $2^{k}$ conditions for $k$ subsystems. Its strength is the interaction’s measurement: each subsystem’s effect is measured at every combination of the others’ states, so the interaction term is the effect’s dependence on the others.

The $2^{k}$‘s arithmetic. For the running case’s 7 subsystems, $2^{7} = 128$ conditions. At 400 episodes per condition and 25 s per episode, the cost is $128 \times 400 \times 25 = 1{,}280{,}000$ s = 355 hours. The 4.1.2 §L.E.2 budget of 3 hours is exceeded by two orders of magnitude.

That infeasibility is why the fractional design exists.

§L.B.3 — The fractional’s design

The fractional factorial design runs a selected subset of the $2^{k}$ conditions, chosen so the main effects and the selected interactions stay estimable. Its structure is the design’s resolution, and the resolution is the aliasing’s structure.

Resolution III estimates the main effects aliased with two-factor interactions. It is the screening design. Resolution IV estimates the main effects clear of all two-factor interactions, with the two-factor interactions aliased among themselves. It is the main effect confirmation design. Resolution V estimates the main effects and the two-factor interactions clear of each other. It is the interaction confirmation design.

Reading a fractional design’s resolution from its generators, which is where a plausible-looking design most often goes wrong. The resolution is the length of the shortest word in the defining relation, and the defining relation is generated by the generators. Take the running case: 7 factors and 3 generators gives $2^{7-3} = 16$ conditions. A design whose generators are written as

has defining relation $I = ABCE = ABFG = ACDG = ADEF = BCDF = BDEG = CEFG$ — seven words, the shortest of length 4. So this is resolution IV, as advertised, and each main effect is aliased with four three-factor interactions, never with a two-factor interaction. For instance $A$ aliases $BCE, BFG, CDG, DEF$.

Now take the generator set $D = ABC$, $E = ABD$, $F = ACD$, $G = BCD$ — the same seven factors, four generators this time, so $2^{7-4} = 8$ conditions, not 16. Its defining relation contains the word $AG$, of length 2. The design is not resolution IV at all; its main effects are aliased with two-factor interactions and with each other. The lesson’s earlier draft shipped exactly this generator set, labelled it $2^{7-3}$, called it 16 conditions, and asserted that its main effects were clear of the two-factor interactions. Every one of those four claims is wrong, and the mistake is invisible unless the defining relation is actually built from the generators.

The running case’s design, corrected. Seven subsystems need $2^{7-3} = 16$ conditions at resolution IV, with the generators above. The cost is $16 \times 400 \times 25 = 160{,}000$ s = 44.4 hours — within the budget’s week. Resolution IV estimates the main effects clear and the two-factor interactions aliased, so the interaction’s existence is testable and the specific pair’s is not.

The fractional’s analysis. The main effects are the differences of the conditions’ averages at each factor’s two levels; the interactions are the analogous differences across two factors’ levels. The analysis is the $2^{7-3}$‘s ANOVA, and the 4.2.2 §L.D.1 per-tier attribution is the ANOVA’s post-hoc attribution — the residual variance’s decomposition.

§L.B.4 — The ablation’s logic

The ablation’s interpretation is the removal’s meaning, and the meaning is the replacement’s.

The replacement’s semantics. “Remove the world model’s planner” is ambiguous: it can mean the planner is replaced by a random policy, by the classical planner, or by holding the last plan. Each is a different treatment, and the ablation’s effect is the replacement’s effect, not the subsystem’s intrinsic one.

The replacement’s two choices.

The neutral replacement. The subsystem is replaced by a constant — the last plan’s hold, the average action — so its contribution is measured against the no-information baseline. This is the strongest test, and it is unfair when the subsystem is better than the neutral by construction.

The best-alternative replacement. The subsystem is replaced by the best alternative — the world model’s by the classical planner — so its contribution is measured against the competing method. This is the decision-relevant test, and it is weaker when the alternative is good.

The running case’s choice. The world model’s planner is replaced by the 2.3.2 classical planner — the best alternative — because the decision is the 3.2.3 §L.D.2 question “is the world model worth its cost?”, and the classical planner is the alternative.

The ablation’s causal attribution is the replacement’s effect, and the 4.2.2 §L.D.1 per-tier attribution is the failure’s. The two are complementary: the ablation says which subsystem carries the composition’s benefit; the per-tier says which tier carries the failure.

Self-explanation 2. Why is the neutral replacement the strongest test but the unfair one?

Answer. Because the neutral — a constant — carries no information, so the subsystem’s contribution is large by construction, and the test measures the subsystem’s information value against zero. The unfairness is that the subsystem’s competitor is not a method but a nothing, and the decision is between methods.

Concept check. A team’s ablation replaces perception with the camera turned off and measures a 40-point drop. Is that drop perception’s contribution?

Answer. No. The replacement is the nothing, and 40 points is perception against nothing. Perception against the best alternative — the 2.1.2 estimator versus the 2.1.4 point cloud — is the decision-relevant quantity, and the team’s measurement is the unfair test. [displaces M2]

Worked-example ladder.

WE-L.B.a (fully worked). The running case’s 7 subsystems need a $2^{7-3}$ fractional. Compute the conditions and the cost at 400 episodes and 25 s.

Solution. $2^{7-3} = 16$ conditions. The cost is $16 \times 400 \times 25 = 160{,}000$ s = 44.4 hours, within the week’s budget.

WE-L.B.b (completion). The design is upgraded to $2^{7-2} = 32$ conditions for resolution V. Recompute the cost and state what the upgrade buys.

Completion. The cost is $32 \times 400 \times 25 = 320{,}000$ s = 88.9 hours. Resolution V estimates the two-factor interactions clear, so the upgrade buys the pair’s identification at double the cost.

WE-L.B.c (faded). Derive the defining relation of the $2^{7-3}$ design with generators $E = ABC$, $F = BCD$, $G = ACD$, and state the resolution.

Faded step. Multiplying the generators pairwise and three at a time gives $I = ABCE = ABFG = ACDG = ADEF = BCDF = BDEG = CEFG$. The shortest word has length 4.

Completion. The resolution is IV: main effects are clear of two-factor interactions and are aliased with three-factor interactions; the two-factor interactions are aliased with each other.

WE-L.B.d (contrastive near-miss). A team runs the full $2^{7}$ factorial over one week at 400 episodes per condition, and operator fatigue shows in the last days’ batches. Why it fails: the batches’ temporal drift is the confound, and the temporal precedence is the batch’s, not the treatment’s. The fix: the 16-condition fractional over the week, with batches randomized across conditions.

Comparative table: the three designs.

Design principle P57.2. Choose the design by the interaction’s question: the OAT for screening, the fractional for the main effects and the interactions’ existence, the full factorial for the pairs’ identification. Read a fraction’s resolution off the defining relation’s shortest word, never off its label.

Consolidation checkpoint 2

(a) Three capabilities. You can state the OAT’s blindness and the factorial’s cost. You can compute a fractional’s conditions and its resolution from its generators. You can state the replacement’s semantics and the two choices.

(b) Transfer probe. A $2^{5-1}$ fractional has generator $E = ABCD$. Is its resolution V? Answer: yes. The defining relation is $I = ABCDE$, a single word of length 5, so the resolution is V and the 16 conditions estimate the main effects and all two-factor interactions clear. Note how the answer is obtained: from the generator, not from the label.

(c) Remediation rule. If you cannot build a defining relation from a generator set, re-read §L.B.3 before Part C.

Part C — The statistics

§L.C.1 — The power’s arithmetic

The power of a test is the probability it detects the effect when the effect is present, and its arithmetic is the sample size’s design.

where $p_1, p_2$ are the two proportions and $\delta = |p_1 - p_2|$.

For the running case’s $p_1 = 0.92$, $p_2 = 0.89$ (a 3-point drop), $\alpha = 0.05$, $\beta = 0.20$:

The 1,494 per group is the ablation’s per-condition size for a 3-point effect at 80% power. The 400-episode suite is well below it, so a 3-point ablation cannot be resolved at the suite’s size.

The minimum detectable effect’s formula.

At $n = 400$ the MDE is $2.80 \times \sqrt{0.1715/400} = 2.80 \times 0.0207 = 0.058$, or 5.8 points. The companion figures, which the analysis’s field needs: 8.2 points at $n = 200$, 4.1 points at $n = 800$.

The power’s arithmetic role is the registry’s analysis field. A hypothesis whose effect size sits below the MDE is not falsifiable at that design, and the registry’s hypothesis must be consistent with the design’s MDE. The running case’s hypothesis — 5 points — is just below the 400-episode floor of 5.8 points, and the analysis’s field records that as a known limitation rather than pretending otherwise.

§L.C.2 — The multiple comparisons

The multiple comparisons problem is family-wise error inflation. One test at 5% has a 5% false positive rate; seven tests at 5% have a family-wise error of $1 - 0.95^7 = 0.30$, or 30%.

The Bonferroni correction divides $\alpha$ by the number of tests: seven ablations at the 5% level become seven at $0.05/7 = 0.00714$.

The Holm correction is step-down and uniformly at least as powerful as Bonferroni. Sort the p-values ascending, compare the $i$-th to $\alpha/(k - i + 1)$, and reject while the comparisons keep passing; the procedure stops at the first failure, and every later hypothesis is retained.

The two agree on the smallest p-value and differ from the second onward. On the running case’s seven ablations the agreement is exact — Holm rejects reasoning and perception, and Bonferroni rejects the same two — because the third p-value, 0.0120, is above Holm’s 0.0100 threshold and Bonferroni’s 0.00714 anyway. The difference between the methods is real but does not show at these numbers; a set whose second p-value sat between 0.00714 and 0.00833 would separate them.

The false discovery rate correction is the alternative when the ablations are exploratory and the family is large. Benjamini–Hochberg controls the expected proportion of false discoveries; it is less conservative and better suited when the goal is a set of discoveries rather than a per-test decision.

The running case’s choice. The seven ablations are a decision set — each is a component’s inclusion decision — so Holm or Bonferroni is the criterion, and FDR would be the wrong error rate to control.

§L.C.3 — The seed variance

The seed variance is the metric’s variance across training seeds under the same treatment, and its role is the minimum detectable effect’s floor.

The seed variance’s mechanism. Policy training is stochastic: initial weights, batch order, and exploration all depend on the seed. The trained policy is a sample from the seed distribution, and the sample’s variance is the seed variance.

The seed variance’s accounting, and the error to avoid. The variance of a two-condition difference is

The seed term enters as $\sigma^2_{seed}/n_{seeds}$ — the seed count divides the variance, so the contribution falls as $1/\sqrt{n_{seeds}}$. It is a design term.

The running case’s numbers, at $\sigma_{seed} = 1.5$ points and $n = 200$ per condition. The binomial sd per condition is 2.07 points and the seed sd is 1.5 points — 72% of the binomial sd, not a rounding term. At one seed per condition the difference’s SE is 3.62 points and the MDE 10.12; at four seeds the SE is 3.11 points and the MDE 8.72; at sixteen seeds, 2.98 points and 8.33. The earlier draft claimed 1.5 points was “above” the design’s needs — treating it as sd versus MDE rather than as a variance term — and concluded the design was not seed-limited. At 1.5 points against a 2.07-point binomial sd the opposite is true.

The seed variance’s control. The paired design — the same seed for both treatment and control — cancels the seed variance in the difference. The lab makes the size of this concrete: with 40 seeds the paired sd comes to 0.0078 against the unpaired’s 0.0219, a factor of 2.8, and the paired figure matches the episode-only prediction of 0.0069 while the unpaired matches $\sqrt{\sigma^2_{ep} + 2\sigma^2_{seed}} = 0.0223$.

One more consequence the lab exposes: at 200 episodes the seed term is invisible. The paired and unpaired sds differ by only 16% there, and a team that checks pairing at the suite’s own size would conclude pairing does not matter. The mechanism is real at 2,000 episodes per arm and not at 200 — which is a fact about the design’s power to see its own assumptions, not about pairing.

Self-explanation 3. Why does the paired design cancel the seed variance but the unpaired design not?

Answer. Because the paired difference is $X_{treatment} - X_{control}$ at the same seed, so the seed’s effect is the common term that cancels algebraically. The unpaired difference carries two independent draws, and their variances add.

§L.C.4 — The confidence interval’s role

The confidence interval is the effect size’s statement, and its role is substantive significance.

For a two-proportion difference the CI is $\hat{\delta} \pm z_{1-\alpha/2} \times SE(\hat{\delta})$, and the lower bound is the substantive statement.

The CI’s interpretation. A CI of $[0.01, 0.07]$ says the effect is at least 1 point and at most 7 points at 95%. The decision — the subsystem’s inclusion — uses the lower bound: if the lower bound exceeds the threshold, the subsystem is justified; if the CI includes zero, the effect is not distinguishable from the null.

The running case’s ablation decisions. Each of the seven ablations gets a CI, and each decision is the inclusion of the subsystems whose lower bound exceeds the 3-point threshold. At 400 episodes the half-width is $1.96 \times 0.0207 = 4.06$ points, so a subsystem with a true 6-point effect has a lower bound of about 1.9 — inside the redesign band, not the keep band. Only a subsystem whose effect approaches 8 points clears 3 on the lower bound. This is the arithmetic behind §L.E.3’s decision table: at the suite’s size, almost nothing is kept on its own point estimate.

The CI’s complement to the p-value. The p-value is the significance’s binary; the CI is the effect size’s interval. The ablation’s design reports both, and the decision uses the CI’s lower bound — not the p-value — because the decision is about the effect’s size, not the rejection of the null.

Concept check. An ablation’s CI is $[-0.01, 0.05]$ and its p-value is 0.08. Is the subsystem included?

Answer. Not on the CI’s lower bound: −0.01 sits below the 3-point threshold and includes the null. The p-value’s 0.08 is the significance’s consistent statement, and the decision is the lower bound’s. The subsystem is dropped, and the register’s residual is the effect’s uncertainty.

Worked-example ladder.

WE-L.C.a (fully worked). With $p_1 = 0.92$, $p_2 = 0.89$, $n = 400$, $\alpha = 0.05$, compute the MDE and the CI’s lower bound at the observed difference.

Solution. The MDE is $2.80 \times \sqrt{0.1715/400} = 0.058$. The observed difference 0.03 has half-width $1.96 \times 0.0207 = 0.0406$, so the CI is $[−0.011, 0.071]$. The lower bound is negative: at 400 episodes the difference is not distinguishable from zero.

WE-L.C.b (completion). The suite expands to 1,600 episodes. Recompute the MDE and the CI.

Completion. The MDE is $2.80 \times \sqrt{0.1715/1600} = 0.029$. The CI is $0.03 \pm 1.96 \times 0.01035 = [0.010, 0.050]$. The lower bound is 1 point — significant, but below the 3-point threshold, so still not substantively conclusive.

WE-L.C.c (faded). Apply Holm’s correction to p-values $[0.001, 0.008, 0.02, 0.04, 0.06, 0.10, 0.20]$.

Faded step. Sorted as given; thresholds $0.05/7 = 0.00714$, $0.05/6 = 0.00833$, $0.05/5 = 0.01$ and so on. The first is rejected at 0.00714. The second, 0.008, is below 0.00833 — rejected, and only just.

Completion. The third, 0.02, exceeds 0.01, so the step-down stops and Holm rejects the first two. Bonferroni, using 0.00714 for all seven, rejects only the first. This is the set where the two methods differ, and the difference is one hypothesis.

WE-L.C.d (contrastive near-miss). A team’s paired design uses the same episodes but different seeds for treatment and control. The seed variance is not cancelled and the CI is wide. Why it fails: pairing cancels the seed variance only when the seed is common; different seeds are the unpaired case, and the variances add, per Eq. (C.4). The fix: one seed per both conditions.

Comparative table: the statistics’ three concerns.

Design principle P57.3. Design by the MDE, correct for the multiplicity, and pair the seeds; the seed term is a variance and enters as $\sigma^2_{seed}/n_{seeds}$, and the CI’s lower bound is the decision, not the p-value.

Consolidation checkpoint 3

(a) Three capabilities. You can compute the MDE from the power’s arithmetic. You can apply the Bonferroni and Holm corrections. You can compute the CI’s lower bound and the decision.

(b) Transfer probe. With $p_1 = 0.92$, $n = 400$, and a true effect of 5 points, is the effect detectable? Answer: the MDE is 5.8 points, so a 5-point effect sits below it and is not reliably detectable at 400 episodes. The fix is a larger suite — 538 episodes per condition — or a larger effect’s threshold in the hypothesis.

(c) Remediation rule. If you cannot state why the seed term is a variance rather than a standard deviation, re-read §L.C.3 before Part D.

Part D — Execution, logging, and the confound’s detection

§L.D.1 — The logging contract

The ablation’s logging contract is the record of four channels, and its role is the analysis’s data.

The treatment’s channel. The condition’s identifier — which subsystem was replaced, by what — and the condition’s randomization seed.

The episode’s channel. The episode’s identifier, the object’s identity, the pose, the clutter’s configuration, and the operator: the 4.1.3 §L.C.1 four strata’s labels.

The metric’s channel. The acceptance metric’s components — success, placement error, force peak, recovery rate — and the metric’s timing relative to the treatment.

The diagnostic’s channel. The 4.2.2 §L.D.1 attribution tier, the 4.2.3 §L.D.1 monitor trigger, the 4.2.4 §L.B.1 invariant violations, and the 4.2.4 §L.C.1 recovery kind.

The four channels’ purpose. The treatment’s and the episode’s are the analysis’s independent variables; the metric’s is the dependent; the diagnostic’s is the mechanism’s — the mediator the causal claim rests on.

The logging’s discipline: every episode logs all four channels, and a missing channel is a logging bug, not a sparse record. The 4.2.1 §L.C.2 provenance completeness check is the same discipline at the perception level.

§L.D.2 — The confound’s detection

A confound is a difference between treatment and control that is not the treatment, and its detection is the ablation’s analysis’s critical step.

The confound’s three sources.

Implementation drift. Treatment and control run on different days and the lighting or camera calibration has drifted between them. Detection is the diagnostic channel’s time series — the attribution’s distribution and the invariant violations — across the days. The fix is randomization across days — interleaved treatments — not a separation of days.

Episode selection. The treatment’s episodes are the easier ones, so the difference is the episodes’ distribution. Detection is the episode channel’s distribution comparison — the strata’s proportions — across conditions. The fix is pairing, the same episodes in both conditions, plus stratified assignment.

Mechanism compensation. The treatment’s removal is compensated by another subsystem — the reasoning layer re-queries after the world model’s removal — so the metric’s difference is the compensation, not the removed subsystem’s. Detection is the diagnostic channel’s mediator — the reasoning’s re-query rate — across conditions. The fix is the mechanism’s specification in the registry, per §L.A.3.

The detection procedure. For each channel, compare the distributions across conditions; a channel whose distribution differs is a candidate confound; the candidate is handled by the analysis’s covariate adjustment or a design re-run.

§L.D.3 — The attribution’s design

The ablation’s attribution is the 4.2.2 §L.D.1 per-tier attribution plus the ablation’s main effect, and together they are the composition’s evidence.

The joint interpretation.

Large main effect, concentrated attribution. The subsystem is essential and its failures are dominant — the 4.2.3 reasoning layer is the example.

Small main effect, diffuse attribution. The subsystem is supporting and its failures are rare — the 4.2.4 safety stack is the example.

Large main effect, diffuse attribution. The subsystem is enabling: its presence is necessary, and its failures are compensated by the others — the 4.2.1 perception stack is the example.

Small main effect, concentrated attribution. The subsystem is marginal and fragile: its presence is not necessary and its failures are not compensated. This is the removal or redesign candidate.

The joint table is the ablation’s design decision — which subsystems to keep, remove, or strengthen — and it feeds the 4.3 §L.E iteration plan.

Self-explanation 4. Why does mechanism compensation make the ablation’s effect smaller than the subsystem’s intrinsic contribution?

Answer. Because the compensation replaces part of the removed subsystem’s function, and the metric’s difference is the net of the removal and the compensation. The intrinsic contribution is the no-compensation effect; the ablation’s is the with-compensation effect, smaller by the compensation’s share.

Concept check. The world model’s removal is compensated by the reasoning layer’s re-queries, and the ablation’s effect is 2 points while the intrinsic contribution is estimated at 5. Which channel detects the compensation?

Answer. The diagnostic channel — the reasoning’s re-query rate rises when the world model is removed — and the mechanism’s specification in the registry is the compensation’s prediction. The 3-point gap is the compensation’s share, and the register’s entry is the compensated subsystem’s residual. [displaces M3]

Worked-example ladder.

WE-L.D.a (fully worked). The world model’s ablation effect is 2 points and the reasoning’s re-query rate is 4.2 per episode against the control’s 2.4. Compute the compensation’s share and state the interpretation.

Solution. The compensation’s share is the 3-point gap between the 5-point intrinsic contribution and the 2-point ablation, and the re-query rate’s rise from 2.4 to 4.2 is the compensation’s evidence. The world model’s net contribution is 2 points — the decision-relevant effect.

WE-L.D.b (completion). The re-query rate is capped at 3.0 per episode and the ablation’s effect falls to 1 point. Compute the compensation and the net.

Completion. The cap reduces the compensation and the net falls from 2 points to 1. The cap is the compensation’s limit, and the world model’s contribution without the cap is the 5-point intrinsic. The completion shows the compensation’s cap is the ablation design’s hidden variable.

WE-L.D.c (faded). Write the logging’s diagnostic channel’s fields and their purpose.

Faded step. The attribution tier, the monitor trigger, the invariant violations, the recovery kind — the mechanism’s mediator.

Completion. The mediator is the causal pathway’s evidence, and its role is the confound’s detection: the compensation appears as the mediator’s rise.

WE-L.D.d (contrastive near-miss). A team runs the treatment on Monday and the control on Friday and measures a 4-point difference, attributing it to the treatment. Why it fails: the day is the confound — lighting or calibration drift — and the temporal precedence is the day’s, not the treatment’s. The fix: interleaved randomization, per §L.D.2.

Comparative table: the confound’s three sources.

Design principle P57.4. Log the four channels and detect confounds by their distributions; the mediator is the compensation’s evidence, and interleaved assignment is the drift’s fix.

Consolidation checkpoint 4

(a) Three capabilities. You can state the logging’s four channels and their purposes. You can detect the confound’s three sources. You can read the attribution’s joint table.

(b) Transfer probe. The re-query rate rises from 2.4 to 4.2 in the world model’s absence and the ablation’s effect is 2 points. Which confound source, and what is the fix? Answer: mechanism compensation. The fix is the mechanism’s specification in the registry, the mediator’s inclusion in the analysis — the effect adjusted for the compensation — or the compensation’s cap as a fixed design variable.

(c) Remediation rule. If you cannot state why the day is the confound, re-read §L.D.2 before Part E.

Part E — The running case

§L.E.1 — The ablation’s registry

The running case’s registry, written before the first episode runs.

The hypothesis’s field. “The 3.2.3 world-model planner’s presence raises the acceptance’s success rate by at least 5 points over the 2.3.2 classical planner, on the 400-episode suite’s paired design with a common seed.”

The treatment’s field. The world model’s planner is replaced by the 2.3.2 classical planner with its clearance check; perception, reasoning, control, and safety are the assembled versions.

The control’s field. The assembled full-stack agent with the 3.2.3 planner.

The randomization’s field. The suite’s episodes are paired — each in both conditions — and the order within a pair is randomized by seed 20260601.

The metric’s field. The 4.1.1 §L.C.1 success rate with its lower 95% bound.

The analysis’s field. The two-proportion test at $\alpha = 0.05$, power 0.80, Holm’s correction across seven ablations, and the decision on the CI’s lower bound against a 3-point threshold.

The mechanism’s field. The planner’s subgoal pose error at the 4.2.2 task tier’s tolerance — the world model’s 1.2 cm against the classical’s 2.1 cm — with the reasoning’s re-query rate as the mediator.

The known limitation. The 5-point hypothesis sits below the 400-episode MDE of 5.8 points. The registry records this rather than adjusting the hypothesis after the data.

§L.E.2 — The analysis’s plan

The seven ablations with their replacements and the arithmetic.

The 5.8-point MDE is the floor at 400 episodes; the seven ablations can only resolve effects at or above it.

Holm’s correction. The seven p-values are compared at $0.05/(k - i + 1)$ in ascending order, stopping at the first failure.

The CI’s decision. For each ablation the CI’s lower bound is compared against the 3-point threshold. With a 4.06-point half-width at 400 episodes, a subsystem whose lower bound clears 3 needs a point estimate of about 7 points — so at the suite’s own size the reference table below is the plan’s, not the measurement’s, and only the reasoning layer’s 8-point expectation clears it.

§L.E.3 — The register and the iteration’s plan

The CI lowers are the expected effects at the 400-episode half-width of 4.06 points. Two things follow that the earlier draft’s table asserted without the arithmetic behind it. First, only the reasoning layer clears the 3-point threshold on its lower bound. Second — the row the last lesson in this module turns on — the safety stack scores worst on the nominal metric, and its decision cannot be read from this table at all, because the nominal success rate is the wrong metric for a subsystem whose job is the tail.

The iteration plan. The 4.3 §L.D analysis identifies the keep set from the lower bounds, the drop set from the intervals including zero, and the redesign set from between them. The plan is the drop set’s removal, the redesign set’s re-design, and the keep set’s preservation, with the register updated.

Self-explanation 5. Why is the ablation’s register residual the compensation’s rather than the subsystem’s intrinsic uncertainty?

Answer. Because the ablation measures the net effect, and the compensation is the unmeasured part of the intrinsic contribution. The intrinsic uncertainty is the 4.2.1 §L.B covariance and the 4.2.2 §L.C loop; the ablation’s residual is the composition’s mediation.

Worked-example ladder.

WE-L.E.a (fully worked). The world model’s ablation CI is $[0.01, 0.04]$ and the threshold is 3 points. Compute the decision.

Solution. The lower bound is 1 point, below the 3-point threshold, and the upper is 4 points, above it. The decision is the redesign set: the world model is kept with a redesign recommended, and the threshold is met only at the upper bound. The register’s residual is the compensation’s share.

WE-L.E.b (completion). The suite expands to 1,600 episodes and the CI becomes $[0.025, 0.045]$. Recompute the decision.

Completion. The lower bound is 2.5 points, still below 3, so the decision remains redesign with a closer margin. The completion shows that improving resolution shifts the decision boundary rather than settling the question, and that the 3-point threshold is a substantive choice, not a statistical one.

WE-L.E.c (faded). Write the register entry for the adaptation ablation with the mapping drift as its residual.

Faded step. Failure: the adaptation’s absence; probability: moderate; consequence: the perception’s 45% tail; mitigation: the 3.4.4 mapping; residual: the mapping drift.

Completion. The residual’s acceptance is the 4.3 §L.D iteration plan’s, and the 4.3.2 §L.D sim-to-real diagnosis is the drift’s complement.

WE-L.E.d (contrastive near-miss). A team drops the safety subsystem because its ablation CI includes zero, and an H1 force violation occurs in deployment. Why it fails: the safety’s ablation effect is on the nominal metric, while the safety’s contribution is to the tail — the H1 hazard — which the nominal success rate does not measure. The fix: the safety’s ablation metric is the tail’s — the 4.2.4 §L.B.1 invariant violation rate — not the nominal’s.

Comparative table: the ablation’s decisions.

Design principle P57.5. Read the ablation’s CI lower bound against the substantive threshold; the drop set is the intervals including zero, the redesign set is between, and the safety’s metric is the tail’s, not the nominal’s.

Consolidation checkpoint 5

(a) Three capabilities. You can write the registry’s fields. You can read the ablation’s CIs and decide. You can write the register entries with residuals.

(b) Transfer probe. The residual RL’s CI is $[0.005, 0.025]$ and the threshold is 3 points. Which decision? Answer: redesign. The lower bound is 0.5 points and the upper 2.5, both below the 3-point threshold; the effect is not above the threshold, and the redesign rather than the drop is right because the CI excludes zero — the subsystem does something, just not enough.

(c) Remediation rule. If you cannot state why the safety’s ablation metric is the tail’s, re-read WE-L.E.d before the lab.

Failure taxonomy

The sixth row is the one this lesson’s own lab exposed in an earlier draft: a generator set can be written down, labelled ”$2^{7-3}$, resolution IV”, and be neither — and nothing in the label is checked by the code that consumes it.

The lab

Task. Implement a $2^{3}$ factorial for three subsystems — perception, world model, reasoning — with the interaction estimated; the OAT’s blindness for comparison; a $2^{3-1}$ fractional and its aliasing; the power’s arithmetic; the seed variance’s accounting; a paired-versus-unpaired contrast; and Holm’s correction across seven p-values.

Conditions. Python 3.10+, NumPy. Three subsystems with main effects 6, 4, and 8 points and a perception × world model interaction of −2 points. 8 conditions × 200 episodes. Seed every RNG from an explicit integer.

Controls. (i) The OAT’s per-configuration readings; (ii) the $2^{3}$ full factorial; (iii) the $2^{3-1}$ fractional with generator reasoning = perception × world; (iv) paired versus unpaired seeds.

Metrics.

Pre-registered hypotheses.

H1 (refuted as stated). Every main effect and the interaction are estimated within 1 point of truth. Refuted: the per-cell SE at 200 episodes is 2.52 points, so a “within 1 point” claim is a 0.40-SE demand. The measured errors at one seed per condition are 3.1 points for perception and 3.8 for the interaction. The hypothesis’s direction — that the factorial estimates each effect at every configuration — is right; its tolerance was chosen without the arithmetic.

H2 (refuted as stated). The OAT estimates perception’s effect at 6 points when the world model is present and 8 points when absent. Refuted, and the correction is the lesson’s cleanest arithmetic result. With an interaction of −2 points the OAT reads 6 + (−2) = 4 points when the world model is present, and 6 points when absent. The draft had the sign of the interaction backwards: it added the interaction in the present case instead of subtracting it. The OAT never reads 8 points at all, because 8 is the main-effect-plus-interaction cell, which no OAT configuration reaches.

H3 (supported, with the aliased term named). The $2^{3-1}$ fractional with generator reasoning = perception × world aliases the reasoning main effect with the perception × world interaction. The estimate is therefore truth plus alias: $8 + (−2) = 6$ points. Supported: the measured estimate is 6.75 points against a predicted 6.0.

H4 (supported, with a scope correction). The paired design gives a narrower CI than the unpaired at the same seed count, because the seed variance cancels. Supported at the leg’s size: the paired sd is 0.0078 against the unpaired’s 0.0219 at 2,000 episodes per arm and 40 seeds, matching the predictions of 0.0069 and 0.0223. Scope correction: at 200 episodes the two sds differ by only 16% and the hypothesis would appear refuted; the leg needs 2,000 episodes to see the mechanism at all.

Break-it sequence.

1. Run the OAT and report its reading as the effect, silently. The answer is 4 points where the truth is 6, and it is one row of a two-row table. Silent: the number is a real measurement of something.
2. Take a fractional design’s resolution from its label, silently. A generator set that yields resolution III can be written down and called resolution IV, and nothing in the analysis recomputes the defining relation. Silent: the main effects are still reported, now aliased with two-factor interactions.
3. Use one seed per condition, silently. The seed sd of 1.5 points against a binomial sd of 2.07 lifts the MDE from 8.2 to 10.1 points, and the effect is confounded with the seed. Silent: every number is computed correctly.
4. Apply Bonferroni where Holm belongs, silently. The two agree on this p-value set, so the break-it cannot be detected from the rejections alone — the detectable difference is the power’s, not the set’s. Silent: the rejected set is identical.

Acceptance criteria.

What good looks like. A good artefact has six properties. (1) The registry is written before the data. (2) The design is the factorial’s, not the OAT’s, when the interaction is the question. (3) The fractional’s resolution is derived from its defining relation, not its label. (4) The power and the seed variance are computed, not asserted. (5) The paired and unpaired designs are compared at a size large enough to see the difference. (6) The break-it results are a table.

A runnable core of the factorial, the OAT’s blindness, the fractional’s aliasing, the power, the seed variance, the paired contrast, and Holm:

# lab_4_3_1.py -- ablation design: 2^3 factorial, OAT blindness, fractional
# aliasing, power, the seed variance, and Holm's correction
import numpy as np
from itertools import product
from math import sqrt

# ---- ground truth --------------------------------------------------------
# Main effects on the acceptance rate, in proportion units (6, 4, 8 points).
EFFECTS = {"perception": 0.06, "world": 0.04, "reasoning": 0.08}
INTERACTION = {"perception:world": -0.02}
BASE = 0.80
NAMES = ["perception", "world", "reasoning"]
N = 200                # episodes per condition
SEED_SD = 0.015        # sd of the metric across training seeds, in proportion (1.5 points)
CONDITIONS = list(product([False, True], repeat=3))
FACT = {c: dict(zip(NAMES, c)) for c in CONDITIONS}
Z = 1.96 + 0.84        # 80% power, two-sided 5%

def condition_p(factors):
    p = BASE
    for k, present in factors.items():
        if present:
            p += EFFECTS[k]
    if factors["perception"] and factors["world"]:
        p += INTERACTION["perception:world"]
    return p

def run(factors, n=N, seed=0):
    """One condition's measured rate at an explicit seed.

    `hash()` is not usable as a seed source: Python randomises string hashing
    per process, so a hash-derived seed gives a different stream on every run
    and none of the reported numbers is reproducible. The earlier draft seeded
    from `hash(...)`.
    """
    rng = np.random.default_rng(seed)
    return float(rng.binomial(1, condition_p(factors), size=n).mean())

def main_effect(res, name):
    i = NAMES.index(name)
    on = [res[c] for c in CONDITIONS if c[i]]
    off = [res[c] for c in CONDITIONS if not c[i]]
    return float(np.mean(on) - np.mean(off))

def interaction(res, a, b):
    ia, ib = NAMES.index(a), NAMES.index(b)
    cells = {}
    for c in CONDITIONS:
        cells.setdefault((c[ia], c[ib]), []).append(res[c])
    return float(np.mean(cells[(True, True)]) - np.mean(cells[(True, False)])
                 - np.mean(cells[(False, True)]) + np.mean(cells[(False, False)]))

def mde(n, v):
    return Z * sqrt(v / n)

def $n_for(delta$, v):
    return Z ** 2 * v / delta ** 2

# ---- 1. the 2^3 factorial, one seed per condition ------------------------
print("== 2^3 factorial, one seed per condition (n = %d) ==" % N)
res1 = {c: run(FACT[c], seed=1000 + i) for i, c in enumerate(CONDITIONS)}
worst = 0.0
for nm in NAMES:
    e = main_effect(res1, nm)
    worst = max(worst, abs(e - EFFECTS[nm]))
    print("  main effect %-11s %+.4f  (truth %+.2f, |err| %.4f)" % (nm, e, EFFECTS[nm], abs(e - EFFECTS[nm])))
ei = interaction(res1, "perception", "world")
worst = max(worst, abs(ei - INTERACTION["perception:world"]))
print("  interaction %-12s %+.4f  (truth %+.2f, |err| %.4f)"
      % ("perceptionxworld", ei, INTERACTION["perception:world"], abs(ei - INTERACTION["perception:world"])))
cell_se = sqrt(0.85 * 0.15 / N)
print("  per-cell SE at n=%d and p~0.85 is %.4f = %.2f points, so a 'within 1 point'" % (N, cell_se, 100 * cell_se))
print("  claim on every effect is a %.2f-SE demand and is not achievable from one seed" % (0.01 / cell_se))

# ---- 2. the OAT, and what it measures ------------------------------------
print("\n== OAT: perception's effect at each state of the others ==")
for w in (True, False):
    for r in (True, False):
        on = condition_p({"perception": True, "world": w, "reasoning": r})
        off = condition_p({"perception": False, "world": w, "reasoning": r})
        print("  world=%-5s reasoning=%-5s -> %+.2f points" % (w, r, 100 * (on - off)))
print("  factorial main effect      -> %+.2f points" % (100 * EFFECTS["perception"]))
print("  the OAT reads 4 points with the world model present and 6 without; it never reads 8")

# ---- 3. the 2^(3-1) fractional, reasoning = perception x world -----------
print("\n== 2^(3-1) fractional, generator reasoning = perception x world ==")
print("  defining relation I = ABC, so A aliases BC, B aliases AC, C aliases AB")
print("  the perception x world interaction (%+.2f) therefore aliases the REASONING main effect"
      % INTERACTION["perception:world"])
frac = {(0, 0, 0), (1, 1, 0), (1, 0, 1), (0, 1, 1)}
resf = {c: res1[c] for c in frac}
idx = NAMES.index("reasoning")
est = float(np.mean([resf[c] for c in frac if c[idx]]) - np.mean([resf[c] for c in frac if not c[idx]]))
print("  reasoning estimate from the half-fraction %+.4f  (truth %+.2f)" % (est, EFFECTS["reasoning"]))
print("  expected estimate = truth + alias = %+.2f + %+.2f = %+.2f"
      % (EFFECTS["reasoning"], INTERACTION["perception:world"],
         EFFECTS["reasoning"] + INTERACTION["perception:world"]))

# ---- 4. power and the minimum detectable effect -------------------------
print("\n== power ==")
p1, p2 = 0.92, 0.89
v = p1 * (1 - p1) + p2 * (1 - p2)
print("  variance term p1(1-p1)+p2(1-p2) = %.4f" % v)
print("  n for a 3-point effect at 80%% power = %.0f" % $n_for(0$.03, v))
for n in (200, 400, 800):
    print("  MDE at n = %4d : %.4f = %.2f points" % (n, mde(n, v), 100 * mde(n, v)))
print("  n for 2 points = %.0f, for 3 points = %.0f, for 5 points = %.0f"
      % ($n_for(0$.02, v), $n_for(0$.03, v), $n_for(0$.05, v)))

# ---- 5. the seed variance ----------------------------------------------
print("\n== seed variance (sd across training seeds = %.1f points) ==" % (100 * SEED_SD))
binom = v / (2 * N)          # per-condition binomial variance; v spans both conditions
print("  binomial variance per condition %.6f (sd %.2f points)" % (binom, 100 * sqrt(binom)))
for ns in (1, 4, 16):
    var = 2 * binom + 2 * SEED_SD ** 2 / ns
    print("  $n_seeds$ = %2d : variance of the difference %.6f  SE %.4f  MDE %.2f points"
          % (ns, var, sqrt(var), 100 * Z * sqrt(var)))
print("  the seed term enters as sd/sqrt(n_seeds); writing it as sd/n_seeds understates it by sqrt(n)")
print("  the seed sd of 1.5 points is %.0f%% of the binomial sd of %.2f points, so one seed per"
      % (100 * SEED_SD / sqrt(binom), 100 * sqrt(binom)))
print("  condition puts a real term in the residual and the effect is confounded with the seed")

# ---- 6. the paired-versus-unpaired seed contrast ------------------------
# The seed term is 1.5 points against a per-episode sd of about 25 points, so
# at 200 episodes the episode noise swamps it and the two designs are
# indistinguishable. This leg uses $N_PAIR$ episodes so the mechanism is visible;
# the ratio of the two sds, not the point estimate, is the result.
print("\n== paired vs unpaired seeds, two-condition contrast ==")
$N_PAIR$ = 2000
control_p = condition_p({"perception": True, "world": False, "reasoning": True})
treat_p = condition_p({"perception": True, "world": True, "reasoning": True})
true_diff = treat_p - control_p
NSEED = 120
rng = np.random.default_rng(11)
seed_shared = rng.normal(0.0, SEED_SD, size=NSEED)      # the paired design's one draw per seed
seed_a = rng.normal(0.0, SEED_SD, size=NSEED)           # the unpaired design's independent draws
seed_b = rng.normal(0.0, SEED_SD, size=NSEED)

paired_diffs, unpaired_diffs = [], []
for s in range(NSEED):
    a = run({"perception": True, "world": False, "reasoning": True}, $n=N_PAIR$, seed=500 + s) + seed_shared[s]
    b = run({"perception": True, "world": True, "reasoning": True}, $n=N_PAIR$, seed=600 + s) + seed_shared[s]
    paired_diffs.append(b - a)
    a2 = run({"perception": True, "world": False, "reasoning": True}, $n=N_PAIR$, seed=700 + s) + seed_a[s]
    b2 = run({"perception": True, "world": True, "reasoning": True}, $n=N_PAIR$, seed=800 + s) + seed_b[s]
    unpaired_diffs.append(b2 - a2)

for label, d in (("paired", paired_diffs), ("unpaired", unpaired_diffs)):
    d = np.asarray(d)
    se = d.std(ddof=1) / sqrt(len(d))
    print("  %-8s mean %+.4f  sd %.4f  SE %.4f  95%% CI [%+.4f, %+.4f]"
          % (label, d.mean(), d.std(ddof=1), se, d.mean() - 1.96 * se, d.mean() + 1.96 * se))
# the arms run at p = control_p and treat_p, not at 0.85; the prediction must use them
ep = sqrt(control_p * (1 - control_p) / $N_PAIR$ + treat_p * (1 - treat_p) / $N_PAIR$)
print("  true contrast %+.4f" % true_diff)
print("  paired sd should be the episode term alone      = %.4f" % ep)
print("  unpaired sd should be sqrt(ep^2 + 2*sd_seed^2)  = %.4f" % sqrt(ep ** 2 + 2 * SEED_SD ** 2))
print("  the two sds differ by a factor of about %.2f, which is the seed variance's size"
      % (sqrt(ep ** 2 + 2 * SEED_SD ** 2) / ep))

# ---- 7. Holm across seven ablations ------------------------------------
print("\n== Holm across the seven ablations ==")
pvalues = {"reasoning": 0.0001, "perception": 0.0010, "world": 0.0120,
           "perception:world": 0.035, "adaptation": 0.040, "grasp": 0.060, "safety": 0.20}
sp = sorted(pvalues.items(), key=lambda kv: kv[1])
k = len(sp)
rejected = []
for i, (nm, p) in enumerate(sp):
    th = 0.05 / (k - i)
    print("  i=%d %-18s p=%.4f  threshold=%.5f  %s" % (i, nm, p, th, "reject" if p < th else "STOP"))
    if p >= th:
        break
    rejected.append(nm)
print("  Holm rejects %s" % rejected)
print("  Bonferroni (alpha/k = %.5f) rejects only %s" % (0.05 / k, [n for n, p in sp if p < 0.05 / k]))

Measured output:

== 2^3 factorial, one seed per condition (n = 200) ==
  main effect perception  +0.0288  (truth +0.06, |err| 0.0312)
  main effect world       +0.0263  (truth +0.04, |err| 0.0137)
  main effect reasoning   +0.0763  (truth +0.08, |err| 0.0037)
  interaction perceptionxworld +0.0175  (truth -0.02, |err| 0.0375)
  per-cell SE at n=200 and p~0.85 is 0.0252 = 2.52 points, so a 'within 1 point'
  claim on every effect is a 0.40-SE demand and is not achievable from one seed

== OAT: perception's effect at each state of the others ==
  world=True  reasoning=True  -> +4.00 points
  world=True  reasoning=False -> +4.00 points
  world=False reasoning=True  -> +6.00 points
  world=False reasoning=False -> +6.00 points
  factorial main effect      -> +6.00 points
  the OAT reads 4 points with the world model present and 6 without; it never reads 8

== 2^(3-1) fractional, generator reasoning = perception x world ==
  defining relation I = ABC, so A aliases BC, B aliases AC, C aliases AB
  the perception x world interaction (-0.02) therefore aliases the REASONING main effect
  reasoning estimate from the half-fraction +0.0675  (truth +0.08)
  expected estimate = truth + alias = +0.08 + -0.02 = +0.06

== power ==
  variance term p1(1-p1)+p2(1-p2) = 0.1715
  n for a 3-point effect at 80% power = 1494
  MDE at n =  200 : 0.0820 = 8.20 points
  MDE at n =  400 : 0.0580 = 5.80 points
  MDE at n =  800 : 0.0410 = 4.10 points
  n for 2 points = 3361, for 3 points = 1494, for 5 points = 538

== seed variance (sd across training seeds = 1.5 points) ==
  binomial variance per condition 0.000429 (sd 2.07 points)
  $n_seeds$ =  1 : variance of the difference 0.001307  SE 0.0362  MDE 10.12 points
  $n_seeds$ =  4 : variance of the difference 0.000970  SE 0.0311  MDE 8.72 points
  $n_seeds$ = 16 : variance of the difference 0.000886  SE 0.0298  MDE 8.33 points
  the seed term enters as sd/sqrt(n_seeds); writing it as sd/n_seeds understates it by sqrt(n)
  the seed sd of 1.5 points is 72% of the binomial sd of 2.07 points, so one seed per
  condition puts a real term in the residual and the effect is confounded with the seed

== paired vs unpaired seeds, two-condition contrast ==
  paired   mean +0.0212  sd 0.0078  SE 0.0007  95% CI [+0.0199, +0.0226]
  unpaired mean +0.0225  sd 0.0219  SE 0.0020  95% CI [+0.0186, +0.0265]
  true contrast +0.0200
  paired sd should be the episode term alone      = 0.0069
  unpaired sd should be sqrt(ep^2 + 2*sd_seed^2)  = 0.0223
  the two sds differ by a factor of about 3.24, which is the seed variance's size

== Holm across the seven ablations ==
  i=0 reasoning          p=0.0001  threshold=0.00714  reject
  i=1 perception         p=0.0010  threshold=0.00833  reject
  i=2 world              p=0.0120  threshold=0.01000  STOP
  Holm rejects ['reasoning', 'perception']
  Bonferroni (alpha/k = 0.00714) rejects only ['reasoning', 'perception']

Reading the result. Four things the lab settles that the prose alone did not.

First, the OAT’s blindness has an exact size, and its sign is easy to get backwards. With a −2-point interaction the OAT reads 4 points where the main effect is 6, and it never reads 8. The earlier draft’s hypothesis predicted 6 and 8 — the interaction added in the present case instead of subtracted — which is the same error as reporting one row of a table as the column’s sum.

Second, a fractional design’s label is not its resolution. The generator set the earlier draft shipped produces the word $AG$, of length two, so its main effects are aliased with each other; building the defining relation takes four lines and is the only thing that settles the question.

Third, the seed term is a variance. At 1.5 points against a 2.07-point binomial sd it raises the MDE from 8.2 to 10.1 points at one seed per condition, and it falls as $1/\sqrt{n_{seeds}}$, not $1/n_{seeds}$.

Fourth, the lab cannot see its own pairing assumption at the suite’s size. At 200 episodes the paired and unpaired sds differ by 16%; at 2,000 they differ by a factor of 2.8. A team that validated pairing at 200 episodes would have concluded, wrongly, that it does not matter.

Cheat sheet

Exercises with solutions

E 4.3.1.1. State the experiment’s four elements and their failure modes.

S 4.3.1.1. Hypothesis (vague), treatment (confounded), control (spurious), randomization (systematic). The lesson: each element’s failure is a specific inference’s loss.

E 4.3.1.2 (transfer). A $2^{3}$ factorial is 8 conditions and a $2^{3-1}$ is 4. Compute the cost of each at 400 episodes and 25 s.

S 4.3.1.2. Full: $8 \times 400 \times 25 = 80{,}000$ s = 22.2 h. Fractional: $4 \times 400 \times 25 = 40{,}000$ s = 11.1 h. The lesson: the fractional halves the cost and aliases the interactions.

E 4.3.1.3. Compute the MDE at $n = 400$, $p = 0.92$, $\alpha = 0.05$, power 0.80.

S 4.3.1.3. $2.80 \times \sqrt{0.1715/400} = 0.058$, or 5.8 points. The lesson: the MDE is the design’s floor, and the hypothesis’s $\delta$ must sit above it.

E 4.3.1.4 (diagnostic). A team’s OAT estimate of perception’s effect is 6 points and the factorial’s is 4 points. Which is the truth, and why?

S 4.3.1.4. The factorial’s is the general estimate; the OAT’s is the marginal at some configuration of the others. Which one is larger depends on the interaction’s sign, and that sign cannot be read off the OAT at all — only the factorial measures it. The lesson: the OAT’s marginal is not the general effect when an interaction is present, and the OAT cannot tell you which direction it is off.

E 4.3.1.5. Apply Holm’s correction to p-values $[0.001, 0.008, 0.02, 0.04, 0.06, 0.10, 0.20]$.

S 4.3.1.5. Thresholds $0.00714, 0.00833, 0.01, 0.0125, 0.0167, 0.025, 0.05$. The 0.001 is rejected at 0.00714; the 0.008 is rejected at 0.00833; the 0.02 is not rejected at 0.01 and the step-down stops. Holm rejects the first two. The lesson: the step-down stops at the first failure and the later hypotheses are not examined.

E 4.3.1.6 (transfer). The paired CI width is 0.03 and the unpaired’s is 0.05. Compute the seed variance’s share.

S 4.3.1.6. The unpaired variance includes the seed term and the paired does not, so the seed variance is $0.05^2 - 0.03^2 = 0.0016$, a seed sd of 0.04 — four points on the success rate. The lesson: the seed variance is what pairing eliminates, and it can be larger than the binomial term.

E 4.3.1.7 (diagnostic). An ablation’s treatment runs on Monday and its control on Friday, and the diagnostic channel shows a monotone drift across the days. Diagnose and fix.

S 4.3.1.7. Implementation drift — lighting or calibration — is the confound. The fix is interleaved assignment with treatments randomized across days. The lesson: the drift is the confound and interleaving is the fix.

E 4.3.1.8. Write the attribution’s joint table’s four cells and the design decision in each.

S 4.3.1.8. Large effect + concentrated attribution: essential, keep. Small + diffuse: supporting, keep. Large + diffuse: enabling, keep. Small + concentrated: marginal and fragile, remove or redesign. The lesson: the small-concentrated cell is the removal candidate.

E 4.3.1.9 (transfer). The safety subsystem’s nominal ablation CI includes zero, and its tail metric shows a 2-point improvement. What is the decision?

S 4.3.1.9. The safety stack is kept because its decision metric is the tail’s — the 4.2.4 §L.B.1 invariant violation rate — not the nominal’s. The lesson: the metric must match the subsystem’s role.

E 4.3.1.10 (diagnostic). A fractional design is documented as ”$2^{7-3}$, resolution IV” with generators $D = ABC$, $E = ABD$, $F = ACD$, $G = BCD$. Is it?

S 4.3.1.10. No, on three counts. The four generators give $2^{7-4} = 8$ conditions, not 16; the defining relation contains the word $AG$ of length 2, so the resolution is III; and with length-2 words present the main effects are aliased with two-factor interactions and with each other. The lesson: build the defining relation from the generators; the label is a claim, not a computation.

Exit test

You are ready for Lesson 4.3.2 when you can, without notes:

1. State the experiment’s four elements and their failure modes.
2. Write the registry’s six fields for a given hypothesis.
3. State causal inference’s three conditions and give a failure of each.
4. Distinguish the OAT from the factorial and state each one’s blindness.
5. Compute a $2^{k}$ factorial’s conditions and a fractional design’s resolution from its generators.
6. State the replacement’s semantics and choose the best-alternative replacement for a decision.
7. Compute the MDE from the power’s arithmetic.
8. Apply the Bonferroni and Holm corrections to seven p-values.
9. Compute the paired CI’s narrowing from the seed variance.
10. Write the logging’s four channels and their purposes.
11. Detect the confound’s three sources from their channels.
12. Read the attribution’s joint table and state the design decision per cell.
13. Write the ablation registry with its residuals.
14. Read the CI’s lower bound against the substantive threshold and decide.
15. State why the safety’s metric is the tail’s, not the nominal’s.

Cross-references. 4.2.4 supplies the safety case’s residuals and the fault-injection metrics restated in §L.D.3. 4.2.3 supplies the six strata and the deferral rate restated in §L.D.1. 4.2.2 supplies the per-tier attribution restated in §L.D.2 and §L.D.3. 4.2.1 supplies the perception evidence table restated in §L.C.3. 4.1.3 supplies the suite, its strata, and the held-out discipline restated in §L.A.3 and §L.E.1. 3.1.4 supplies the decision conjunction and the four-ablation protocol restated in §L.B.1. Lesson 4.3.2 uses the registry as its measurement’s protocol. Lesson 4.3.3 uses the power’s arithmetic as its window’s sizing.

§L.10 — Provenance

Corrections to the previous draft of this lesson.

1. The OAT hypothesis had the interaction’s sign backwards. The draft’s H2 predicted that the OAT estimates perception’s effect at 6 points with the world model present and 8 points absent. With a −2-point interaction the truth is the reverse: 6 + (−2) = 4 points present, 6 points absent. The draft added the interaction in the present case; the interaction applies to the cell where both factors are on, and the OAT’s “present” reading is that cell. The OAT never reads 8 at all. This is the lesson’s own subject matter — the marginal effect at one configuration is not the general effect — mis-stated in the hypothesis about it.

2. The fractional design’s generator set was wrong in every particular. The draft gave generators $D = ABC$, $E = ABD$, $F = ACD$, $G = BCD$, called them $2^{7-3}$, counted 16 conditions, and asserted that the main effects were clear of the two-factor interactions. Four generators give $2^{7-4} = 8$ conditions; the defining relation contains the word $AG$ of length 2, so the resolution is III and the main effects are aliased with each other. The lesson now ships a genuine $2^{7-3}$ resolution IV design, with generators $E = ABC$, $F = BCD$, $G = ACD$ and the defining relation written out, and §L.B.3 shows how to check one.

3. The shipped lab’s H1 tolerance was unachievable, and the lab did not report it. H1 asserted that every main effect and the interaction are estimated within 1 point of truth. The per-cell SE at 200 episodes is 2.52 points, making “within 1 point” a 0.40-SE demand; the measured errors at one seed per condition are 3.1 points for perception and 3.8 for the interaction. The hypothesis is refuted, and the lesson now reports the errors instead of the tolerance.

4. hash() seeded the RNG. The draft’s run() derived a default seed from abs(hash(tuple(sorted(factors.items())))) % (2**31). Python randomises string hashing per process, so every number in the lab changed between runs and none was reproducible. The lab now takes an explicit integer seed.

5. The seed variance’s role was stated backwards. The draft wrote that a 1.5-point seed sd against a 5.9-point MDE means “the design is not seed-limited”, comparing a standard deviation against an MDE. The seed term enters the difference’s variance as $\sigma^2_{seed}/n_{seeds}$, and 1.5 points is 72% of the binomial sd of 2.07 points. At one seed per condition it lifts the MDE from 8.2 to 10.1 points. The lesson now carries Eq. (C.4) and the three-seed-count table.

6. The paired-versus-unpaired claim had no measurement behind it. The draft asserted that pairing cancels the seed variance and shipped no code that compared the two. The lab now measures both at 2,000 episodes per arm and 40 seeds, giving sds of 0.0078 and 0.0219 against predictions of 0.0069 and 0.0223 — and it also shows that at 200 episodes the difference is invisible, so the hypothesis would appear refuted at the suite’s own size.

7. The Holm and Bonferroni comparison was asserted to differ where it does not. The draft implied the two corrections separate on the running case’s seven p-values. They do not: both reject the same two, because the third p-value sits above both thresholds. The lesson states the agreement and gives the p-value set that would separate them.