Lesson 4.3.2 — Diagnosing Sim-to-Real Failures

Week 39

Prerequisite lessons: 4.3.1 (systematic experimentation and ablations), 3.4.1 (the reality gap as a spectrum), 3.4.2 (domain randomization), 3.4.3 (system identification), 3.4.4 (real-to-sim adaptation), 4.1.3 (evaluation design and held-out test suites), 4.2.4 (safety and recovery).

Prior results used in this lesson. From 4.3.1: the experiment registry’s six fields, causal inference’s three conditions, the factorial design’s blindness, and the confound’s three sources, restated in §L.A.3 and §L.D.1. From 3.4.1: the four axes — dynamics, perception, actuation, timing — the twin-track’s tail, and the axis decomposition’s controlled experiment, restated in §L.A.2 and §L.B.4. From 3.4.2: the randomization’s range-from-uncertainty rule, restated in §L.B.1. From 3.4.3: the identification’s interval and its structure-error allowance, restated in §L.B.2. From 3.4.4: the adaptation loop and its two-sample MMD, restated in §L.B.3. From 4.1.3: the held-out discipline and the 400-episode suite, restated in §L.A.3. From 4.2.4: the fault-injection suite’s four metrics, restated in §L.D.2. Each is restated at the point of use.

Hands forward. The diagnosis’s decision tree, the discriminating tests’ costs, the expected information gain’s arithmetic, and the stopping rule. Lesson 4.3.3 (long-horizon robustness) uses the stopping rule’s residual to size its window. Module 4.4 (the master’s defence) uses the diagnosis’s evidence chain as its unknown-unknowns artefact.

The running case in one sentence. The arm, gripper, eye-in-hand camera, table, cube, bin, divider, and three distractors are unchanged; the policy’s success in the simulator is 0.94 and on the real cell is 0.82, and this lesson asks which axis’s error produces the 12-point gap, which test discriminates the hypotheses, and when to stop testing.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) classify the twin-track’s divergence by the mechanism it implicates; (2) state the diagnosis’s logic — symptom, hypothesis, discriminating test, confirmation — and its cost trade; (3) state the four axes’ discriminating tests and what each holds fixed; (4) compute a test’s expected information gain and sequence the tests by it; (5) apply the stopping rule and state its residual criterion; (6) name the three sim-to-real confounds and each one’s detection channel; (7) distinguish the policy-sim pair’s confounding from the model’s error; (8) state the four failure modes’ signatures; (9) assemble the running case’s evidence chain with confidence bounds; (10) write the diagnosis register’s entries with residuals.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 3.4.1 §L.A.3 the divergence’s three mechanisms were dynamics, perception, and action-scale, with signatures: dynamics grows with trajectory length, perception is present at $t=1$, action-scale is proportional to action magnitude. Which mechanism does the running case’s 12-point outcome gap implicate, and why?

Answer. None of them directly. The signatures are properties of the state’s divergence; the outcome’s 12-point gap is consistent with any mechanism, and the state’s divergence signature is the disambiguating data. Used in §L.A.1 to state the diagnosis’s first step.

Q2. In 4.3.1 §L.A.3 the causal inference’s three conditions were temporal precedence, mechanism pathway, and counterfactual. Which does the twin-track’s paired design satisfy, and which does the policy freeze satisfy?

Answer. The paired design satisfies the counterfactual — the same episodes in the sim and the real — and the policy freeze satisfies the temporal condition, because the treatment’s variation precedes the metric. The mechanism pathway is the discriminating test’s, operationalized by the axis swap. Used in §L.D.1 to state the diagnosis’s confounds.

Q3. In 3.4.3 §L.E.2 the identification’s interval was the statistical uncertainty and the structure-error allowance widened the range. What is the diagnosis’s analogue of that allowance, and why does it matter?

Answer. The analogue is the policy’s difference between sim and real retraining. The structure error is the model’s form; the policy’s retraining error is the pair’s form. It matters because the pair’s effect is what the diagnosis measures, and the model’s alone is only the ranking’s hypothesis. Used in §L.D.2 to state the policy-sim pair’s confounding.

Q4. In 4.1.3 §L.D.2 the drift threshold was the suite’s own sampling bound. What does the diagnosis’s stopping rule take from drift detection, and what differs?

Answer. The stopping rule uses the residual within tolerance — the axis’s residual after treatment — while drift detection uses a distributional shift. The stopping rule is a one-time decision; drift detection is an ongoing monitor. Used in §L.C.3 to state the stopping rule.

Part A — The symptom, the hypotheses, and the diagnosis’s logic

§L.A.1 — The symptom’s classification

The symptom is the observation that the policy’s performance differs between sim and real, and its classification is the diagnosis’s first step.

The running case’s symptom.

The outcome gap. The sim’s success rate is 0.94, the real cell’s 0.82 — a 12-point gap. The 3.4.1 §L.C.1 coverage-ratio error is $(1-0.82)/(1-0.94) = 3.0$, at the edge of the 3.2.1 §L.D.3 threshold.

The state divergence’s signature. The 3.4.1 §L.A.2 twin-track’s state divergence at $t = 15$ is 1.2 cm mean and 4.5 cm at the 95th percentile, against the 3.4.1 §L.E 3 cm tolerance. The divergence grows from 3 mm at $t=1$ to 1.2 cm at $t=15$ — that is a factor of 4 over a factor of 15 in horizon, giving §L.A.1’s exponent 0.51 — and from 1.2 cm at $t=15$ to 2 cm at $t=20$, an exponent of 1.78.

The mechanism’s implication. Growth that steepens with the trajectory’s length — exponent 0.51 early, 1.78 late — is the dynamics mismatch’s signature, per 3.4.1 §L.A.3, and it is not the perception’s or the action-scale’s, whose signatures are respectively constant and action-proportional.

The caveat. Super-linear growth does not exclude perception and action-scale; it ranks dynamics as dominant under the signature’s interpretation. The signature is the first hypothesis; the discriminating test is the evidence.

Self-explanation 1. Why is the signature’s interpretation a ranking rather than a conclusion?

Answer. Because the signatures overlap. A dynamics mismatch can produce a perception-like constant offset if the mismatch is in contact stiffness, so the signature is the marginal under each hypothesis, not sufficient evidence. The signature ranks the hypotheses; the discriminating test is the evidence.

§L.A.2 — The four axes’ hypotheses

The hypotheses are the four axes’ contributions, each with a parameter form and a magnitude.

The dynamics hypothesis. The simulator’s contact-law parameters $(m, \mu, k_c, b_c)$ differ from the real cell’s by more than the 3.4.3 §L.E.2 intervals’ width. The magnitude is the share of the tail that the mismatch produces.

The perception hypothesis. The simulator’s rendered-image distribution differs from the camera’s by more than the 3.4.4 §L.D.1 MMD threshold. The magnitude is the tail’s share that perception produces.

The actuation hypothesis. The simulator’s actuator-model delay and gain differ from the real’s by more than the 3.4.3 §L.B.4 resolution. The magnitude is the tail’s share.

The timing hypothesis. The simulator’s control period and jitter differ from the real’s by more than the 3.4.1 §L.B.2 intervals. The magnitude is the tail’s share.

The hypotheses’ priors come from the 3.4.1 §L.B.2 axis ranking — dynamics 35%, perception 45%, actuation 15%, timing 5% — and the prior is the starting distribution, not the answer.

The hypotheses’ joint structure is compositional. Summing the axes’ contributions assumes additivity, and the axes’ interaction — a dynamics mismatch whose effect depends on perception — is the 4.3.1 §L.B.1 factorial question raised to the sim-to-real level.

§L.A.3 — The diagnosis’s logic and its cost

The diagnosis infers from the symptom to the axis attribution, and its logic has four steps.

Step 1: classify the symptom. The twin-track’s divergence signature and the outcome gap’s magnitude rank the four hypotheses.

Step 2: select the hypothesis. The highest-ranked hypothesis is the first test’s target, and the test must be discriminating — its result separates that hypothesis from the others.

Step 3: run the discriminating test. The test is the 3.4.1 §L.B.3 controlled experiment: one axis varied, the others held. Its result is that axis’s contribution to the tail.

Step 4: confirm and stop. The result confirms if the axis’s contribution exceeds the alternatives’ and the residual is within the deployment’s tolerance. If not, the next hypothesis is the target and the loop continues.

The diagnosis’s cost trade. Each test is a twin-track run, and the twin-track’s cost is real-cell time. The 3.4.1 §L.C.2 50 paired episodes at a 25 s cycle is 21 minutes per test. Four tests in sequence is 84 minutes, and the sequencing rule is the information value of each.

The diagnosis’s constraints. The policy must be frozen across tests — the 4.3.1 §L.A.3 temporal condition — or retraining is the confound. Episodes must be paired across sim and real — the counterfactual condition — or the distribution is the confound. The sim version must be constant — implementation drift — or the version is the confound.

The diagnosis’s output is the evidence chain: symptom, ranking, test results, confirmation. The 4.3.1 §L.A.2 registry records it.

Concept check. The symptom’s signature implicates dynamics, and the dynamics test’s result is a 2-point contribution to the 12-point gap. Is the diagnosis complete?

Answer. No. The 2 points leave 10 unexplained, and the residual is not within the deployment’s 3-point tolerance. The next hypothesis — perception, by the prior ranking — is the next target, and the loop continues. [displaces M2]

Worked-example ladder.

WE-L.A.a (fully worked). The twin-track’s state divergence is 3 mm at $t=1$, 1.2 cm at $t=15$, and 2 cm at $t=20$. Compute the 3.2.2 §L.D.3 power-law exponent over $t=15$ to $t=20$ and state the mechanism’s implication.

Solution. The magnitude ratio is $2/1.2 = 1.667$ over a horizon factor of $20/15 = 1.333$; the exponent is $\log(1.667)/\log(1.333) = 0.511/0.288 = 1.78$. An exponent above 1 is super-linear growth, and growth that steepens with trajectory length is the dynamics mismatch’s signature. The same computation over $t=1$ to $t=15$ gives $\log(1.2/0.3)/\log(15) = 0.51$ — sub-linear — so the steepening, not the growth alone, is what implicates feedback dynamics.

WE-L.A.b (completion). The divergence is 3 mm at $t=1$ and 4 mm at $t=20$. Compute the exponent and state the implication.

Completion. The ratio is $4/3 = 1.333$ over a horizon factor of 20; the exponent is $\log(1.333)/\log(20) = 0.288/2.996 = 0.096$. An exponent near zero is constant divergence: the error is present from the start and persists, which is the perception or action-scale signature.

WE-L.A.c (faded). Write the four hypotheses’ priors from the 3.4.1 §L.B.2 axis ranking and state the prior’s role.

Faded step. Dynamics 35%, perception 45%, actuation 15%, timing 5%. The prior’s role is sequencing — highest first — and it comes from the 3.4.1 §L.B.3 sensitivity analysis, not from the answer.

Completion. The prior is the starting distribution and the test’s result is the posterior update. The prior’s strength is what the 3.4.1 §L.B.3 sensitivity analysis probes, and the posterior is the test’s output.

WE-L.A.d (contrastive near-miss). A team’s symptom classification implicates dynamics, but the team’s first test is the perception test. The perception result is a 6-point contribution and the team’s second test is dynamics. Why it fails: the sequencing was the prior’s reverse, and the two tests’ results are the same regardless of order; sequencing is a cost decision, not a correctness one, and running the low-prior test first spends budget before the high-prior information arrives. The fix: sequence by prior, per §L.A.2.

Comparative table: the four hypotheses and their signatures.

Design principle P58.1. Diagnose the axis, not the symptom; the symptom’s signature ranks the hypotheses, and the discriminating test is the evidence.

Consolidation checkpoint 1

(a) Three capabilities. You can classify the tail’s signature by the mechanism it implicates. You can write the four hypotheses with their priors and signatures. You can state the diagnosis’s four steps and its cost trade.

(b) Transfer probe. The divergence is 5 mm at $t=1$ and 50 mm at $t=20$, with action magnitudes around 2 cm. Which mechanism does the signature implicate? Answer: the action-proportional one, if the divergence scales with action magnitude — the actuation hypothesis — and the 3.4.3 §L.B.4 step response is the discriminating test. The initial 5 mm is the actuation delay’s signature.

(c) Remediation rule. If you cannot state why steepening growth implicates dynamics, re-read §L.A.1 before Part B.

Part B — The discriminating tests

§L.B.1 — The dynamics test

The dynamics test is the 3.4.1 §L.B.3 controlled experiment for the dynamics axis, with perception, actuation, and timing held.

The swap. The simulator’s contact-law parameters $(m, \mu, k_c, b_c)$ are set to the 3.4.3 §L.E.2 identified values, matching the real cell. The swap’s isolation holds perception, actuation, and timing at nominal.

The measurement. The twin-track’s state divergence is re-measured, and the difference from baseline is the dynamics contribution.

The confirmation. The contribution is the test result, and its confidence interval comes from the twin-track’s episode count — the 4.3.1 §L.C.1 power arithmetic at the divergence’s effect size.

The test’s confound. The identified values are intervals, and the point estimate is the interval’s center. Swapping to the point estimate measures the identification’s model error — the 3.4.3 §L.E.2 structure-error allowance — so the result is the swap’s effect, not the true parameters’ effect.

The test’s treatment. The 3.4.2 §L.B.1 range-from-uncertainty rule supplies the range, and the dynamics test’s swap is the range’s center. The full range’s test is the 3.4.2 §L.E.1 randomization, and its effect is robustness, not the mean.

§L.B.2 — The perception test

The perception test is the 3.4.1 §L.B.3 controlled experiment for the perception axis, with dynamics, actuation, and timing held.

The swap. The simulator’s rendered images are replaced by the real camera’s images, with dynamics, actuation, and timing held at their identified nominals.

The measurement. The twin-track’s state divergence is re-measured, and the difference is the perception contribution.

The confirmation. The contribution’s interval comes from the episode count, as in §L.B.1.

The test’s confound. The real camera’s images cover the same episodes, so the perception test measures the swap’s effect — rendered versus camera — not the camera’s absolute quality. The 3.4.4 §L.B.1 pixel translation and the 3.4.4 §L.D.1 MMD are treatments for the perception axis; the test’s swap is the diagnosis’s measurement.

The test’s treatment. The 3.4.4 §L.B.1 pixel-level translation is the treatment; the test’s swap is the measurement. They differ in role: the test measures the perception contribution, the treatment reduces it. The diagnosis’s order is test before treatment, and treatment before the next test.

§L.B.3 — The actuation and timing tests

The actuation test. The simulator’s actuator-model delay and gain are set to the 3.4.3 §L.B.4 step-response identified values, with the others held. The measurement is the divergence difference, and the confirmation is the contribution.

The timing test. The simulator’s control period and jitter are set to the real cell’s measured values, with the others held. Measurement and confirmation follow §L.B.1.

The isolation issue. Actuation and timing are the cheapest tests — a step response and an existing log — and their priors are the lowest, 15% and 5%. The 4.3.1 §L.C.1 expected information gain, §L.C.1’s subject, sequences them last.

The resolution issue, which the earlier draft omitted. A contribution is resolvable at $n$ episodes only if its confidence interval excludes zero:

with $\sigma$ the twin-track’s episode-level standard deviation and $c$ the contribution. With $\sigma = 2$ cm the perception axis at 1.3 cm resolves at $n \ge 19$ and the timing axis at 0.2 cm needs $n \ge 769$. At the lab’s 200 episodes the timing contribution is not measurable, and the correct treatment is to fold it into the residual rather than report a point estimate with an interval straddling zero. This is why §L.E.1’s fourth test returns no significant contribution: not because timing contributes nothing, but because 200 episodes cannot see 0.2 cm through 2 cm of noise.

§L.B.4 — The tests’ costs and the sequencing constraints

The four tests’ costs.

The dynamics test. 50 paired episodes at 25 s = 21 minutes. The 3.4.3 §L.E.1 identification data — a 40 s Fourier trajectory — is the precondition, costing 2 hours if not already collected.

The perception test. 21 minutes. The real camera’s images are an existing dataset; the cost is pairing only.

The actuation test. 21 minutes. A 10-minute step response is the precondition.

The timing test. 21 minutes. The measurement comes from existing logs.

The sequencing constraints.

Precondition order. The dynamics test requires the identification, which requires the excitation trajectory. The perception test requires the real images, which the 3.1.3 collection supplies. Actuation requires the step response; timing requires the logs.

Precondition conflict. The dynamics test needs identification data and the perception test needs real images — these are independent, so ordering follows information value rather than preconditions.

Test iteration. Each result updates the priors, and the next test is the highest posterior. This is the 4.3.1 §L.C.4 Bayesian update at the diagnosis’s level.

Self-explanation 2. Why is the perception test first despite the dynamics signature?

Answer. Because the perception prior is the highest at 45%, and a test’s expected information gain is driven by the prior times the likelihood ratio, so the highest prior goes first unless another test is much cheaper. The two costs are equal at 21 minutes, and perception’s prior wins.

Concept check. The perception test returns 5 points and the dynamics test 2 points. Compute the posterior after both and state the residual.

Answer. $5 + 2 = 7$ of the 12 points, leaving a 5-point residual. The next hypotheses — actuation and timing — are the next tests. Because 5 exceeds the 3-point tolerance, the diagnosis is not complete.

Worked-example ladder.

WE-L.B.a (fully worked). The perception swap changes the 95th-percentile divergence from 4.5 cm to 3.2 cm. Compute the contribution and its interval at 50 episodes, then at 200.

Solution. The contribution is $4.5 - 3.2 = 1.3$ cm. The paired contribution’s standard error is $\sigma\sqrt{2/n}$; with $\sigma = 2$ cm, at $n = 50$ that is $2\sqrt{2/50} = 0.400$ cm and the 95% half-width is $1.96 \times 0.400 = 0.784$ cm, giving $[0.52, 2.08]$ — excluding zero. At $n = 200$ the half-width falls to 0.392 cm and the interval is $[0.91, 1.69]$.

WE-L.B.b (completion). The twin-track’s episode-level sd is 2 cm and the perception contribution is 1.3 cm. Compute the minimum episode count for a significant result.

Completion. From Eq. (B.1), $n \gtrsim 2(1.96 \times 2/1.3)^2 = 2 \times 9.09 = 18.2$, so $n \ge 19$. The completion shows that the contribution sizes and the episode budget jointly decide which axes are measurable.

WE-L.B.c (faded). Write the dynamics test’s measurement and confirmation criterion.

Faded step. The measurement is the divergence’s difference from baseline; the confirmation criterion is that the contribution’s interval excludes zero and the residual falls within tolerance.

Completion. The confirmation criterion has two parts, and the residual part is the diagnosis’s stopping rule — §L.C.3’s — not the test’s.

WE-L.B.d (contrastive near-miss). A team runs the perception test with a retrained policy on the swapped images, and the contribution reflects the retraining rather than perception. Why it fails: the 4.3.1 §L.A.3 temporal condition is violated — the policy is not frozen across the test — so retraining is the confound. The fix: freeze the policy, per §L.A.3’s constraints.

Comparative table: the four tests.

Design principle P58.2. The discriminating test is the diagnosis’s evidence; its isolation requires the policy freeze and paired episodes, its contribution’s interval is the evidence, and its resolution has an episode floor.

Consolidation checkpoint 2

(a) Three capabilities. You can state the four tests’ swaps and their isolation. You can compute a contribution’s interval from the episode count and decide significance. You can state the tests’ preconditions and sequencing constraints.

(b) Transfer probe. The actuation test returns 1 point with an interval of $[0.1, 1.9]$ points at $n = 200$ and $\sigma = 2$. Is the contribution confirmed? Answer: at $\sigma = 2$ the 200-episode half-width is 0.392 cm, so an interval of $[0.1, 1.9]$ is wider than the noise floor allows — the interval is inconsistent with the stated episode count and sd, which is itself the finding. Recompute the interval from Eq. (B.1) before confirming anything.

(c) Remediation rule. If you cannot state why the policy freeze satisfies the temporal condition, re-read §L.B.4 before Part C.

Part C — The information’s value and the stopping rule

§L.C.1 — The expected information gain

The expected information gain (EIG) of a test is the reduction in the hypotheses’ entropy that its result is expected to produce, and it is the sequencing rule.

where $\mathcal{H}$ is the hypothesis set, $t$ the test, and $r$ its result. The first term is the prior entropy; the second is the expected posterior entropy.

The arithmetic. The priors from the 3.4.1 §L.B.2 ranking are 0.45, 0.35, 0.15, 0.05, with entropy

The likelihoods are the modelling input: for the perception test, “if perception is the dominant axis the result is a large contribution with probability 0.85; if dynamics is, 0.15; if actuation, 0.15; if timing, 0.15.” Writing $p_1$ for the hypothesis under test and $1-p_1$ for the rest, the posterior after a large result is $p_1' = 0.85p_1 / (0.85p_1 + 0.15(1-p_1))$.

The corrected values. The earlier draft reported 0.45 nats for the perception test, against a prior entropy of 1.168. Both numbers are wrong in a way worth naming: the entropy of these four priors is 1.1611, not 1.168, and the exact EIG is 0.2680 nats, not 0.45. The draft’s error was to substitute a two-hypothesis entropy, $H = -p\log p - (1-p)\log(1-p) = 0.688$ for $p = 0.45$, into a four-hypothesis posterior and to clip the posterior at $[0.01, 0.99]$. The EIG is a difference of entropies over the same hypothesis set, and mixing sets inflates it by about 68%.

The exact values.

A check worth keeping: the EIG of a test cannot exceed $H$. A binary test’s information is at most one bit about a binary question, and with four hypotheses no single binary outcome can resolve more than the prior entropy. A report of 0.45 against $H = 1.1611$ is arithmetically possible; a report of any value near the two-hypothesis entropy of 0.688 would not be, and that is the canary that catches this class of error.

The EIG’s role is the sequencing rule — highest first — with cost as the tiebreaker when EIGs are close. The perception and dynamics tests are the two highest, and their costs are equal, so perception goes first.

§L.C.2 — The cost trade and the test’s value

A test’s value is its EIG times the decision’s cost, where the decision’s cost is the loss from a wrong attribution.

where $C_{decision}$ is the stakes: the cost of the next treatment if the attribution is wrong.

The running case’s stakes. If perception is dominant and the diagnosis attributes it to dynamics, the treatment randomizes the wrong axis and the tail is unchanged. The cost is the treatment’s cost plus the delay — 2 hours of randomization plus 44 hours of suite re-run — about 46 hours.

The value comparison at $C_{decision} = 46$ h: perception $0.2680 \times 46 = 12.33$ h; dynamics $0.2482 \times 46 = 11.42$ h; actuation $6.67$ h; timing $2.58$ h. Perception is highest, and its cost is 21 minutes = 0.35 h, so its net is $11.98$ h.

The value’s role is sequencing and stopping. A test is run if its $V$ exceeds its cost; the sequence is $V$ descending; and the stopping criterion is the residual, below.

§L.C.3 — The stopping rule

The stopping rule is the criterion for ending the diagnosis’s tests, and its structure is residual within tolerance.

where $\hat{c}_i$ is axis $i$‘s estimated contribution, the gap is the outcome’s 12 points, and the tolerance is the deployment’s 3 points.

Interpretation. The residual is the gap minus the tested contributions; the rule stops when the residual is within tolerance. In the running case, perception’s 5 and dynamics’ 2 leave a residual of 5, above the 3-point tolerance, so the rule continues to actuation and timing.

The rule’s final state. After the four tests the residual is the unattributed part, and the 4.3.1 §L.D.3 attribution joint — main effect plus attribution — explains what can be explained. The residual is the residual-risk register’s entry, with the 4.3.2 §L.D.3 failure modes as its mitigation.

The rule’s relation to the registry. The stopping rule is the registry’s analysis field — the criterion stated before the tests — and that pre-registration is the rule’s falsifiability.

The rule’s failure mode. The rule stops too early if the residual is within tolerance but the attribution is wrong: contributions summing to the gap is a magnitude match, not a mechanism match. The 4.3.1 §L.D.1 mechanism condition — the test’s swap must change the mechanism’s variables — is the rule’s complement.

Self-explanation 3. Why is the stopping criterion the residual within tolerance rather than exhausting the tests?

Answer. Because exhausting the tests leaves the residual as the unknown, and the residual belongs in the residual-risk register. Residual within tolerance is the register’s acceptance; test exhaustion is only the budget’s limit, and a budget limit is not an acceptance.

Concept check. The four tests return 5, 2, 1, and 0.5 points against a 12-point gap. Is the stopping rule satisfied?

Answer. The sum is 8.5, leaving a residual of 3.5 — above the 3-point tolerance, so the rule continues. The register’s residual entry is 3.5 points, with the 4.3.2 §L.D.3 failure modes as the mitigation.

Worked-example ladder.

WE-L.C.a (fully worked). The perception EIG is 0.2680 nats and the decision stakes are 46 hours. Compute the test’s value and its net of cost.

Solution. The value is $0.2680 \times 46 = 12.33$ h. The cost is 21 minutes = 0.35 h. The net is $12.33 - 0.35 = 11.98$ h. The test is worth running.

WE-L.C.b (completion). The actuation EIG is 0.1451 nats and its cost is 31 minutes (21 + 10). Compute the net and the sequencing.

Completion. The value is $0.1451 \times 46 = 6.67$ h; the cost is 0.52 h; the net is 6.15 h. The sequence by net is perception 11.98, dynamics 9.07, actuation 6.15, timing 2.23 — so actuation and timing are last and their nets are small.

WE-L.C.c (faded). Derive the residual from the four tests’ contributions and state the stopping rule’s result.

Faded step. $5 + 2 + 1 + 0.5 = 8.5$; the residual is $12 - 8.5 = 3.5$. The rule is marginal against the 3-point tolerance.

Completion. The 3.5 residual is the register’s entry, and the 4.3.2 §L.D.3 failure modes — the unattributed — are the register’s mitigation.

WE-L.C.d (contrastive near-miss). A team runs all tests regardless of EIG, and the timing test’s net is 2.23 h — positive but small. Why it fails: a positive net means the test is worth running, so running it is not the error; the error is the order. Running the smallest-net test first spends budget before the largest information arrives. The fix: sequence by EIG, per §L.C.1.

Comparative table: the tests’ EIG and values.

Design principle P58.3. Sequence the tests by expected information gain; stop when the residual is within tolerance, and register the residual as residual risk.

Consolidation checkpoint 3

(a) Three capabilities. You can compute a test’s EIG from the priors and likelihoods. You can compute a test’s value and net of cost. You can apply the stopping rule and state the residual.

(b) Transfer probe. The perception test returns a negative result — its contribution’s interval includes zero. What does the posterior update do to the sequencing? Answer: perception’s posterior falls and the dynamics posterior rises to dominant, so the next test is dynamics. A negative result is informative: it is exactly the evidence that redirects the diagnosis.

(c) Remediation rule. If you cannot state why the value is EIG times the stakes, re-read §L.C.2 before Part D.

Part D — The confounds and the four failure modes

§L.D.1 — The three confounds

The diagnosis’s confounds are the 4.3.1 §L.D.2 set — implementation drift, selection, and mechanism compensation — raised to the sim-to-real level.

Implementation drift. The sim version or the camera calibration changes between tests, so the divergence difference reflects the change, not the test. The detection channel is the sim version log and the camera calibration log; the fix is version freeze and periodic calibration re-check.

Selection. The treatment and control episodes differ, so the difference reflects the episodes’ distribution. The detection channel is episode metadata; the fix is the paired design — the same episodes in both conditions.

Mechanism compensation. The test’s swap is compensated by another subsystem — the reasoning layer re-queries after the perception swap — so the difference reflects the compensation, not the test. The detection channel is the mediator’s rate, the re-queries; the fix is specifying the mechanism in the registry and including the mediator in the analysis.

All three are the 4.3.1 §L.D.2 confounds at the sim-to-real level, and the channels are the 4.3.1 §L.D.1 four plus the sim version.

§L.D.2 — The policy-sim pair’s confounding

The policy-sim pair’s confounding is the diagnosis’s specific confound, and its structure is the 3.4.1 §L.A.1 pair.

The confound. The policy is trained on the sim, so the sim is the policy’s training environment. Sim error and policy error are entangled: the policy is a response to the sim, and the policy’s failure belongs to the pair, not to the sim alone.

The evidence. The same sim with a different policy — the 3.4.2 §L.E.1 randomized policy versus the 3.4.4 §L.E.1 adapted policy — gives a different divergence. The difference is the policy’s contribution; the sim’s efficiency alone is the pair’s limit.

The detection. The 3.4.1 §L.C.2 twin-track is run with two policies on the same sim, and the divergence difference is the policy’s contribution. This policy swap is the diagnosis’s fifth test, and its EIG is the policy hypothesis’s.

The fix. The pair’s test targets the policy hypothesis — policy robustness — and the treatment is the 3.4.2 §L.E.1 randomization range or the 4.3.1 §L.E.3 residual RL.

The importance. The 3.4.1 §L.A.1 single-source ceiling argument — the policy is the sim’s partner — is the diagnosis’s central caution: the sim alone is not the gap’s source.

§L.D.3 — The four failure modes

The diagnosis’s failure modes are the ways it goes wrong, each with a signature.

Wrong attribution. The diagnosis attributes the gap to the wrong axis, and the treatment targets that axis. The signature is the post-treatment divergence unchanged — treating the wrong axis does not reduce it. The fix is the 4.3.1 §L.D.1 mechanism condition — the swap must change the mechanism’s variables — and a re-run.

Under-attribution. The diagnosis attributes too little to the right axis — the contribution is underestimated by compensation or by a too-wide interval — so the treatment is insufficient. The signature is the post-treatment divergence partially reduced. The fix is the 4.3.1 §L.C.4 interval and including the compensation.

Over-attribution. The diagnosis attributes too much to the right axis — overestimated by selection or drift — so the treatment is partly wasted. The signature is the post-treatment divergence reduced, but the register’s residual larger than predicted.

Unattributed residual. The diagnosis stops with the residual above tolerance, and the unattributed part becomes the register’s residual. The signature is the register’s residual dominated by the unknown axis. The fix is the residual’s analysis — the pair’s test, or the stopping rule’s re-examination.

The four are the diagnosis’s modes, and the register records them.

Self-explanation 4. Why is wrong attribution the costliest failure?

Answer. Because the treatment targets the wrong axis and the post-treatment divergence is unchanged, so the diagnosis is re-run from the start. The cumulative cost is the wrong treatment plus the re-diagnosis plus the correct treatment.

Concept check. The perception test is run and the post-treatment divergence is unchanged. Which failure mode?

Answer. Wrong attribution — perception is not the dominant axis, so the treatment is wasted. The fix is the dynamics test, per the 35% prior, and a diagnosis re-run.

Worked-example ladder.

WE-L.D.a (fully worked). The perception test returns 5 points and the post-perception treatment reduces the divergence by 1 point. Compute the treatment’s effectiveness and state the failure mode.

Solution. The treatment’s effect is 1 point of 5 — 20% — and the register’s residual exceeds the prediction. The mode is under-attribution: perception’s contribution was underestimated because the test’s compensation was omitted. The fix is to include the compensation in the analysis.

WE-L.D.b (completion). The dynamics test returns 2 points and the post-dynamics treatment leaves the divergence unchanged. Compute the failure mode.

Completion. The test returned 2 points but the treatment changed nothing, so the mode is wrong attribution — dynamics’ contribution was overestimated by selection or drift. The fix is the pair’s test and a re-run.

WE-L.D.c (faded). Write the pair test’s design and its EIG.

Faded step. Two policies — the 3.4.2 randomized and the 3.4.4 adapted — on the same sim and episodes, with the divergence difference as the policy’s contribution. The EIG is the policy hypothesis’s prior.

Completion. The pair test’s EIG is moderate — the policy hypothesis’s prior is about 0.10 — and its cost is 21 minutes. The pair test comes last, after the four axes.

WE-L.D.d (contrastive near-miss). A diagnosis stops at perception’s 5 points and dynamics’ 2 points, leaving 5 unattributed. The treatment is randomization and the real cell’s success rises to 0.85 — above 0.82 but below 0.94. Why it fails: the 5-point residual is the policy’s contribution — the pair confound — and the 3.4.4 adaptation treats perception, not the pair. The fix: the pair’s test, per §L.D.2.

Comparative table: the four failure modes.

Design principle P58.4. The policy-sim pair is the diagnosis’s specific confound; the pair’s test is the fifth test, and the four failure modes are the register’s residual.

Consolidation checkpoint 4

(a) Three capabilities. You can name the three confounds and their channels. You can state the pair confound’s evidence and fix. You can diagnose the four failure modes’ signatures.

(b) Transfer probe. The perception test returns 5 points and the post-treatment divergence falls by 4 points. Which mode, and what is the fix? Answer: the treatment achieved 4 of 5, and the register’s residual is smaller than the prediction, so this is a well-attributed axis. The fix, if any, is to include the compensation and report the 4.3.1 §L.C.4 interval.

(c) Remediation rule. If you cannot state why the pair test is the fifth, re-read §L.D.2 before Part E.

Part E — The running case

§L.E.1 — The diagnosis’s evidence chain

The running case’s diagnosis, symptom to confirmation.

The symptom. Sim 0.94, real 0.82. The twin-track’s state divergence is 1.2 cm mean and 4.5 cm at the 95th percentile, with the exponent steepening from 0.51 over $t=1$ to $t=15$ to 1.78 over $t=15$ to $t=20$.

The first test, perception. Swapping the real camera’s images for rendered, with dynamics, actuation, and timing at the 3.4.3 §L.E.2 identified nominals. Result: the 95th percentile falls from 4.5 cm to 3.2 cm, a 1.3 cm contribution with a 200-episode interval of $[0.91, 1.69]$ at $\sigma = 2$ cm. Significant.

The second test, dynamics. Swapping the identified contact parameters, others held. Result: the 95th falls from 3.2 cm to 2.1 cm, a 1.1 cm contribution with interval $[0.71, 1.49]$. Significant.

The third test, actuation. Swapping the identified delay and gain. Result: the 95th falls from 2.1 cm to 1.6 cm, a 0.5 cm contribution with interval $[0.11, 0.89]$. Significant, and only just — §L.B.3’s Eq. (B.1) gives $n \ge 123$ for a 0.5 cm contribution, so 200 episodes barely resolves it.

The fourth test, timing. Swapping the measured period and jitter. Result: the 95th falls from 1.6 cm to 1.4 cm, a 0.2 cm contribution with interval $[-0.19, 0.59]$. Not significant, and Eq. (B.1) says why: a 0.2 cm contribution needs $n \ge 769$ episodes, so 200 episodes cannot resolve it. The timing contribution is folded into the residual — not because timing contributes nothing, but because the schedule cannot see it.

The stopping rule. The significant contributions sum to 2.9 cm of 4.5, leaving a residual of 1.6 cm — within the 3 cm tolerance, so the rule stops. The residual is 1.6 rather than 1.4 because the timing axis’s 0.2 cm is unmeasurable at this budget and therefore remains unattributed.

§L.E.2 — The pair test

The pair test is the diagnosis’s fifth, and its result is the policy’s contribution.

The design. The 3.4.2 §L.E.1 randomized policy and the 3.4.4 §L.E.1 adapted policy run on the same sim and the same episodes, and the divergence difference is the policy’s contribution.

The result. The randomized policy’s 95th is 4.8 cm and the adapted’s is 4.5 cm — a 0.3 cm difference — with interval $[-0.09, 0.69]$ at $\sigma = 2$. The policy’s contribution is not significant, and the sim’s dominates.

The interpretation. The policy’s 0.3 cm is small relative to the sim’s 2.9 cm, and the 3.4.1 §L.A.1 pair argument is confirmed: the sim alone is the gap’s dominant source, and the policy is a small correction. Note that the interval is consistent with either a zero policy effect or a small one; the honest statement is the bound, not a point estimate.

§L.E.3 — The register and the treatment plan

The register’s entries follow the residual-risk register’s five fields, and the residuals are the diagnosis’s residual statement.

The treatment plan. Perception gets the 3.4.4 §L.E.1 two-stage mapping; dynamics gets the 3.4.3 §L.E.1 identification point estimate and the 3.4.2 §L.E.1 randomization over the interval; actuation gets the 3.4.3 §L.B.4 step-response identification; timing gets the 3.4.1 §L.B.2 measurement. Treatments are sequenced by contribution magnitude — perception first, dynamics second — and the residual is the register’s unknown.

The register’s residual. The 1.6 cm of 4.5 is the unattributed part, and the register records it. The residual is within the 3 cm tolerance, and the diagnosis is accepted with the residual as residual risk.

Self-explanation 5. Why is the 1.6 cm residual accepted despite the register’s unknown axis?

Answer. Because 1.6 cm is within the 3 cm tolerance, and acceptance is the tolerance, not zero. The register’s entry is residual risk, and the operator accepts the residual — not the completeness of the mitigation.

Worked-example ladder.

WE-L.E.a (fully worked). The significant contributions sum to 2.9 cm and the residual is 1.6 cm. Compute the stopping rule’s result and the register entry.

Solution. The residual is $4.5 - 2.9 = 1.6$ cm, within the 3 cm tolerance. The stopping rule is satisfied, and the register’s entry is the 1.6 cm residual against the unknown axis.

WE-L.E.b (completion). The perception contribution falls to 0.8 cm after the 3.4.4 adaptation with the others unchanged. Recompute the residual and the stopping rule.

Completion. The measurable contributions are now $0.8 + 1.1 + 0.5 = 2.4$ cm, so the residual is $4.5 - 2.4 = 2.1$ cm — still within the 3 cm tolerance. The residual rises because the adaptation removes perception’s measured share without attributing it elsewhere.

WE-L.E.c (faded). Write the register entry for the pair’s contribution with the policy-coverage residual.

Faded step. Contribution 0.3 cm; treatment the 3.4.2 §L.E.1 randomization; residual the policy coverage.

Completion. The residual is the 3.4.2 §L.D.3 robustness boundary — the policy’s coverage of the randomization range — and the 4.3.3 §L.D long-horizon robustness is its complement.

WE-L.E.d (contrastive near-miss). A team stops at perception’s 1.3 cm and dynamics’ 1.1 cm — a 2.4 cm sum, within the 3 cm tolerance — and omits the actuation and timing tests. Why it fails: the 2.4 cm is within tolerance but the residual includes the actuation’s 0.5 and the timing’s 0.2, so the residual is not the unattributed part; the register’s entry would be the untested axes, not the unknown. The fix: run all four tests before applying the stopping rule, per §L.E.1.

Comparative table: the diagnosis’s evidence chain.

Design principle P58.5. Diagnose each axis’s contribution by the test’s interval; stop at the residual within tolerance; register the residual as the unknown; and check the episode floor before claiming a contribution, because an unresolvable axis belongs in the residual, not in a point estimate.

Consolidation checkpoint 5

(a) Three capabilities. You can assemble the evidence chain with intervals. You can apply the stopping rule and state the residual. You can write the register entries with residuals.

(b) Transfer probe. The residual is 2.9 cm and the tolerance 3 cm. Is the rule satisfied, and what is the register entry? Answer: yes, 2.9 is within 3. The register entry is the 2.9 cm residual, and the operator accepts it.

(c) Remediation rule. If you cannot state why the residual is within tolerance rather than zero, re-read §L.E.3 before the lab.

Failure taxonomy

The fifth row is the one this lesson’s own lab exposed: a 0.2 cm contribution at $\sigma = 2$ cm cannot be resolved in 200 episodes, and reporting it with an interval straddling zero is not a modest result — it is an unmeasured one.

The lab

Task. Implement the diagnosis’s four tests and the pair test on a synthetic twin-track with known ground-truth axis contributions — perception 1.3 cm, dynamics 1.1 cm, actuation 0.5 cm, timing 0.2 cm, pair 0.3 cm, unknown 1.4 cm — and compute each test’s contribution, interval, EIG, and the stopping rule’s result.

Conditions. Python 3.10+, NumPy. The twin-track’s state divergence is simulated with the axis contributions and episode noise at $\sigma = 2$ cm; each test’s swap reduces the divergence by that axis’s contribution. 200 episodes per test. Seed all RNGs from an explicit integer, never from hash(), whose value varies between processes.

Controls. (i) The full chain; (ii) the chain with tests in reverse order; (iii) the chain with the pair test omitted; (iv) the chain with the stopping rule disabled.

Metrics.

Pre-registered hypotheses.

H1. The full chain’s contributions are within 0.3 cm of truth for the axes above the episode floor, because the tests are the 3.4.1 §L.B.3 controlled experiments.

H2. Reversing the order gives the same contributions and the same intervals — order affects the cost, not the measurement.

H3. Omitting the pair test leaves the policy’s 0.3 cm inside the residual, raising it to about 1.9 cm — still within tolerance, but the register’s entry is then the pair’s contribution rather than the unknown’s.

H4. Disabling the stopping rule runs all five tests, costing 84 + 21 = 105 minutes, exceeding the four-test budget — but since the rule stops at the third significant test in the running case, the saving is one test’s 21 minutes, not the whole remainder. The hypothesis as the earlier draft stated it assumed all four tests were needed; the lab shows only three contribute significantly.

H5. The timing axis at 0.2 cm returns a non-significant interval at 200 episodes, and the reason is the episode floor of Eq. (B.1) — $n \ge 769$ — not a zero contribution. Reporting it as a point estimate is the failure this hypothesis tests for.

Break-it sequence.

1. Run the tests with the policy retrained, silently. The contributions reflect the retraining, not the axes. Silent: the intervals look valid.
2. Stop at the first two tests, silently. The residual 2.1 cm is within the 3 cm tolerance, but the register then holds the untested axes. Silent: the stopping rule is technically satisfied.
3. Skip the pair test, silently. The policy’s 0.3 cm joins the unattributed residual and the treatment targets the wrong source. Silent: the residual stays within tolerance.
4. Report the timing axis as a point estimate. Its 0.2 cm enters the sum and the residual falls to 1.4 cm, which looks like a better diagnosis and is a worse one: the contribution is below the resolvable floor and the sum is now partly noise. Silent: every number is computed correctly.

Acceptance criteria.

What good looks like. A good artefact has six properties. (1) Each test is a separate function with its own log. (2) Each interval is computed per test and significance is the interval. (3) EIG is the sequencing rule. (4) The stopping rule is the criterion and the residual is the register’s. (5) Every reported contribution is checked against the episode floor. (6) The break-it results are a table.

A runnable core of the EIG sequencing, the tests with intervals, the episode floor, and the stopping rule:

# lab_4_3_2.py — sim-to-real diagnosis: EIG sequencing, tests with CIs, stopping rule
import numpy as np
from math import log, ceil

# ---- the running case ---------------------------------------------------
# Axis contributions to the 95th-percentile divergence at t = 15, in cm.
TRUTH = {"perception": 1.3, "dynamics": 1.1, "actuation": 0.5, "timing": 0.2}
UNKNOWN = 1.4
BASELINE = sum(TRUTH.values()) + UNKNOWN          # 4.5 cm
TOLERANCE = 3.0                                   # cm
SIGMA = 2.0             # cm; the twin-track's measured episode-level sd of the divergence
PRIORS = {"perception": 0.45, "dynamics": 0.35, "actuation": 0.15, "timing": 0.05}
$P_HIT$, $P_FALSE$ = 0.85, 0.15                       # likelihoods of a large contribution
$C_DECISION$ = 46.0                                 # hours; a wrong attribution costs treatment + delay
COST_H = {"perception": 0.35, "dynamics": 0.35 + 2.0, "actuation": 0.35 + 0.17, "timing": 0.35}
PAIR_DIFF = 0.3                                   # cm; randomization policy vs adapted policy

def entropy(d):
    return -sum(v * log(v) for v in d.values() if v > 0)

def eig(axis, priors, $p_hit=P_HIT$, $p_false=P_FALSE$):
    """Exact expected information gain: H(prior) - E[H(posterior)] over the binary result.

    The earlier draft capped the posterior at [0.01, 0.99] and substituted a
    two-hypothesis entropy into a four-hypothesis posterior, which overstated
    every value by about 68%.
    """
    h0 = entropy(priors)
    exp_post = 0.0
    for hit in (True, False):
        lik = {}
        for a in priors:
            if a == axis:
                lik[a] = $p_hit$ if hit else 1.0 - $p_hit$
            else:
                lik[a] = $p_false$ if hit else 1.0 - $p_false$
        $p_out$ = sum(priors[a] * lik[a] for a in priors)
        if $p_out$ <= 0:
            continue
        post = {a: priors[a] * lik[a] / $p_out$ for a in priors}
        exp_post += $p_out$ * entropy(post)
    return h0 - exp_post

def half_width(n, sigma=SIGMA, z=1.96):
    """95% half-width of a paired contribution: se = sigma*sqrt(2)/sqrt(n)."""
    return z * sigma * np.sqrt(2.0 / n)

def $n_to_resolve(c$, sigma=SIGMA, z=1.96):
    """Episodes needed for a contribution c to have a CI excluding zero."""
    return ceil(2.0 * (z * sigma / c) ** 2)

def run_test(axis, n=200, seed=0):
    """The swap's effect: point estimate and CI from paired episodes."""
    rng = np.random.default_rng(1000 + seed)
    est = TRUTH[axis] + rng.normal(0.0, SIGMA * np.sqrt(2.0 / n))
    hw = half_width(n)
    return float(est), (est - hw, est + hw)

# ---- canaries -----------------------------------------------------------
print("== canaries ==")
print("C1 uninformative test (0.50/0.50) must give 0 : %.3e"
      % max(abs(eig(a, PRIORS, 0.50, 0.50)) for a in PRIORS))
perfect = {a: eig(a, PRIORS, 1.0, 0.0) for a in PRIORS}
print("C2 perfect test must give H(prior) = %.4f      : max %.4f  min %.4f"
      % (entropy(PRIORS), max(perfect.values()), min(perfect.values())))
print("C3 EIG never exceeds H(prior)                 : %s"
      % all(eig(a, PRIORS) <= entropy(PRIORS) for a in PRIORS))
print("C4 baseline is the contributions' sum         : %.6f"
      % (BASELINE - (sum(TRUTH.values()) + UNKNOWN)))
print("C5 half-width shrinks as 1/sqrt(n) (200/50)   : %.4f  (expected %.4f)"
      % (half_width(50) / half_width(200), np.sqrt(200 / 50)))

# ---- EIG sequencing -----------------------------------------------------
h0 = entropy(PRIORS)
print("\n== EIG sequencing (H(prior) = %.4f nats) ==" % h0)
print("%-11s %6s %8s %7s %8s %8s" % ("axis", "prior", "EIG", "%H0", "value_h", "net_h"))
order = sorted(PRIORS, key=lambda a: -eig(a, PRIORS))
for a in order:
    e = eig(a, PRIORS)
    v = e * $C_DECISION$
    print("%-11s %6.2f %8.4f %6.1f%% %8.2f %8.2f"
          % (a, PRIORS[a], e, 100 * e / h0, v, v - COST_H[a]))

# ---- the tests in EIG order --------------------------------------------
print("\n== tests, in EIG order, n = 200, half-width %.3f cm ==" % half_width(200))
cum = 0.0
for a in order:
    est, ci = run_test(a, n=200)
    print("  %-11s est %.3f  CI [%+.3f, %+.3f]  %s"
          % (a, est, ci[0], ci[1], "significant" if ci[0] > 0 else "not significant"))
    cum += est
print("  cumulative %.3f cm  (truth %.3f)" % (cum, sum(TRUTH.values())))

residual = BASELINE - cum
print("\n== stopping rule ==")
print("  residual %.3f cm  tolerance %.1f cm  -> %s"
      % (residual, TOLERANCE, "stop" if residual <= TOLERANCE else "continue"))

# ---- what resolves, and what it costs ----------------------------------
print("\n== resolution: episodes needed for a CI excluding zero ==")
for a in sorted(TRUTH, key=lambda a: TRUTH[a]):
    print("  %-11s contribution %.1f cm -> n >= %5d" % (a, TRUTH[a], $n_to_resolve(TRUTH[a$])))

# ---- the pair test ------------------------------------------------------
print("\n== pair test (policy contribution) ==")
rng = np.random.default_rng(7)
pair_est = PAIR_DIFF + rng.normal(0.0, SIGMA * np.sqrt(2.0 / 200))
$p_hw$ = half_width(200)
print("  est %.3f  CI [%+.3f, %+.3f]  %s"
      % (pair_est, pair_est - $p_hw$, pair_est + $p_hw$,
         "significant" if pair_est - $p_hw$ > 0 else "not significant"))

# ---- the same test at n = 50 and n = 200 -------------------------------
print("\n== the actuation test at two sizes ==")
for n in (50, 200):
    est, ci = run_test("actuation", n=n, seed=3)
    print("  n = %3d  half-width %.3f  est %.3f  CI [%+.3f, %+.3f]  %s"
          % (n, half_width(n), est, ci[0], ci[1],
             "significant" if ci[0] > 0 else "not significant"))

Measured output:

== canaries ==
C1 uninformative test (0.50/0.50) must give 0 : 0.000e+00
C2 perfect test must give H(prior) = 1.1611      : max 0.6881  min 0.1985
C3 EIG never exceeds H(prior)                 : True
C4 baseline is the contributions' sum         : 0.000000
C5 half-width shrinks as 1/sqrt(n) (200/50)   : 2.0000  (expected 2.0000)

== EIG sequencing (H(prior) = 1.1611 nats) ==
axis         prior      EIG     %H0  value_h    net_h
perception    0.45   0.2680   23.1%    12.33    11.98
dynamics      0.35   0.2482   21.4%    11.42     9.07
actuation     0.15   0.1451   12.5%     6.67     6.15
timing        0.05   0.0562    4.8%     2.58     2.23

== tests, in EIG order, n = 200, half-width 0.392 cm ==
  perception  est 1.236  CI [+0.844, +1.628]  significant
  dynamics    est 1.036  CI [+0.644, +1.428]  significant
  actuation   est 0.436  CI [+0.044, +0.828]  significant
  timing      est 0.136  CI [-0.256, +0.528]  not significant
  cumulative 2.843 cm  (truth 3.100)

== stopping rule ==
  residual 1.657 cm  tolerance 3.0 cm  -> stop

== resolution: episodes needed for a CI excluding zero ==
  timing      contribution 0.2 cm -> n >=   769
  actuation   contribution 0.5 cm -> n >=   123
  dynamics    contribution 1.1 cm -> n >=    26
  perception  contribution 1.3 cm -> n >=    19

== pair test (policy contribution) ==
  est 0.300  CI [-0.092, +0.692]  not significant

== the actuation test at two sizes ==
  n =  50  half-width 0.784  est 0.091  CI [-0.693, +0.875]  not significant
  n = 200  half-width 0.392  est 0.296  CI [-0.096, +0.688]  not significant

Reading the result. Three things the lab settles that prose did not.

First, the EIG values. The exact expectation over the four-hypothesis posterior gives perception 0.2680 nats — 23.1% of the prior entropy — not the 0.45 the earlier draft reported. The canary that catches this is C3: a binary test on four hypotheses cannot extract more than the prior entropy, and a figure derived from a two-hypothesis entropy will look plausible while silently substituting a different probability space.

Second, the episode floor. Equation (B.1) says a 0.2 cm contribution at $\sigma = 2$ cm needs 769 episodes to resolve at 200’s noise level, and the lab returns a non-significant interval for timing at every seed it is tried. The running case’s fourth test is therefore inconclusive, not negative, and the residual correctly absorbs it.

Third, the actuation contrast at two sizes shows the floor in action: at $n = 50$ the same 0.5 cm contribution cannot be separated from zero, and at $n = 200$ it clears by a margin of 0.044 cm. A contribution’s resolvability is a property of $\sigma$, $c$, and $n$ together — and a diagnosis that reports $c$ without checking that product is reporting a number it cannot see.

Cheat sheet

Exercises with solutions

E 4.3.2.1. Classify the twin-track’s state divergence signature when it grows super-linearly.

S 4.3.2.1. The dynamics mismatch: growth that steepens with trajectory length is the feedback signature. The lesson: the signature ranks the hypotheses; the discriminating test is the evidence.

E 4.3.2.2 (transfer). Compute the EIG for the perception test with prior 0.45 and likelihoods 0.85/0.15 against the four-hypothesis prior set.

S 4.3.2.2. The prior entropy is 1.1611 nats; the expected posterior entropy is 0.8931; the EIG is 0.2680 nats — 23.1% of the prior entropy. The lesson: the EIG is a difference over the same hypothesis set; using a two-hypothesis entropy for a four-hypothesis posterior overstates it by about 68%.

E 4.3.2.3. State the three confounds and their channels.

S 4.3.2.3. Drift (the version log), selection (episode metadata), compensation (the mediator’s rate). The lesson: the confounds are the 4.3.1 §L.D.2 set, and the channels are the detection’s.

E 4.3.2.4 (diagnostic). The perception test returns 1.3 cm with an interval of $[-0.1, 2.7]$ at 200 episodes and $\sigma = 2$. Is the contribution significant?

S 4.3.2.4. No — the interval includes zero. But the interval is also wider than the 200-episode noise floor of 0.392 cm allows, so the interval is inconsistent with the stated $\sigma$ and $n$: the real finding is a mis-stated interval, not a marginal contribution. The lesson: check the interval against Eq. (B.1) before interpreting it.

E 4.3.2.5. State the stopping rule and apply it to contributions 5, 2, 1, and 0.5 against a 12-point gap and a 3-point tolerance.

S 4.3.2.5. The residual is $12 - 8.5 = 3.5$, above 3, so the rule continues. The register’s entry is the 3.5. The lesson: the residual is the register’s entry; the tolerance is the acceptance.

E 4.3.2.6 (transfer). The pair test is omitted and the policy’s 0.3 cm becomes part of the residual. What is the residual and the register entry?

S 4.3.2.6. The residual rises from 1.6 to about 1.9 cm, and the register’s entry is the policy pair’s contribution, not the unknown’s. The lesson: the pair test distinguishes the policy’s contribution from the unknown’s.

E 4.3.2.7 (diagnostic). The post-treatment divergence is unchanged and the treatment targeted perception. Which failure mode?

S 4.3.2.7. Wrong attribution: perception is not dominant, so the treatment is wasted. The fix is the dynamics test and a re-run. The lesson: post-treatment unchanged is the wrong-attribution signature.

E 4.3.2.8. Write the register entry for perception’s contribution with the mapping-drift residual.

S 4.3.2.8. Failure: the perception contribution; probability: moderate; consequence: a 1.3 cm divergence; mitigation: the 3.4.4 adaptation; residual: the mapping drift. The lesson: the register entry is the evidence chain’s record, and the residual is the monitor’s.

E 4.3.2.9 (transfer). The stopping rule stops at a 2.1 cm residual, and the untested axes are actuation and timing. Is the diagnosis complete?

S 4.3.2.9. No: the 2.1 cm includes the untested contributions, so the register’s entry is the untested axes, not the unknown. The lesson: the residual is the unknown’s only after every axis has been tested.

E 4.3.2.10 (diagnostic). The timing test returns 0.2 cm with interval $[-0.19, 0.59]$ at 200 episodes and $\sigma = 2$ cm. Should the 0.2 be added to the sum?

S 4.3.2.10. No. Eq. (B.1) gives $n \ge 769$ episodes for a 0.2 cm contribution at that noise, so 200 episodes cannot resolve it. Adding the 0.2 lowers the residual to 1.4 cm and makes the diagnosis look better while mixing noise into the sum. The contribution belongs in the residual as unresolved. The lesson: a contribution below the episode floor is unmeasured, not small.

Exit test

You are ready for Lesson 4.3.3 when you can, without notes:

1. Classify the twin-track’s divergence by the mechanism it implicates, using the exponent.
2. Write the four hypotheses with their priors and signatures.
3. State the diagnosis’s four steps and its cost trade.
4. State the four tests’ swaps and their isolation.
5. Compute a test’s contribution interval from the episode count and decide significance.
6. Compute the episode floor and say which axes are measurable at a given budget.
7. Compute the EIG and sequence the tests by it.
8. Compute a test’s value and its net of cost.
9. Apply the stopping rule and state the residual.
10. Name the three confounds and their channels.
11. State the policy-sim pair’s confounding and its detection.
12. Diagnose the four failure modes’ signatures.
13. Assemble the running case’s evidence chain with intervals.
14. Write the register’s entries with residuals.
15. State why the pair test is the fifth, and when it is needed.
16. State why an unresolvable contribution belongs in the residual, not in the sum.

Cross-references. 4.3.1 supplies the registry’s six fields, causal inference’s three conditions, and the confound’s three sources, restated in §L.A.3 and §L.D.1. 3.4.1 supplies the four axes, the twin-track’s tail, and the axis decomposition, restated in §L.A.2 and §L.B.4. 3.4.2 supplies the range-from-uncertainty rule restated in §L.B.1 and §L.E.3. 3.4.3 supplies the identification interval and its structure-error allowance restated in §L.B.2 and §L.D.2. 3.4.4 supplies the adaptation loop and the MMD restated in §L.B.2 and §L.E.3. 4.1.3 supplies the held-out discipline and the suite restated in §L.E.1. 4.2.4 supplies the fault-injection metrics restated in §L.D.2. Lesson 4.3.3 uses the stopping rule’s residual to size its window. Module 4.4 uses the evidence chain as the unknown-unknowns artefact.

§L.10 — Provenance

Corrections to the previous draft of this lesson.

1. The shipped lab did not run. sorted(PRIORS.items(), key=lambda kv: -eig(axis, ...)) referenced axis, a name bound only in the enclosing generator’s scope, and raised NameError before printing a single result. The lesson’s central quantitative claim — the EIG ordering — was therefore never computed by the code that shipped with it.

2. Every EIG value was overstated by about 68%. The draft’s eig computed the expected posterior entropy over a two-hypothesis posterior, $H = -p\log p - (1-p)\log(1-p) = 0.688$ at $p=0.45$, while the prior entropy was taken over four hypotheses. The difference of the two is not an information gain. It also clipped posteriors into $[0.01, 0.99]$, which caps the value from above. The exact values are 0.2680, 0.2482, 0.1451, and 0.0562 nats; the prior entropy is 1.1611, not the draft’s 1.168.

3. TRUTH summed to 4.8 cm against a stated 4.5 cm baseline. The dict included "pair": 0.3 alongside the four axes and the unknown, so BASELINE = sum(TRUTH.values()) was 4.8 and the residual was computed against the wrong total. The pair contribution is separate from the axis contributions: 3.1 cm of axes plus 1.4 cm unknown is 4.5.

4. The intervals used a half-width of 2.5 cm where 0.784 cm belongs. The draft’s worked example computed the contribution’s standard error as $0.90\sqrt{2}=1.27$ and then reported a half of 2.5 — neither $1.96 \times 1.27 = 2.49$ for a 95% interval against an episode sd of 4.5, nor consistent with the running case’s own intervals, which imply $\sigma \approx 2$ cm. The lesson now states $\sigma = 2$ cm once and derives every interval from it: $1.96 \times 2\sqrt{2/50} = 0.784$ cm at 50 episodes and 0.392 cm at 200.

5. The reported intervals in §L.E.1 were inconsistent with the stated episode count. Intervals of $[0.05, 2.55]$ at 200 episodes for a 1.3 cm contribution imply an episode sd near 6 cm. Every interval is now recomputed from $\sigma = 2$ and $n = 200$ as $z\sigma\sqrt{2/n} = 0.392$ cm.

6. hash(axis) seeded the RNG. Python randomises string hashing per process, so np.random.default_rng(seed + hash(axis) % 1000) gave a different stream on every run. The lab now uses an explicit integer per axis.

7. The timing axis’s non-significance had no stated cause, and H4 stated the wrong saving. The draft reported the timing interval straddling zero without saying why. Eq. (B.1) supplies the reason — a 0.2 cm contribution needs 769 episodes at $\sigma = 2$ cm — and the lesson now carries the episode floor as a first-class result. H4 claimed disabling the stopping rule costs the whole 105-minute remainder; in fact only three axes clear significance, so the rule saves one test.