Lesson 4.1.3 — Evaluation Design and Held-Out Test Suites

Week 33

Prerequisite lessons: 4.1.1 (defining the capstone), 4.1.2 (choosing the embodiment and the evaluation), 3.1.3 (data collection and splits), 3.1.4 (VLA fine-tuning), 3.4.1 (the reality gap as a spectrum), 3.4.4 (real-to-sim adaptation).

Prior results used in this lesson. From 4.1.1: the capstone’s six metrics, the four axes, the three preconditions of distribution, freeze, and baseline, and the risk register. From 4.1.2: the fifth axis, the frozen design, the test set’s size as the maximum of the metrics’ decisions’ resolutions, and the allocation’s weighting. From 3.1.3: the episode-level split’s discipline, the stratification by task, object, operator, and batch, and the effective sample size as the episode count. From 3.1.4: the lower-bound acceptance and the four-axis audit. From 3.4.1: the twin-track’s tail and the coverage ratio. From 3.4.4: the adaptation loop’s monitor and the held-out’s alignment’s statistic. Each is restated at the point of use.

What this lesson hands forward. The stratification’s cells, the episode’s selection rule, the leakage’s audit, the suite’s versioning, and the drift’s detection. Module 4.2 builds the subsystems on the chosen embodiment and this suite. Module 4.3 iterates against the suite’s metrics and uses the drift’s detection as its experiment’s signal. Module 4.4 defends the result on the suite’s evidence.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, divider, and three distractor objects are unchanged; the frozen design from 4.1.2 names five axes and 400 episodes, and this lesson builds the 400-episode suite’s strata, selection rule, and hygiene so that the capstone’s acceptance is evidence.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) define a stratum as a cross-product of the axes’ levels and state its role in the acceptance; (2) enumerate the running case’s strata and count them; (3) state the episode’s selection rule — random within stratum, deterministic across the suite — and its two properties; (4) assign the 400 episodes to strata proportionally to the claim’s emphasis and check the per-stratum resolution; (5) name the four leakage channels and the audit for each; (6) state the episode-level split’s discipline and why frame-level, task-level, and time-level splits leak; (7) design the suite’s version’s manifest and the freeze’s timestamp; (8) state the drift’s detection’s statistic and threshold; (9) assemble the running case’s 400-episode suite with its allocation, audit, and manifest; (10) state why a suite without a manifest is not evidence.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 4.1.2, the test set’s size was the maximum of the metrics’ decisions’ resolutions, and the axes’ allocation was proportional to the claim’s emphasis. What does the allocation’s proportionality protect against?

Answer. It protects against the resolution’s gap on the axis the claim emphasises; a uniform allocation spends the same episodes on the least informative axis as on the key axis, and the key axis’s interval is wider than necessary while the other axis’s is narrower than the decision needs. Used in §L.B.2 to define the allocation’s rule.

Q2. In 3.1.3, the episode-level split was required because adjacent frames are near-duplicates. What does the leakage’s magnitude depend on, and what is the audit that catches it?

Answer. It depends on the near-duplicate’s distance’s magnitude and the nearest-neighbour predictor’s sensitivity; the audit is the frame-split-versus-episode-split’s metric’s comparison, and a ratio of five or more is the leakage’s evidence. Used in §L.C.1 to define the suite’s leakage’s audit.

Q3. In 4.1.2, the commitment’s cost had three parts: the held-out suite’s invalidation, the baselines’ invalidation, and the metrics’ history’s invalidation. Which part does the suite’s version’s manifest protect?

Answer. The metrics’ history’s invalidation, because the manifest records which suite the history’s numbers were measured on; a change without a version’s bump is a silent redefinition, and the history’s comparison is invalid. Used in §L.D.1 to argue that the manifest is the history’s record.

Q4. In 3.4.4, the adaptation loop’s monitor measured the alignment’s MMD and the drift’s threshold was the held-out’s upper confidence bound. What is the suite’s analogue of the alignment’s MMD?

Answer. The suite’s distribution’s statistic — a two-sample test between the suite’s strata and the deployment’s episodes’ distribution — and its threshold is the suite’s own sampling’s confidence bound. Used in §L.D.2 to define the suite’s drift’s statistic.

Part A — The stratum’s definition

§L.A.1 — The cross-product and the claim

A stratum is a cell of the cross-product of the axes’ levels. The running case’s five axes have the levels of 4.1.2 §L.C.1:

• Seen task: the running case’s pick-and-place with the nominal instruction.
• Unseen object: the cup, the screwdriver, and the sponge.
• Unseen phrasing: the four paraphrases’ register of 3.1.3 §L.C.2.
• Unseen clutter: the divider’s presence’s configuration and the distractors’ identities.
• Unseen condition: the camera’s pose’s bins and the object’s position’s bins.

The full cross-product’s count is the levels’ product, which is enormous — the five axes’ levels multiply to hundreds of strata — and the suite’s 400 episodes cannot populate all of them. The stratification’s design is therefore not the enumeration; it is the selection of the strata that the claim’s content requires.

The claim’s content, from 4.1.2 §L.C.1, is the running case’s performance across the five axes’ marginal distributions and the interactions the claim names. The strata’s selection is the claim’s marginal-and-interaction structure’s expression:

The marginals. Each axis’s main effect is the average across the other axes’ levels, and its stratum is the axis’s level. The suite’s 400 episodes are allocated across the axes’ levels, and each level’s average is the marginal’s estimate.

The interactions. The claim’s content may name an interaction — the clutter’s effect on the unseen object’s success — and the interaction’s stratum is the pair of the two axes’ levels. The interaction’s episodes are a subset of the marginals’ and their allocation is the pair’s share.

The stratification’s design’s principle: the suite’s strata are the claim’s margins and the claim’s named interactions, not the cross-product’s enumeration. The cross-product’s count is a warning against over-stratification, not a target.

Self-explanation 1. Why is the cross-product’s enumeration a warning rather than a target?

Answer. Because each stratum’s decision needs a minimum episode count for its interval, and the cross-product’s hundreds of strata at 400 episodes give one or two episodes per stratum, which is a decision the suite cannot support; the cross-product’s count is the precision’s ceiling, not the coverage’s.

§L.A.2 — The stratum’s size’s minimum

Each stratum’s episode count is set by its decision’s resolution. The 4.1.2 §L.C.2’s metrics’ decisions apply per stratum, and the smallest stratum’s count is the coarsest decision’s resolution’s requirement.

For the running case’s primary decisions, the success rate’s 3-point resolution needs 384 episodes for the whole suite’s estimate, not per stratum. The per-stratum’s resolution is the stratum-level claim’s, and its requirement is set by the claim’s precision’s need at the stratum.

The distinction is worth stating carefully. The suite’s aggregate success rate’s 3-point resolution needs 384 episodes. The unseen object axis’s marginal’s 5-point resolution needs

which is the axis’s marginal’s requirement. The unseen object × unseen clutter interaction’s 10-point resolution needs $n \approx 35$, which is the interaction’s requirement.

The suite’s 400 episodes are allocated to meet the binding requirement, which is the aggregate’s 384. The marginals’ 138 and the interaction’s 35 are within the aggregate, not additional; the suite’s size is not the sum.

The stratum’s minimum’s design is the interaction’s 35 for the running case’s named interactions, and the marginal’s 138 for the four marginals. The 400’s allocation is the aggregate’s 384’s ceiling with the 16-episode margin.

§L.A.3 — The levels’ design

Each axis’s levels are the claim’s scope, and the levels’ choice is the axis’s design.

The seen task’s levels. One level — the nominal task — because the seen task’s claim is the base rate.

The unseen object’s levels. Three: the cup, the screwdriver, and the sponge, because the 3.1.3 §L.C.2’s object coverage’s requirement is the three unseen objects. The three levels are the axis’s strata, and each level’s marginal’s estimate is the object’s success.

The unseen phrasing’s levels. Four: the four paraphrases’ register, because the 3.1.3 §L.C.2’s paraphrase minimum is three to five and the four is the register’s coverage.

The unseen clutter’s levels. Three: the divider’s present-with-the-primary-bin’s configuration, the divider’s present-with-the-second-bin’s, and the distractors’ three’s configuration. The three levels are the clutter’s configurations.

The unseen condition’s levels. Four: the camera’s pose’s two bins (the trained pose and the shifted pose) and the object’s position’s two bins (the nominal and the shifted). The four levels are the condition’s bins, and the cross-product’s 16 is the interaction’s stratum.

The levels’ counts’ product is $1 \times 3 \times 4 \times 3 \times 4 = 144$, which is the cross-product’s count. The suite’s 400 episodes at 144 strata give 2.8 episodes per stratum, which is far below the interaction’s 35. The levels’ design is therefore the marginals’ design; the cross-product’s 144 is the interaction’s ceiling, and the interactions are only the ones the claim names.

Concept check. A learner proposes a cross-product stratification of 144 strata with 400 episodes, giving 2.8 episodes per stratum. Which principle does the proposal violate?

Answer. The stratum’s minimum’s principle: each stratum’s decision needs enough episodes for its resolution, and 2.8 episodes support no decision at any useful resolution. The proposal’s coverage is nominal and its precision is zero, and the correct design is the marginals’ 138 plus the named interactions’ 35, not the cross-product’s 144. [displaces M1]

Worked-example ladder.

WE-L.A.a (fully worked). The five axes’ levels are 1, 3, 4, 3, 4. Compute the cross-product’s count and the aggregate’s 3-point resolution’s episodes.

Solution. The cross-product is $1 \times 3 \times 4 \times 3 \times 4 = 144$ strata. The aggregate’s 3-point resolution needs $1.96^2 \times 0.09/0.03^2 = 384$ episodes. The 384 divided by the 144 gives 2.67 episodes per stratum, which is below the interaction’s 35 and the marginal’s 138.

WE-L.A.b (completion). The marginals’ 5-point resolution needs 138 episodes per axis. Compute the marginals’ total if the five axes’ levels are allocated the 138 each, and state whether it fits the 400.

Completion. The five axes’ marginals at 138 each would need 690 episodes, above the 400. The allocation is across the levels of each axis, not per axis; the unseen object’s three levels at 138 each would be 414, which is above the 400. The completion shows the levels’ allocation is the within-axis split, and the 138 is the axis’s total across its levels.

WE-L.A.c (faded). Design the interaction’s allocation for the unseen object × unseen clutter pair at the 10-point resolution’s 35 episodes.

Faded step. The pair’s 35 episodes are a subset of the marginals’ and their allocation is the three objects × three clutter configurations’ nine cells at about 4 episodes each. The 4 is below the interaction’s cell’s resolution, and the interaction’s aggregate across the nine cells is the 35.

Completion. The interaction’s claim is the average across the nine cells, not each cell’s; the 35 episodes are the average’s estimate, and the cells’ individual estimates are the marginals’ at a coarser resolution.

WE-L.A.d (contrastive near-miss). A learner’s suite has 144 strata with 2.8 episodes each and reports per-stratum success rates. Why it fails: each stratum’s 2.8 episodes gives a half-width of $1.96 \times \sqrt{0.09/2.8} = 0.35$, which is a 70-point interval; the per-stratum report’s precision is zero, and the aggregate’s precision is the only supportable claim. The fix: the marginals’ allocation, per §L.A.2.

Comparative table: the axes, levels, and allocations.

Design principle P52.1. Stratify by the claim’s margins and named interactions, not the cross-product; a stratum’s decision needs enough episodes for its resolution, and the cross-product’s count is the precision’s ceiling.

Consolidation checkpoint 1

(a) Three capabilities. You can define a stratum and compute the cross-product’s count. You can compute a marginal’s and an interaction’s episode requirement. You can design the axes’ levels.

(b) Transfer probe. The unseen condition’s four levels at a 10-point resolution need $n = 1.96^2 \times 0.09/0.10^2 = 35$ each, so 140 for the axis. Is the axis’s 40 in the table consistent? Answer: no; the axis’s 40 is the marginal’s estimate at a coarser resolution, roughly 25 points, and the table’s 40 is the claim’s scope at that coarser resolution. The choice is the claim’s precision’s decision.

(c) Remediation rule. If you cannot state why the cross-product’s enumeration is a warning, re-read §L.A.1 before Part B.

Part B — The episode’s selection and the allocation

§L.B.1 — The selection rule

The episode’s selection rule is the procedure that picks which episodes enter the suite. The rule has two properties, and each is necessary.

Random within a stratum. The episode is drawn uniformly from the stratum’s population — the deployment’s distribution’s episodes — so the stratum’s sample is representative. The randomness is the statistical validity’s requirement; a hand-picked episode is a bias, and the bias’s direction is the picker’s.

Deterministic across the suite. The suite’s set of episodes is reproducible from the selection rule and a fixed seed. The suite’s identity is the seed and the strata’s definitions, and the reproducibility is the evidence’s requirement; a suite whose episodes are selected differently on each run is a suite whose acceptance is not comparable across the capstone’s iterations.

The rule’s implementation is the 3.1.3 §L.D’s episode-level discipline plus the strata’s labels. The strata’s labels are the episodes’ metadata — the task, the object, the phrasing, the clutter’s configuration, the condition’s bin — and the selection is the random draw within each stratum’s labelled subset.

The rule’s two properties trade off against each other: a stricter randomness can produce a suite with the same seed but a different set, and a stricter determinism can bias the sample toward the seed’s stratum. The running case’s resolution is the stratified random sample with a fixed seed per stratum: each stratum’s draw is its own RNG, and the suite’s set is the union of the strata’s draws.

§L.B.2 — The allocation’s arithmetic

The allocation is the episodes’ distribution across the axes’ levels, and the arithmetic is the 4.1.2 §L.C.3’s weighting’s implementation.

The running case’s 400 episodes, allocated by the claim’s emphasis.

Seen task’s 100. The base rate’s estimate at a 0.9 success rate’s 5.9-point half-width.

Unseen object’s 100, allocated 40-30-30 across the cup, screwdriver, and sponge. The cup’s 40 gives a 9.3-point half-width; the screwdriver’s and the sponge’s 30 give a 10.7-point half-width. The allocation is not uniform because the cup’s grasp is the 3.1.3 §L.E.3’s hardest, and the claim’s emphasis on the cup is the allocation’s justification.

Unseen phrasing’s 80, allocated 20 per paraphrase. The four paraphrases’ 20 each give a 13-point half-width, which is coarse; the axis’s claim is the average across the paraphrases at a 6.6-point half-width, and the per-paraphrase’s estimate is reported but not decided.

Unseen clutter’s 80, allocated 30-30-20 across the three configurations. The divider-present-with-primary’s and the divider-present-with-second’s 30 each give a 10.7-point half-width; the distractors’ configuration’s 20 gives a 13-point half-width. The interaction with the unseen object’s 35 is a subset of the 80’s allocation, and its 35 gives a 10-point half-width.

Unseen condition’s 40, allocated 10 per bin. The four bins’ 10 each give a 18.6-point half-width, which is very coarse; the axis’s claim is the average at a 9.3-point half-width, and the per-bin’s estimate is reported as the condition’s range, not as a decision.

The allocation’s table.

The table’s role is the claim’s precision’s statement. The per-level’s half-widths are the claim’s precision, and the axes with 10-episode levels have per-level claims that are not decisions; the aggregate’s and the axis’s marginal’s claims are the decisions.

Self-explanation 2. Why are the condition’s per-bin estimates reported as a range rather than as a decision?

Answer. Because the 10-episode bins’ half-widths are 0.186, which is a 37-point interval; the interval cannot distinguish a good bin from a bad one, and the range’s purpose is the variation’s description, not the bin’s decision.

§L.B.3 — The split’s level and the leakage

The suite’s episodes are drawn from the population the capstone’s system did not train on. The split’s level is the discipline that guarantees the separation, and the three wrong levels are the 3.1.3 §L.D.2’s failure modes restated.

Frame-level split. The test’s frames are near-duplicates of the training’s, and the metric is interpolation. The 3.1.3 §L.D.2’s leakage ratio’s audit applies.

Task-level split. The test’s task is one the training did not see, and the metric is task generalization. The task-level split is correct only if the claim is task generalization; the running case’s claim is the within-task performance across the object, phrasing, clutter, and condition axes, and the task-level split would make the claim a different one.

Time-level split. The test’s episodes are a contiguous time window, and the metric is temporal generalization. The time-level split is correct only if the claim is temporal; the running case’s claim is not.

The correct level for the running case is the episode-level split, with the strata’s labels and the 3.1.3 §L.D.2’s batch’s stratification. The batch’s stratification prevents the collection’s burst from concentrating in one side; the suite’s episodes are drawn from all batches, and the training’s episodes are drawn from all batches, with the same seed’s separation.

The split’s audit is the 3.1.3 §L.D.2’s frame-versus-episode comparison. The running case’s audit is the nearest-neighbour predictor’s frame-split error divided by its episode-split error; a ratio above five is the leakage’s evidence, and the ratio is the 3.3.4 §L.B.2’s semantic layer’s complement at the evaluation’s level.

Concept check. A team’s suite uses a task-level split, so the training’s tasks and the test’s tasks are disjoint. Does the split support the capstone’s claim?

Answer. Only if the claim is task generalization. The capstone’s claim is the within-task performance across the object, phrasing, clutter, and condition axes, and a task-level split changes the claim to a different one — the mastery’s demonstration of a new task’s acquisition, which is not the capstone’s definition. [displaces M2]

Worked-example ladder.

WE-L.B.a (fully worked). The seen task’s 100 episodes at a 0.9 success rate give a 5.9-point half-width. The unseen object’s cup’s 40 give a 9.3-point half-width. Compute both intervals and state the claim’s precision’s comparison.

Solution. The seen task’s interval is $[0.9 - 0.059, 0.9 + 0.059] = [0.841, 0.959]$; the cup’s is $[0.9 - 0.093, 0.9 + 0.093] = [0.807, 0.993]$. The cup’s interval is 1.6 times the seen task’s, and the claim’s precision’s comparison is the allocation’s emphasis’s cost.

WE-L.B.b (completion). The unseen condition’s 10-episode bins give a 0.186 half-width. State why the per-bin’s estimate is not a decision and what the axis’s claim is instead.

Completion. The 0.186 is a 37-point interval, which cannot distinguish a 0.9 bin from a 0.5 bin; the axis’s claim is the average across the four bins at a 0.093 half-width, and the per-bin’s estimate is the range’s description. The completion shows the claim’s precision is the axis’s marginal’s, not the bin’s.

WE-L.B.c (faded). Design the split’s batch’s stratification for the running case’s collection’s 50 episodes per batch.

Faded step. The 50 batches’ episodes are labelled with the batch’s ID, and the split’s assignment is stratified by the batch so that each side has 25 batches.

Completion. The stratification’s purpose is the collection’s burst’s separation; a batch concentrated on one side is the environmental diversity’s loss, and the split’s stratification is the diversity’s protection.

WE-L.B.d (contrastive near-miss). A team’s suite uses a time-level split, with the last month’s episodes as the test set. The test’s episodes are from a different lighting’s season, and the metric’s fall is attributed to the system’s degradation. Why it fails: the time-level split’s claim is temporal generalization, and the fall is the season’s effect, not the system’s; the claim is a different one. The fix: the episode-level split, with the season’s batch as a stratum, not as the split’s level.

Comparative table: the split’s levels.

Design principle P52.2. Split at the level the claim is about; the level determines the distribution the interval is about, and the wrong level’s precision is precision about the wrong quantity.

Consolidation checkpoint 2

(a) Three capabilities. You can state the selection rule’s two properties. You can compute the per-level’s half-width from the allocation. You can state the split’s level’s rule and the three wrong levels.

(b) Transfer probe. The unseen object’s 100 episodes allocated 40-30-30 give per-level half-widths of 0.093, 0.107, and 0.107. Is the cup’s 40 the claim’s emphasis’s justification? Answer: yes, if the cup’s grasp is the 3.1.3 §L.E.3’s hardest and the claim’s emphasis is the cup; the 40 is the precision’s investment in the emphasis’s axis.

(c) Remediation rule. If you cannot state why the task-level split changes the claim, re-read §L.B.3 before Part C.

Part C — Leakage and the audit

§L.C.1 — The four leakage channels

Leakage is the suite’s information’s contamination by the training’s. The 3.1.3 §L.D.2’s frame-level leakage is one channel; the capstone’s suite has four.

The frame channel. Adjacent frames within an episode are near-duplicates, and a split that mixes frames across the train-test boundary leaks the episode’s trajectory. The audit is the 3.1.3 §L.D.2’s ratio.

The episode channel. The same episode appears in both the training and the test sets because the collection’s export duplicated it or the split’s assignment’s label mismatched. The audit is the episode’s ID’s duplicate’s check, and the duplicate’s removal is the fix.

The operator channel. The test’s operator is the training’s operator, and the operator’s style leaks. The 3.1.3 §L.C.1’s single-annotator’s failure is the channel’s; the audit is the operator’s identity’s label’s coverage, and the fix is the operator’s stratification.

The batch channel. The test’s episodes are from the same collection’s batch as the training’s, and the batch’s environmental condition leaks — the lighting, the table’s surface, the camera’s calibration’s state. The audit is the batch’s ID’s label’s coverage, and the fix is the batch’s stratification.

The four channels are not independent. The operator’s style and the batch’s condition correlate; the operator’s rotation’s schedule and the batch’s timing correlate. The audit’s design is the channels’ joint check, not the channels’ separate checks.

§L.C.2 — The leakage audit’s procedure

The leakage audit is the procedure that measures each channel’s leakage and states the suite’s contamination.

The procedure’s five steps.

Step 1: the labels’ inventory. Each episode’s metadata is the channels’ labels: the episode’s ID, the operator’s ID, the batch’s ID, and the frames’ timestamps. The inventory’s completeness is the audit’s premise.

Step 2: the duplicate’s check. The episode’s IDs’ duplicates are the episode channel’s leakage; the duplicates are removed, and their count is the audit’s first entry.

Step 3: the frame’s ratio. The 3.1.3 §L.D.2’s nearest-neighbour ratio is the frame channel’s leakage; a ratio above five is the channel’s failure, and the fix is the episode-level split’s assurance.

Step 4: the operators’ and batches’ coverage. Each side’s operators’ and batches’ labels are tabulated; a side with one operator’s episodes is the operator channel’s leakage, and a side with one batch’s is the batch channel’s.

Step 5: the report’s table. The four channels’ entries are the audit’s output, and the suite’s acceptance requires each channel’s entry to be at or below its threshold.

The audit’s thresholds are the 3.1.3 §L.D.2’s ratio’s five, the duplicates’ zero, and the operators’ and batches’ both sides’ presence’s requirement — each side must have at least three operators and at least three batches, because a side with fewer is the style’s or the condition’s concentration.

§L.C.3 — The audit’s automation and the manifest

The audit’s automation is the suite’s construction’s pipeline: the labels’ inventory, the duplicate’s check, the ratio’s computation, and the coverage’s tabulation are a script, and the script’s output is the suite’s manifest.

The manifest has four fields.

The suite’s version. An integer that increments on any change to the suite’s episodes, strata, or allocation.

The selection’s seed and the strata’s definitions. The two ingredients that make the suite reproducible.

The audit’s results. The four channels’ entries and the thresholds’ comparison.

The freeze’s timestamp. The moment the suite was frozen, which is the 4.1.2 §L.D.1’s commitment’s beginning.

The manifest’s role is the 4.1.2 §L.D.1’s metrics’ history’s record. The history’s numbers are measured on the manifest’s suite, and a change without the version’s bump is the history’s invalidation. The manifest’s automation is the discipline that makes the history’s comparison possible; a suite whose audit is a manual’s check is a suite whose history’s comparability depends on the auditor’s memory.

Self-explanation 3. Why is the manifest’s audit’s automation more important than the audit’s manual’s thoroughness?

Answer. Because the audit runs on every suite’s construction, and a manual’s check is run on the suite’s first construction only; the automation is the repeated check, and the history’s comparability requires each construction’s audit’s evidence, not the first’s.

Concept check. A suite’s audit shows a frame ratio of 4.2, which is below the five threshold, and the suite’s acceptance passes. What does the 4.2 tell the auditor?

Answer. It tells the auditor that the frame channel’s leakage is present but below the failure’s threshold; the metric’s precision is slightly inflated by the near-duplicates, and the inflation’s magnitude is a factor of the ratio’s. The threshold is a policy choice, and the 4.2’s acceptance is the choice’s decision, not the leakage’s absence.

Worked-example ladder.

WE-L.C.a (fully worked). The suite’s duplicates’ count is 3, the frame ratio is 4.8, the operators’ coverage is 4 on the training’s side and 3 on the test’s, and the batches’ coverage is 12 on each side. Compute the four channels’ entries and state the suite’s acceptance.

Solution. The duplicates’ 3 is above zero, so the episode channel fails and the duplicates must be removed. The frame ratio’s 4.8 is below five, so the frame channel passes. The operators’ 4 and 3 are above three, so the operator channel passes. The batches’ 12 and 12 are above three, so the batch channel passes. The suite’s acceptance fails on the episode channel’s duplicates, and the removal’s rerun is the fix.

WE-L.C.b (completion). The duplicates are removed, and the audit’s rerun gives the duplicates’ 0, the frame ratio’s 4.6, the operators’ 4 and 3, and the batches’ 12 and 12. Compute the suite’s acceptance.

Completion. All four channels pass, and the suite’s acceptance is the four’s conjunction’s. The completion shows the episode channel’s removal’s necessity and the other channels’ stability.

WE-L.C.c (faded). Design the audit’s script’s structure for the four channels.

Faded step. The script’s functions are inventory, check_duplicates, frame_ratio, coverage, and the script’s output is the manifest’s audit’s field.

Completion. The script’s structure’s principle is the reproducibility: the same labels’ inventory and the same thresholds give the same audit’s output, and the manifest’s version is the evidence.

WE-L.C.d (contrastive near-miss). A team’s audit’s operators’ coverage is 1 and 1, and the team’s suite’s acceptance passes because the frame ratio and the duplicates are fine. Why it fails: the operator channel’s threshold is three per side, and the 1 and 1 is the style’s concentration; the metric’s result is one operator’s style’s, not the population’s. The fix: the operator’s stratification, with at least three operators per side.

Comparative table: the leakage channels.

Design principle P52.3. Audit the four channels jointly and automate the audit; the manifest is the history’s record, and a suite without a manifest is not evidence.

Consolidation checkpoint 3

(a) Three capabilities. You can name the four leakage channels and their audits. You can run the audit’s five steps. You can write the manifest’s four fields.

(b) Transfer probe. The suite’s operators’ coverage is 2 on the test’s side. Which channel fails, and what is the fix? Answer: the operator channel; the fix is the operator’s stratification, with the suite’s episodes drawn from at least three operators on each side.

(c) Remediation rule. If you cannot state why the manifest’s automation is the history’s comparability’s requirement, re-read §L.C.3 before Part D.

Part D — The suite’s versioning and the drift’s detection

§L.D.1 — The version’s discipline

The suite’s version is an integer that increments on any change to the suite’s episodes, strata, or allocation. The version’s discipline is the 4.1.2 §L.D.1’s metrics’ history’s protection.

The version’s rules.

Any change is a version’s bump. A new episode, a removed episode, a stratum’s redefinition, or an allocation’s change all bump the version. The change’s magnitude is irrelevant; the version’s purpose is the record that the change is a choice.

The version is referenced in every metric’s report. The report’s numbers are on the version’s suite, and the reference is the comparison’s premise; a number without a version’s reference is a number whose suite is unknown.

The version’s changes are documented. The manifest’s history is the changes’ list, with each change’s reason. A version’s bump without a documented reason is the silent redefinition, and the reason’s documentation is the version’s discipline.

The discipline’s enforcement is the review’s gate: the 4.1.1 §L.E.3’s artefacts’ review checks the report’s version’s references against the manifest’s history, and a mismatch is the report’s rejection.

The version’s discipline’s cost is the administrative overhead — the manifest’s maintenance, the report’s references, the review’s check. The cost is small relative to the risk it mitigates: the metrics’ history’s invalidation is the capstone’s progress’s evidence’s loss, and the version’s discipline is the history’s record.

§L.D.2 — The drift’s detection

The suite is frozen, but the deployment’s distribution is not. The drift is the divergence between the suite’s distribution and the deployment’s, and its detection is the 3.4.4 §L.C.3’s monitor’s principle applied to the suite.

The drift’s statistic is the two-sample test’s MMD between the suite’s strata and the deployment’s episodes’ distribution, in the 3.1.2 §L.B.1’s feature space — the policy’s vision encoder’s output.

The statistic’s computation.

The suite’s sample. The 400 episodes’ images, in the encoder’s feature space.

The deployment’s sample. The deployment’s last N episodes’ images, in the same space. The N is the window’s size, which the 3.1.4 §L.D.2’s monitor’s window’s size’s choice.

The MMD’s computation. The 3.4.4 §L.D.1’s MMD between the two samples.

The statistic’s threshold is the suite’s own sampling’s confidence bound: the MMD between the suite’s two halves, computed at the suite’s construction, gives the baseline’s statistic, and the threshold is the baseline plus the confidence’s width. The deployment’s statistic above the threshold is the drift’s evidence.

The drift’s response is the 3.4.4 §L.C.3’s loop’s re-fit. The suite’s episodes are not re-collected — the version’s freeze is intact — but the deployment’s monitor’s statistic triggers the adaptation’s re-fit, and the suite’s evidence’s claim’s scope is updated to include the drift’s observation.

The drift’s distinction from the 3.4.4 §L.C.3’s monitor: the adaptation’s monitor measures the mapping’s alignment, and the suite’s drift’s monitor measures the deployment’s distribution’s shift. The two are complementary — the mapping’s misalignment is one cause of the deployment’s shift, and the deployment’s shift can occur without the mapping’s change (a new object’s introduction, a lighting’s change).

Self-explanation 1. Why is the drift’s threshold the suite’s own sampling’s bound rather than a fixed number?

Answer. Because the MMD’s value depends on the suite’s sample and the deployment’s sample’s sizes, and the threshold’s scale is the samples’ statistical variation; a fixed number is meaningful for one sample’s sizes only, and the suite’s bound is the scale’s reference.

§L.D.3 — The suite’s life cycle

The suite has a life cycle with four stages, and each stage is the 4.1.2 §L.D.1’s commitment’s temporal extent.

Stage 1: the construction. The audit’s automation, the manifest’s creation, and the version’s 1.

Stage 2: the freeze. The suite’s episodes are fixed, and the version’s increments only on a documented change.

Stage 3: the monitoring. The drift’s statistic is computed at each evaluation’s run, and the threshold’s crossing is the drift’s evidence.

Stage 4: the re-version. A drift’s crossing or a deliberate scope’s change triggers the version’s bump, with the change’s documentation and the audit’s rerun.

The life cycle’s discipline is the stage 4’s gate. A version’s bump is the 4.1.2 §L.D.1’s commitment’s renewal — the new version’s numbers are not comparable to the old’s aggregate, and the comparison is the within-version’s metrics’ history, with the across-version’s comparison as a separate analysis.

The life cycle’s cost is the drift’s monitoring’s frequency. The 3.1.4 §L.D.2’s window’s size is the frequency’s choice, and the running case’s is the 100-episode window at the deployment’s cadence.

Concept check. The suite’s drift’s statistic crosses the threshold, and the team re-collects the test set’s episodes and bumps the version. Is the capstone’s history preserved?

Answer. The within-version’s history is preserved, and the across-version’s comparison is the new analysis; the old version’s numbers are the baseline for the new’s, and the claim’s scope is the two versions’ distributions’ union. The history’s preservation is the version’s documentation, not the episodes’ continuity. [displaces M4]

Worked-example ladder.

WE-L.D.a (fully worked). The suite’s baseline’s MMD between its two halves is 0.03 with a confidence’s width of 0.02, so the threshold is 0.05. The deployment’s MMD is 0.04. Compute the drift’s status.

Solution. The 0.04 is below the 0.05 threshold, so the drift is not detected; the deployment’s distribution is within the suite’s sampling’s variation. The threshold’s crossing is the drift’s evidence, and the 0.04 is the margin’s status.

WE-L.D.b (completion). The deployment’s MMD rises to 0.07 after three months. Compute the status and the response.

Completion. The 0.07 is above the 0.05 threshold, so the drift is detected; the response is the 3.4.4 §L.C.3’s loop’s re-fit and the suite’s version’s bump with the documentation. The completion shows the drift’s detection’s two responses.

WE-L.D.c (faded). Design the suite’s version’s manifest’s history’s format for the two versions’ comparison.

Faded step. The manifest’s history has the version, the date, the change’s reason, the audit’s results, and the metrics’ report’s reference.

Completion. The history’s format’s purpose is the two versions’ comparison’s premise; the change’s reason’s documentation is the comparison’s interpretation’s source.

WE-L.D.d (contrastive near-miss). A team’s drift’s statistic crosses the threshold and the team does not re-fit the adaptation, arguing the suite’s freeze is intact. Why it fails: the freeze protects the evaluation’s evidence, not the deployment’s performance; the drift’s detection is the deployment’s signal, and the freeze is the evaluation’s. The two are different mechanisms, and the team’s confusion is the 4.1.2 §L.D.1’s commitment’s scope’s misreading. The fix: the drift’s response’s separation from the suite’s freeze.

Comparative table: the suite’s life cycle.

Design principle P52.4. Version every change and monitor the drift; the freeze protects the evaluation’s evidence, the monitor protects the deployment’s claim, and the two are different mechanisms.

Consolidation checkpoint 4

(a) Three capabilities. You can state the version’s three rules. You can compute the drift’s statistic’s threshold and status. You can lay out the suite’s four life cycle’s stages.

(b) Transfer probe. The suite’s baseline’s MMD is 0.02 with a confidence’s width of 0.015, and the deployment’s MMD is 0.05. Is the drift detected? Answer: the threshold is 0.035, and the 0.05 is above it; the drift is detected, and the response is the adaptation’s re-fit and the version’s bump.

(c) Remediation rule. If you cannot state why the drift’s threshold is the suite’s own bound, re-read §L.D.2 before Part E.

Part E — The running case

§L.E.1 — The suite’s assembly

The running case’s 400-episode suite, assembled from the axes’ levels and the allocation’s table.

The strata. The five axes’ marginals — the seen task’s 1 level, the unseen object’s 3 levels, the unseen phrasing’s 4 levels, the unseen clutter’s 3 levels, and the unseen condition’s 4 levels — plus the unseen object × unseen clutter’s interaction’s 35 episodes.

The allocation. The seen task’s 100, the unseen object’s 100 (40-30-30), the unseen phrasing’s 80 (20 each), the unseen clutter’s 80 (30-30-20), and the unseen condition’s 40 (10 each). The interaction’s 35 is a subset of the unseen clutter’s 80 and the unseen object’s 100; the subset’s selection is the two axes’ cells’ draw, and its episodes are counted once.

The selection’s seed. The 4.1.3 §L.B.1’s stratified random sample with the fixed seed. The suite’s reproducibility is the seed and the strata’s definitions.

The audit’s results. The four channels: the frame ratio 4.3, the duplicates 0, the operators 4 and 3, the batches 12 and 12. The conjunction passes.

The manifest’s version. Version 1, with the freeze’s timestamp at the 3.1.4 §L.A.1’s Phase 1’s baseline audit’s start.

The suite’s cost is 400 episodes at the 25-second cycle, which is 2.78 hours, within the 4.1.2 §L.E.2’s 3-hour budget.

§L.E.2 — The suite’s precision’s statement

The suite’s report’s precision’s table, from the allocation’s per-level’s half-widths.

The statement’s scope: the seen task’s and the unseen object’s claims are the decisions at the 6-point and 10-point resolutions; the unseen phrasing’s and the unseen clutter’s claims are the marginal averages at the 7-point and 8-point resolutions, with the per-level’s estimates reported but not decided; the unseen condition’s claim is the range’s description, not a decision.

The statement’s honesty is the 4.1.2 §L.D.2’s “what would change my mind” test’s item: the condition’s per-bin’s claim would be wrong if the per-bin’s estimates were used as decisions, and the statement’s no-decision is the choice’s falsifiability’s item.

§L.E.3 — The manifest and the drift’s monitor

The manifest’s four fields.

Version. 1.

Seed and strata. The seed is 20260601, and the strata’s definitions are the §L.E.1’s five axes’ levels’ lists.

Audit’s results. The four channels’ entries of §L.E.1.

Freeze’s timestamp. The 3.1.4 §L.A.1’s Phase 1’s baseline audit’s start’s date.

The drift’s monitor’s design.

The statistic. The MMD between the suite’s 400 episodes’ features and the deployment’s 100-episode window’s features, in the 3.1.2 §L.B.1’s encoder’s space.

The threshold. The suite’s baseline’s MMD (the two halves’ comparison) plus the 95% confidence’s width, computed at the suite’s construction.

The response. The threshold’s crossing triggers the 3.4.4 §L.C.3’s loop’s re-fit and the version’s bump with the documentation.

The monitor’s cost is the 100-episode window’s feature’s extraction, which is the 2.1.2 estimator’s inference’s 30 ms per image and the encoder’s 5 ms, so about 3.5 seconds per window’s extraction, and the cadence is the deployment’s.

Self-explanation 2. Why is the manifest’s seed’s field the suite’s reproducibility’s requirement?

Answer. Because the selection’s rule is the stratified random sample, and the randomness’s realization is the seed’s function; the same seed and the same strata’s definitions produce the same suite, and the reproducibility is the acceptance’s comparability’s premise.

Worked-example ladder.

WE-L.E.a (fully worked). The seen task’s 100 episodes give a 5.9-point half-width, and the unseen object’s cup’s 40 give a 9.3-point. The baseline’s difference is 0.04. Is the seen task’s difference significant?

Solution. The seen task’s interval at the baseline’s 0.86 is $[0.86 - 0.059, 0.86 + 0.059] = [0.801, 0.919]$, and the 0.90 is inside; the seen task’s difference is not significant at the 100 episodes. The seen task’s aggregate at 400 episodes gives the significance, and the seen task’s marginal at 100 gives the coarse’s comparison.

WE-L.E.b (completion). The unseen condition’s 40 episodes give a 9.3-point half-width for the average and a 18.6-point for each bin. State the report’s claim’s scope and the “what would change my mind” item.

Completion. The report’s claim is the average’s 9.3-point precision, and the “what would change my mind” item is the per-bin’s claim’s absence; if the per-bin’s estimate were used as a decision, the choice would be wrong. The completion shows the scope’s precision’s statement is the falsifiability’s item.

WE-L.E.c (faded). Design the drift’s monitor’s cadence for the running case’s deployment.

Faded step. The monitor’s cadence is the 100-episode window’s completion, which at the cell’s 200 episodes per day is the half-day’s cadence.

Completion. The cadence’s principle is the window’s size’s statistical validity’s requirement; a smaller window gives a noisier statistic, and the 100-episode window is the noise’s and the cadence’s balance.

WE-L.E.d (contrastive near-miss). A team’s report uses the unseen condition’s per-bin’s estimate as a decision and declares the “shifted camera’s pose” bin’s success 0.72 as a failure. Why it fails: the per-bin’s 18.6-point half-width gives the interval $[0.53, 0.91]$, which includes the requirement’s 0.90, and the 0.72 is not distinguishable from the requirement at the bin’s resolution. The fix: the report’s claim’s scope’s statement, per §L.E.2.

Comparative table: the suite’s precision’s statement.

Design principle P52.5. State the suite’s precision’s scope; a per-level’s estimate is a claim at the level’s resolution, and a decision above the level’s resolution is the claim’s scope’s violation.

Consolidation checkpoint 5

(a) Three capabilities. You can assemble the suite’s strata and allocation. You can state the precision’s table’s decision’s statuses. You can design the manifest and the drift’s monitor.

(b) Transfer probe. The unseen phrasing’s per-paraphrase’s 20 episodes give a 13.1-point half-width. Is the per-paraphrase’s estimate a decision? Answer: no; the axis’s average’s 6.6-point half-width is the decision, and the per-paraphrase’s estimate is reported at the level’s coarser precision, not decided.

(c) Remediation rule. If you cannot state why the manifest’s seed is the reproducibility’s requirement, re-read §L.E.3 before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: the false alarm, the manifest’s absence, and the strata’s re-collection.

The lab

Task. Build the running case’s 400-episode suite’s strata and allocation, run the leakage audit’s four channels, write the manifest’s four fields, and compute the precision’s table and the drift’s monitor’s threshold.

Conditions. Python 3.10+, NumPy. The suite’s episodes are the 3.1.3 dataset’s labelled subset; the selection’s seed and the strata’s definitions are the suite’s construction’s inputs. Seed all RNGs.

Controls. (i) The nominal suite; (ii) the cross-product’s 144 strata’s control; (iii) the time-level split’s control; (iv) the drift’s monitor’s threshold’s computation.

Metrics.

Pre-registered hypotheses.

H1. The cross-product’s 144 strata at 400 episodes give a per-stratum’s 2.8 episodes, and the per-stratum’s half-width is above 0.30.

H2. The marginals’ allocation gives the seen task’s 100 episodes a 5.9-point half-width and the unseen condition’s 40 a 9.3-point.

H3. The time-level split’s audit shows the batch’s concentration, and the operator’s coverage is one per side.

H4. The drift’s threshold from the suite’s two halves’ MMD plus the confidence’s width is below the deployment’s typical’s statistic’s scale.

Break-it sequence.

1. Enumerate the cross-product silently. The 144 strata’s per-stratum’s precision is zero, and the report’s per-stratum’s estimates are noise. Silent: the strata’s count looks complete.
2. Use the time-level split silently. The season’s effect is the metric’s, not the system’s. Silent: the split is technically valid.
3. Bump the version without the documentation silently. The history’s comparison is invalid, and the report’s numbers are on different suites. Silent: the version’s number is correct.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The strata’s count is the marginals’ and the interactions’, not the cross-product’s. (2) The allocation’s table’s per-level’s half-widths are computed, not asserted. (3) The audit’s four channels are automated and the manifest’s audit’s field is the script’s output. (4) The drift’s threshold is the suite’s own sampling’s bound. (5) The break-it results are a table.

A runnable core of the suite’s construction, seeded and annotated:

# lab_4_1_3.py — strata, allocation, audit, manifest
import numpy as np
np.random.seed(0)

AXES = {
    "seen_task":  {"levels": 1, "episodes": 100},
    "unseen_obj": {"levels": 3, "episodes": 100, "split": [40, 30, 30]},
    "unseen_phrase": {"levels": 4, "episodes": 80, "split": [20, 20, 20, 20]},
    "unseen_clutter": {"levels": 3, "episodes": 80, "split": [30, 30, 20]},
    "unseen_cond": {"levels": 4, "episodes": 40, "split": [10, 10, 10, 10]},
}

def half_width(n, p=0.90):
    return 1.96 * np.sqrt(p * (1 - p) / max(n, 1))

total = 0
for axis, spec in AXES.items():
    n = spec["episodes"]; total += n
    print(f"{axis:16s} levels={spec['levels']}  episodes={n}  "
          f"avg_half={half_width(n / spec['levels']):.3f}")
print(f"total episodes (marginals) = {total}")

# Interaction: object x clutter, 35 episodes as a subset
inter = 35
print(f"interaction obj x clutter: episodes={inter}  half={half_width(inter):.3f}")

# Leakage audit's four channels
audit = {
    "frame_ratio":     4.3,     # < 5 passes
    "duplicates":      0,       # == 0 passes
    "operators_train": 4,       # >= 3 passes
    "operators_test":  3,
    "batches_train":  12,       # >= 3 passes
    "batches_test":   12,
}
checks = {
    "frame": audit["frame_ratio"] < 5,
    "dup":   audit["duplicates"] == 0,
    "op":    audit["operators_train"] >= 3 and audit["operators_test"] >= 3,
    "batch": audit["batches_train"] >= 3 and audit["batches_test"] >= 3,
}
print("audit:", checks, "  all pass:", all(checks.values()))

# Manifest
manifest = {
    "version": 1,
    "seed": 20260601,
    "strata": list(AXES.keys()),
    "audit": audit,
    "frozen_at": "2026-06-01T00:00Z",
}
print("manifest:", manifest["version"], manifest["seed"])

# Drift's threshold
baseline_mmd, ci_width = 0.03, 0.02
print(f"drift threshold = {baseline_mmd + ci_width:.3f}")

Line 4–10 define the five axes’ levels and allocations. Lines 12–14 compute the per-level’s half-width. Lines 16–20 sum the marginals and add the interaction. Lines 22–38 run the audit’s four channels and print the conjunction. Lines 40–46 write the manifest. Line 48–49 compute the drift’s threshold from the baseline plus the confidence’s width.

Cheat sheet

Exercises with solutions

E 4.1.3.1. Define a stratum and compute the running case’s cross-product’s count.

S 4.1.3.1. A cross-product’s cell of the axes’ levels; the count is $1 \times 3 \times 4 \times 3 \times 4 = 144$. The lesson: the cross-product’s count is the precision’s ceiling, and the claim’s margins and interactions are the strata’s design.

E 4.1.3.2 (transfer). A marginal’s 5-point resolution needs 138 episodes. The 400-episode suite’s unseen object’s 100 are below 138. Does the axis’s marginal’s claim’s resolution’s requirement fail?

S 4.1.3.2. The 100’s half-width is 5.9 points, above the 5-point requirement; the requirement’s 138 is the 5-point resolution’s, and the 100 is the 5.9’s. The claim’s scope is 5.9 points, and the requirement’s 5-point is not met. The lesson: the claim’s scope is the allocation’s half-width, not the axis’s full’s.

E 4.1.3.3. Compute the seen task’s 100 episodes’ 95% half-width at a 0.90 success rate.

S 4.1.3.3. $1.96 \times \sqrt{0.09/100} = 0.0588$. The lesson: the allocation’s half-width is the claim’s precision, and the 100’s 5.9 points is the base rate’s statement.

E 4.1.3.4 (diagnostic). A suite’s per-stratum’s estimate gives 0.72 for the shifted camera’s bin. Is the estimate a decision?

S 4.1.3.4. No; the bin’s 10 episodes give an 18.6-point half-width, and the 0.72’s interval is $[0.53, 0.91]$, which includes the 0.90 requirement. The estimate is the range’s description, not a decision. The lesson: the per-level’s estimate is a decision only at the level’s resolution.

E 4.1.3.5. Name the four leakage channels and their audits.

S 4.1.3.5. Frame (NN ratio), episode (duplicates), operator (coverage per side), batch (coverage per side). The lesson: the four channels are the leakage’s decomposition, and their joint audit is the manifest’s evidence.

E 4.1.3.6 (transfer). The frame ratio is 4.2 and the threshold is 5. What does the 4.2 tell the auditor?

S 4.1.3.6. The leakage is present and below the failure’s threshold; the metric’s precision is slightly inflated by the near-duplicates, and the threshold’s acceptance is a policy’s choice. The lesson: the threshold is a policy’s number, and the 4.2 is the leakage’s magnitude’s statement.

E 4.1.3.7 (diagnostic). The drift’s statistic is 0.06 and the threshold is 0.05. What are the two responses?

S 4.1.3.7. The adaptation’s re-fit and the suite’s version’s bump with the documentation. The two responses address the deployment’s shift and the history’s record. The lesson: the drift’s detection’s two responses are different mechanisms.

E 4.1.3.8. State why the version’s bump without the documentation invalidates the metrics’ history’s comparison.

S 4.1.3.8. The history’s numbers are measured on the suite’s version, and the version’s change without the documentation is the silent redefinition; the comparison’s interpretation’s source is the change’s reason, and its absence is the comparison’s invalidity. The lesson: the manifest’s history is the comparability’s record.

E 4.1.3.9 (transfer). The suite’s time-level split gives a 0.72 success and the episode-level gives a 0.88. Which is the capstone’s claim?

S 4.1.3.9. The episode-level’s 0.88, because the time-level split’s claim is temporal generalization and the capstone’s claim is the within-task performance. The lesson: the split’s level determines the claim’s distribution.

Exit test

You are ready for Module 4.2 when you can, without notes:

1. Define a stratum and compute the cross-product’s count.
2. State the cross-product’s enumeration’s warning.
3. Compute a marginal’s and an interaction’s episode requirement.
4. State the selection rule’s two properties.
5. Allocate the 400 episodes and compute the per-level’s half-width.
6. State the split’s level’s rule and the three wrong levels.
7. Name the four leakage channels and their audits.
8. Run the audit’s five steps.
9. Write the manifest’s four fields.
10. State the version’s three rules.
11. Compute the drift’s statistic’s threshold.
12. Lay out the life cycle’s four stages.
13. Assemble the running case’s suite’s strata and allocation.
14. State the precision’s statement’s scope.
15. Design the drift’s monitor’s cadence.

Cross-references. 4.1.1 supplies the capstone’s six metrics, four axes, and three preconditions restated in §L.A.1 and §L.C.1. 4.1.2 supplies the fifth axis, the frozen design, and the size’s maximum restated in §L.A.2 and §L.E.1. 3.1.3 supplies the episode-level split and the leakage’s ratio restated in §L.B.3 and §L.C.1. 3.1.4 supplies the lower-bound acceptance restated in §L.E.2. 3.4.1 supplies the twin-track’s tail restated in §L.E.2. 3.4.4 supplies the adaptation’s monitor and the MMD restated in §L.D.2. Module 4.2 builds the seven subsystems on the chosen embodiment and this suite. Module 4.3 iterates against the suite’s metrics. Module 4.4 defends the result on the suite’s evidence.

Ready. The next lesson to write is 4.2.1 — Perception Stack Integration. Say “next” to generate it.