Lesson 4.2.1 — Perception Stack Integration

Week 34

Prerequisite lessons: 4.1.3 (evaluation design and held-out test suites), 4.1.2 (choosing the embodiment and the evaluation), 2.1.1 (camera models and image formation), 2.1.2 (object detection and 6D pose estimation), 2.1.4 (point clouds and registration), 2.1.5 (visual SLAM and odometry), 3.4.4 (real-to-sim adaptation).

Prior results used in this lesson. From 4.1.1: the perception-to-scene-understanding interface contract (pose, covariance, freshness, frame) and the 30 ms perception budget. From 4.1.3: the suite’s strata and the drift’s monitor. From 2.1.1: the projection $\tilde{\mathbf{p}} = \frac{1}{Z}\mathbf{K}\mathbf{X}_C$ with the calibrated $\mathbf{K}$ and distortion. From 2.1.2: the 6D pose estimator and its covariance. From 2.1.4: point-cloud registration as a rigid-transform estimator. From 2.1.5: the eye-in-hand kinematic prior $T_W^C(q) = T_W^E(q)T_E^C$ and the relative-pose estimator. From 3.4.4: the adaptation’s mapping and its monitor’s MMD. Each is restated at the point of use.

What this lesson hands forward. The fused perception contract, the consistency checks’ thresholds, the failure ladder, and the latency’s scheduling. Lesson 4.2.2 (the policy layer) consumes the contract’s outputs; Lesson 4.2.3 (the reasoning layer) consumes the scene’s serialization built from them; Lesson 4.2.4 (safety and recovery) consumes the failure ladder as its response’s source. Module 4.3 (iteration and failure analysis) uses the consistency checks as its diagnostic’s basis.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, divider, and three distractors (cup, screwdriver, sponge) are unchanged; this lesson assembles the seven perception sources — RGB-D pose, point cloud, SLAM’s relative pose, kinematic prior, adaptation’s mapping, freshness’s clock, and the drift’s monitor — into the single perception stack that the 400-episode suite’s evaluation measures.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) state the contract’s five requirements and the failure mode each guards; (2) explain why a single-source perception stack cannot satisfy the contract and which requirement it fails; (3) state the three fusion levels and choose one for the running case; (4) fuse two pose estimates with their covariances and compute the fused covariance; (5) run the Mahalanobis consistency check and the temporal plausibility check with their thresholds; (6) name the four perception failure modes and their detection signals; (7) lay out the failure ladder’s three rungs and their triggers; (8) compute the 95th percentile’s latency from per-source budgets and state the tail’s management; (9) assemble the running case’s stack with its contract, fusion, checks, ladder, and schedule; (10) write the perception’s register’s entries with residuals.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 4.1.1 §L.B.2, the perception-to-scene-understanding contract’s data was the pose with its covariance and its freshness’s timestamp. What does the freshness’s timestamp guard against, and what is the guard’s mechanism?

Answer. It guards against the stale pose: the reasoning grounds a reference on a pose that has moved. The guard is the freshness bound and the 3.3.4 §L.B.2’s reference check, which rejects a grounding whose pose’s age exceeds the bound. Used in §L.A.1 to define the contract’s freshness requirement.

Q2. In 2.1.2, the 6D pose estimator’s covariance was the uncertainty of the pose’s estimate from a single RGB-D frame. What does the covariance not include, and why is that a problem for the integration?

Answer. It does not include the estimator’s model’s error — its bias on objects or poses outside its training’s distribution — and it does not include the frame’s error — the camera’s calibration’s drift. The integration’s problem is that the covariance understates the total uncertainty, and the downstream consumer’s tolerance is checked against an optimistic number. Used in §L.B.2 to motivate the consistency check’s complement.

Q3. In 2.1.5, the eye-in-hand camera’s pose was available from forward kinematics $T_W^C(q) = T_W^E(q)T_E^C$. What is the kinematic prior’s role in the perception stack, and what is its limit?

Answer. Its role is to supply the camera’s pose when the visual odometry’s estimate is unavailable or degraded; its limit is that it does not include the object’s pose, so it cannot substitute for the object’s estimator. Used in §L.A.2 to explain the single-source’s ceiling.

Q4. In 3.4.4, the adaptation’s monitor was the MMD between the suite’s features and the deployment’s. What does the perception stack’s analogue of the monitor measure, and what is the response?

Answer. It measures the alignment between the stack’s inputs — the camera’s images — and the suite’s images’ distribution; the response is the adaptation’s re-fit, per 3.4.4 §L.C.3. Used in §L.D.3 to add the drift’s monitor to the stack’s latency budget.

Part A — The contract’s five requirements

§L.A.1 — What the perception stack must deliver

The perception stack’s output is consumed by the scene understanding (3.3.1), the grasp planning (2.3.1), and the safety stack (3.3.4). Each consumer’s tolerance imposes a requirement, and the requirements are the contract’s five fields.

The pose. The object’s 6D pose in the world frame, $T_W^{Obj}$. The pose’s form is the 2.1.2 estimator’s convention — the rotation’s representation and the translation’s units — and the consumer’s frame’s expectation is the contract’s first clause.

The covariance. The pose’s uncertainty, as a $6 \times 6$ matrix in the same frame. The covariance’s form is the estimator’s, and its correctness is the contract’s second clause; the downstream consumer’s Mahalanobis check is the correctness’s test.

The freshness. The timestamp of the observation the pose was computed from, and the age of that observation relative to the request. The bound is the consumer’s tolerance — the 3.3.1 reasoning’s reference check and the 2.3.1 grasp’s pose are the consumers whose tolerances set the bound.

The frame. The frame the pose is expressed in, with its definition. The world frame $W$ is the running case’s, defined by the table’s surface and the arm’s base; the contract’s fourth clause is the frame’s identity, not only its label, because a label without a definition is an ambiguity.

The provenance. The sources the pose was computed from, and their individual covariances. The provenance is the contract’s fifth clause, and it is the one most often omitted; the consumers’ failure analysis is impossible without it, because a fused pose’s error cannot be attributed to a source that was not recorded.

The five requirements are not independent. The pose and the covariance are the value; the freshness and the frame are the context; the provenance is the audit. A stack that delivers the value and omits the context and the audit is a stack whose consumers cannot check it, and the 3.3.4 §L.B.2’s semantic layer’s reference check has nothing to check against.

Self-explanation 1. Why is the provenance a requirement rather than a debugging convenience?

Answer. Because the fused pose’s error is a function of the sources’ errors, and the 4.2.4 failure recovery’s selection of the ladder’s rung depends on which source failed; without the provenance, the recovery is a guess, and the guess is the 3.3.4 §L.D’s register’s residual.

§L.A.2 — The single-source’s ceiling

The simplest perception stack is the 2.1.2 estimator on every frame, with the 2.1.5 kinematic prior as a fallback when the estimator’s confidence is low. This single-source stack satisfies the pose’s requirement, and it fails the other four in a specific order.

The covariance’s understatement. The 2.1.2 covariance is the estimator’s statistical uncertainty on its training’s distribution; the objects’ and poses’ variations that the training did not cover are a bias the covariance does not include. The 3.4.3 §L.E.2’s structure-error’s allowance is the same argument in the identification’s coordinates. The single-source stack’s covariance understates the total, and the downstream consumer’s Mahalanobis check passes where it should fail.

The freshness’s variability. The estimator’s inference time varies with the object’s size and the image’s content. A small object, like the sponge’s corner, takes longer, and the pose’s age varies; the stack that delivers the pose as soon as it is computed has no mechanism to hold the pose until the consumer’s request’s deadline. The 3.3.1 reasoning’s request’s cadence and the estimator’s cadence are not synchronized, and the single-source stack’s age is the difference.

The frame’s single point. The estimator’s output’s frame is the camera’s, transformed to the world by the kinematic prior. The transform’s error is the 2.1.5 SLAM’s or the kinematic’s, and the single-source stack has no independent estimate to check it. A drift in the camera’s pose’s estimate is invisible, and the object’s pose inherits the drift.

The provenance’s emptiness. The single-source stack’s provenance is one source, and the 3.3.4 §L.B.2’s semantic layer’s check has one number to check. A single number’s error is not attributable, and the recovery’s rung is undetermined.

The single-source’s ceiling is the integration’s rather than the estimator’s. The estimator is good; the integration’s requirements are five, and the single source satisfies one and a half of them.

§L.A.3 — The multi-source’s logic

The multi-source stack’s logic is the contract’s five requirements’ independent satisfaction. Each source is chosen for the requirement it satisfies best, and the fusion combines the sources’ values while keeping their provenance.

The running case’s sources and their best requirements.

The RGB-D pose estimator (2.1.2) is the value’s source: it is the most precise on the training’s distribution, and its covariance is the statistical uncertainty’s measure.

The point-cloud registration (2.1.4) is the value’s second source: it is less precise on the RGB-D’s distribution and more robust to the lighting’s changes; its covariance is a complementary estimate, and the two’s agreement is the consistency check’s test.

The SLAM’s relative pose (2.1.5) is the frame’s source: it estimates the camera’s motion independently of the kinematic prior, and its disagreement with the prior is the drift’s signal.

The kinematic prior (2.1.5) is the frame’s second source: it is the prior on the camera’s pose, and it is the fallback when the SLAM’s estimate is unavailable.

The adaptation’s mapping (3.4.4) is the value’s third source: it maps the simulated renderer’s images to the camera’s, and its presence is the training’s distribution’s alignment.

The freshness’s clock is the freshness’s source: it timestamps each observation, and the stack’s scheduler uses the timestamps to hold the pose until the consumer’s deadline.

The drift’s monitor is the provenance’s source: it records the sources’ individual covariances and their agreement, and its statistic is the consistency check’s output.

The sources’ count is seven, and their independent requirements’ coverage is complete: value from three sources, frame from two, freshness from one, provenance from one. The fusion’s benefit is the independent sources’ count — the value’s three are correlated by their common camera, and the frame’s two are independent by their different mechanisms — and the correlation’s accounting is §L.B’s subject.

Concept check. A team adds a fourth pose estimator on the same RGB-D frame. Does the fourth source reduce the fused covariance by the full factor?

Answer. No; the estimator’s errors share the camera’s noise and the frame’s calibration’s error, so the fourth source’s error is correlated with the existing three, and the reduction is less than the independent-source’s formula predicts. The mechanism is the correlation, and the fix is a source with a different physical mechanism — the point cloud’s registration is such a source. [displaces M2]

Worked-example ladder.

WE-L.A.a (fully worked). The single-source stack delivers the cube’s pose with the 2.1.2 covariance $\Sigma_1 = \mathrm{diag}(0.0004, 0.0004, 0.0009, 0.0001, 0.0001, 0.0003)$ in metres squared and radians squared. The consumer’s Mahalanobis threshold is $\chi^2_{0.95, 6} = 12.59$. Compute the maximum pose error that passes the check along each axis.

Solution. The single-axis’s threshold is $\sqrt{12.59} \times \sigma_i$. For the $x$-axis: $\sqrt{12.59} \times 0.02 = 0.071$ m; for the $z$-axis: $\sqrt{12.59} \times 0.03 = 0.106$ m; for the roll: $\sqrt{12.59} \times 0.01 = 0.035$ rad. The check’s axes’ errors’ tolerances are the square roots of the variances’ scaling, and the pose’s error is the sum over axes if the axes are treated jointly.

WE-L.A.b (completion). The 2.1.4 point cloud’s covariance is $\Sigma_2 = \mathrm{diag}(0.0016, 0.0016, 0.0025, 0.0004, 0.0004, 0.0010)$. Compute the fused covariance under the independence assumption and state the fused covariance’s largest axis.

Completion. The fused covariance’s diagonal is $(\Sigma_1^{-1} + \Sigma_2^{-1})^{-1}$ per axis: $x$: $(2500 + 625)^{-1} = 0.00032$; $z$: $(1111 + 400)^{-1} = 0.00066$. The fused covariance’s largest axis is still the $z$‘s, and the reduction is 30% on $x$ and 26% on $z$ — less than the factor of 2 that the naive average suggests.

WE-L.A.c (faded). Write the contract’s five fields for the cube’s pose and state the failure mode each guards.

Faded step. Pose (the value), covariance (the uncertainty’s correctness), freshness (the staleness), frame (the ambiguity), provenance (the attribution).

Completion. The failure modes are: wrong value (caught by the consistency check), wrong uncertainty (caught by the 4.2.4 recovery’s mis-calibration), stale (caught by the reference check), ambiguous frame (caught by the frame’s identity’s test), unattributable (caught by the provenance’s absence).

WE-L.A.d (contrastive near-miss). A team’s stack fuses the RGB-D and point-cloud poses and omits the provenance. The cube’s pose is 2 cm off, and the recovery’s rung cannot be selected. Why it fails: the fused value is correct on average and the error’s source is unknown; the recovery’s three rungs — re-observe, degrade, abstain — are selected by the source’s failure, and the provenance’s absence removes the selection’s basis. The fix: the provenance’s field, per §L.A.1.

Comparative table: the contract’s five requirements.

Design principle P53.1. Deliver the pose’s distribution with its context and its audit; the value alone is one of five requirements, and the other four are the integration’s.

Consolidation checkpoint 1

(a) Three capabilities. You can state the five requirements and their failure modes. You can state the single-source’s ceiling’s four failures. You can name the seven sources and their best requirements.

(b) Transfer probe. The 2.1.2 estimator’s covariance is $0.0004$ in $x$, and the true error’s variance is $0.0010$. Which requirement fails? Answer: the covariance’s correctness; the estimator’s understated uncertainty makes the Mahalanobis check pass when it should fail, and the fused covariance inherits the understatement.

(c) Remediation rule. If you cannot state why the provenance is a requirement, re-read §L.A.1 before Part B.

Part B — The fusion architecture

§L.B.1 — The three fusion levels

The fusion’s level is the stage of the pipeline at which the sources are combined, and each level has a different error’s propagation.

Early fusion. The sources are combined at the input’s stage — the images are stitched, the point clouds are merged — and the estimator runs on the combined input. The advantage is the estimator’s exploitation of the combined data; the disadvantage is that a source’s failure contaminates the combined input, and the attribution is lost.

Late fusion. The sources are combined at the output’s stage — each source produces a pose, and the poses are fused by the inverse-variance rule — and the estimator’s individual failures are isolated. The advantage is the attribution and the failure’s isolation; the disadvantage is that the sources’ cross-information is not exploited.

Hybrid fusion. The sources are combined at an intermediate stage — the features are aligned and then the estimate is fused — and the advantages of both levels are partly realized. The hybrid is the 3.4.4 §L.B.2’s feature-level alignment’s structure, and its cost is the intermediate’s design’s complexity.

The running case’s fusion is the late fusion of the value’s three sources: the RGB-D pose, the point-cloud pose, and the adaptation’s aligned pose. The choice is by the attribution’s requirement: the 4.2.4 recovery’s rung selection needs the sources’ individual errors, and the early fusion’s contamination removes the attribution.

The frame’s sources are fused differently. The SLAM’s relative pose and the kinematic prior are compared, not averaged, because their disagreement is the drift’s signal: a fused frame’s estimate would hide the disagreement, and the 3.3.4 §L.B.2’s semantic layer’s frame’s check needs the disagreement’s magnitude.

Self-explanation 2. Why is the frame’s fusion a comparison rather than an average?

Answer. Because the frame’s sources are the drift’s detectors — the SLAM’s estimate and the kinematic prior disagree when the kinematic prior has drifted — and the average would be a value that hides the drift; the comparison’s output is the disagreement’s magnitude, which is the drift’s signal.

§L.B.2 — The value’s fusion’s arithmetic

The value’s late fusion’s formula is the inverse-variance fusion, extended to the pose’s six dimensions:

where $T \ominus T_i$ is the pose difference in the tangent space’s coordinates and $\Sigma_i$ the source’s covariance. For the small differences the linearization is exact, and the closed form is

where $\xi_i$ is the source’s pose in the tangent space and $\hat{\xi}$ the fused pose.

The formula’s properties are the ones §L.B.1’s choice expects. The fused covariance’s diagonal is the sum of the inverses’ inverse — less than each source’s — and the source with the smallest variance dominates. The fused pose is the sources’ weighted mean, and the weights are the inverse variances.

The formula’s assumption is the sources’ independence. The RGB-D pose and the point-cloud pose share the camera’s noise and the frame’s error; their covariances are not independent, and the fused covariance is optimistic. The consistency check of §L.B.3 is the independence’s test: the fused estimate’s Mahalanobis distance from each source’s estimate is distributed as $\chi^2$ with the effective degrees of freedom, and the effective count is less than the source’s count when the sources are correlated.

The fused pose’s form is the distribution, not the point. The stack’s contract’s value is the fused distribution’s mean and covariance, and the consumer’s 3.3.3 affordance check’s tolerance is the distribution’s coverage.

§L.B.3 — The consistency checks

The consistency checks are the stack’s runtime’s failure detectors, and they are the 3.3.4 §L.B.2’s semantic layer’s complement at the perception’s level.

The cross-modal check. The RGB-D pose and the point-cloud pose are compared by the Mahalanobis distance

Under the independence’s assumption and the Gaussian’s model, $d_M^2$ is distributed as $\chi^2_6$, and the threshold at the 95% confidence is 12.59. A $d_M^2$ above the threshold is the sources’ disagreement, and it is the check’s failure’s signal.

The temporal check. The pose’s change between two consecutive observations is compared to the physical plausibility’s bound. The bound is the arm’s maximum velocity’s product with the observation’s period, plus the measurement’s noise’s allowance. A change above the bound is the jump, and it is the check’s second signal.

The freshness’s check. The pose’s age at the consumer’s request is compared to the consumer’s tolerance. The 3.3.1 reference check’s tolerance is the reasoning’s cadence’s half-period; the 2.3.1 grasp’s tolerance is the controller’s settling time’s fraction. A pose whose age exceeds the tolerance is the check’s third signal.

The frame’s check. The SLAM’s relative pose and the kinematic prior’s difference is compared to the drift’s bound. The bound is the 2.1.5 §L.E.2’s kinematic prior’s accuracy’s budget; a difference above the bound is the drift’s signal.

The four checks’ conjunction is the stack’s health, and the failure ladder of Part C is the conjunction’s response. The checks’ thresholds are the calibration’s numbers, and the calibration is the 4.1.3 suite’s strata’s replay: the thresholds are set so that the nominal episodes pass and the injected faults fail, and the 4.2.4 recovery’s rung’s selection’s data is the check’s failure’s signature.

§L.B.4 — The adaptation’s place

The 3.4.4 adaptation’s mapping is the value’s third source, and its place in the stack is the aligned observation’s source.

The mapping is applied to the simulated renderer’s image, and the aligned image is the input to a pose estimator whose training’s distribution is the real camera’s. The aligned pose is the third source’s estimate, and its covariance is the estimator’s on the aligned image.

The adaptation’s monitor is the stack’s input’s drift’s detector, and its MMD is computed on the raw camera’s images, not the aligned ones. The distinction matters: the mapping’s monitor measures the raw input’s alignment with the suite’s raw input’s distribution, and the aligned image’s distribution is the mapping’s output’s, which is not what the suite’s strata compare against.

The adaptation’s role in the fusion is the third source’s contribution, and its weight is the inverse variance’s. The mapping’s error is the source’s bias, and its covariance is the estimator’s on the aligned image; the bias is not in the covariance, and the 4.2.4 recovery’s rung’s selection uses the mapping’s MMD as the bias’s signal.

Self-explanation 3. Why is the mapping’s monitor on the raw images rather than the aligned ones?

Answer. Because the monitor’s purpose is the deployment’s distribution’s drift, and the deployment’s input is the raw camera’s image; the aligned image’s distribution is the mapping’s output’s, and its comparison to the suite’s raw images would measure the mapping’s fidelity, not the input’s drift.

Concept check. The cross-modal check’s $d_M^2$ is 15.2 and the threshold is 12.59. What is the check’s output, and what is the next check that fires?

Answer. The check’s output is the disagreement’s failure. The next check that fires is the temporal check if the disagreement’s jump exceeds the bound, or the frame’s check if the disagreement’s source is the frame; the provenance’s individual covariances are the data for the ordering.

Worked-example ladder.

WE-L.B.a (fully worked). The RGB-D pose’s $\xi_1 = [0.50, 0.20, 0.30, 0, 0, 0.785]$ and the point-cloud pose’s $\xi_2 = [0.505, 0.198, 0.301, 0, 0, 0.787]$. The covariances are $\Sigma_1 = \mathrm{diag}(0.0004, 0.0004, 0.0009, 0.0001, 0.0001, 0.0003)$ and $\Sigma_2 = \mathrm{diag}(0.0016, 0.0016, 0.0025, 0.0004, 0.0004, 0.0010)$. Compute the Mahalanobis $d_M^2$ and decide.

Solution. The difference $\xi_1 - \xi_2 = [-0.005, 0.002, -0.001, 0, 0, -0.002]$. The combined covariance’s diagonal is $0.0020, 0.0020, 0.0034, 0.0005, 0.0005, 0.0013$. The Mahalanobis is $0.005^2/0.002 + 0.002^2/0.002 + 0.001^2/0.0034 + 0 + 0 + 0.002^2/0.0013 = 1.25 \times 10^{-5}/0.002 + \dots = 0.0125 + 0.002 + 0.0003 + 0 + 0 + 0.0031 = 0.0179$. The $d_M^2$ is far below the 12.59 threshold, so the sources agree.

WE-L.B.b (completion). The point cloud’s pose is $\xi_2 = [0.53, 0.20, 0.30, 0, 0, 0.785]$. Recompute and decide.

Completion. The $x$‘s difference is $-0.03$, and the Mahalanobis’s $x$‘s term is $0.0009/0.002 = 0.45$; the total is still below the threshold, so the check passes. The completion shows the check is powered by the covariance’s size: a 3 cm difference with a 4 cm standard deviation is not a disagreement’s evidence.

WE-L.B.c (faded). Derive the fused pose’s covariance under the two sources’ independence and state the assumption’s failure’s consequence.

Faded step. The fused covariance’s diagonal is the inverse of the inverses’ sum; the assumption’s failure is the covariance’s optimistic understatement.

Completion. The fused covariance’s $x$-axis is $0.00032$ under the independence; the true’s is between the sources’ and the independence’s, and the check’s effective degrees of freedom are between 1 and 6.

WE-L.B.d (contrastive near-miss). A stack’s cross-modal check is disabled because the two sources’ disagreement “fires on the nominal data”. Why it fails: the check’s threshold was set for the correlated sources’ distribution’s degrees of freedom, and the correlated sources’ $d_M^2$ is chi-squared with fewer degrees of freedom — the threshold’s 12.59 is too high, and the nominal data’s $d_M^2$ can exceed it. The fix: the threshold’s calibration on the correlated sources’ distribution, or the check’s dimensions’ reduction’s effective count.

Comparative table: the three fusion levels.

Design principle P53.2. Fuse at the level the attribution’s requirement demands; the value’s late fusion preserves the provenance, and the frame’s comparison preserves the drift’s signal.

Consolidation checkpoint 2

(a) Three capabilities. You can state the three fusion levels and choose by attribution. You can compute the inverse-variance fusion and the fused covariance. You can run the cross-modal check and calibrate its threshold.

(b) Transfer probe. The two sources’ $d_M^2$ is 0.02 and the threshold is 12.59. What does the 0.02 indicate? Answer: the sources agree; the 0.02 is far below the threshold, and the sources’ agreement is the nominal’s case. The check’s power is the small $d_M^2$‘s evidence: the sources’ errors are not in disagreement, not that they are zero.

(c) Remediation rule. If you cannot state why the frame’s fusion is a comparison, re-read §L.B.1 before Part C.

Part C — Failure detection and recovery

§L.C.1 — The four failure modes

The stack’s failure modes are the contract’s four requirements’ violations’ sources, and each has a detection signal and a recovery’s rung.

The pose’s failure. The estimator returns a pose that is wrong beyond its covariance’s claim. The cause is the object’s occlusion, the reflective surface, or the training’s distribution’s gap. The detection is the cross-modal check’s $d_M^2$ — the point-cloud’s estimate disagrees — or the temporal check’s jump. The recovery is the re-observe’s rung.

The frame’s failure. The camera’s pose’s estimate has drifted, and the object’s pose inherits the drift. The cause is the 2.1.5 SLAM’s drift or the kinematic prior’s calibration’s change. The detection is the frame’s check — the SLAM’s estimate disagrees with the prior — or the cross-modal check’s systematic bias. The recovery is the re-observe’s rung with the drift’s correction.

The freshness’s failure. The pose’s age exceeds the consumer’s tolerance. The cause is the estimator’s latency’s variance or the scheduler’s miss. The detection is the freshness’s check’s timestamp. The recovery is the degrade’s rung: the consumer uses the pose at the coarser accuracy the staleness implies, or the consumer’s request is delayed.

The provenance’s failure. The sources’ agreement’s data is missing, and the recovery’s rung cannot be selected. The cause is a source’s absence or a logging’s failure. The detection is the provenance’s completeness’s check. The recovery is the abstain’s rung: the stack reports the failure’s kind — “provenance unavailable” — and the consumer’s 3.3.4 §L.B.2’s semantic layer’s check rejects the downstream action.

The four modes are the contract’s requirements’ violations, and the recovery’s ladder is the modes’ three responses: the re-observe, the degrade, and the abstain.

§L.C.2 — The detection signals

The checks of §L.B.3 are the detection’s mechanisms, and their signals’ combination is the mode’s diagnosis.

The cross-modal’s $d_M^2$. A large $d_M^2$ is the pose’s failure’s primary signal; the which source is wrong is the provenance’s individual covariances’ comparison — the source with the larger deviation from the fused value is the suspect.

The temporal’s jump. A jump above the bound is the pose’s failure’s secondary signal; the jump’s direction is the physical plausibility’s test, and a jump toward the divider is the collision’s signature.

The freshness’s age. The age’s exceeding the tolerance is the freshness’s failure’s only signal; the age’s pattern — a persistent rise, an occasional spike — is the scheduler’s diagnosis’s data.

The frame’s disagreement. The SLAM’s and the prior’s difference is the frame’s failure’s signal; the difference’s drift — a slow rise over time — is the drift’s signature, and the difference’s jump is the prior’s calibration’s change.

The provenance’s completeness. The sources’ list’s coverage is the provenance’s failure’s signal; a missing source’s entry is the failure’s detection, and the missing entry’s pattern — the same source failing repeatedly — is the source’s fault’s diagnosis.

The five signals are the detection’s mechanisms, and their thresholds are the 4.1.3 suite’s replay’s calibration. The signals’ joint distribution is the stack’s health’s state, and the 4.2.4 recovery’s rung’s selection is the state’s function.

§L.C.3 — The failure ladder

The failure ladder is the stack’s three rungs, and each rung is a different claim’s accuracy.

The re-observe rung. The stack requests a new observation, and the pose is recomputed. The rung’s claim is that the failure was transient — the occlusion, the reflection, the one-frame’s jump — and the new observation’s pose is the correct one. The rung’s cost is the re-observation’s latency, and its acceptance is the new pose’s passing the checks.

The degrade rung. The stack reports the pose at the coarser accuracy the sources’ staleness or disagreement implies. The rung’s claim is that the failure is persistent and the coarser accuracy is still usable. The rung’s output is a pose with a larger covariance, and the consumer’s tolerance’s check is the 3.3.3 affordance’s robustness’s test.

The abstain rung. The stack reports the failure’s kind and refuses to deliver a pose. The rung’s claim is that the failure’s accuracy is not usable and the consumer’s action would be unsafe. The rung’s output is the failure’s record, and the consumer’s 3.3.4 §L.B.2’s semantic layer’s check rejects the action.

The ladder’s order is the re-observe, the degrade, the abstain, and the order is by cost. The re-observe’s latency is the smallest’s cost; the degrade’s larger covariance is the middle’s; the abstain’s action’s rejection is the largest’s. The order’s discipline is the first rung that succeeds: a re-observe that passes the checks is the acceptance, and the degrade is used only when the re-observe’s failure persists.

The ladder’s trigger is the check’s failure’s kind. A cross-modal’s failure triggers the re-observe; a freshness’s failure triggers the degrade; a provenance’s failure triggers the abstain. The trigger’s mapping is the 4.2.4 lesson’s design’s input, and the running case’s mapping is the §L.E.2’s table.

Self-explanation 1. Why is the abstain the last rung rather than the first?

Answer. Because the abstain’s claim is the failure’s accuracy’s unusability, and the claim is only known after the re-observe and the degrade have failed; the abstain’s earlier use would refuse actions the re-observe would have saved, and the refusal’s cost is the 4.2.4 §L.D’s availability’s loss.

Concept check. The cross-modal check fails at $t$, the re-observe’s pose passes at $t+1$, and the temporal check fails at $t+1$. Which rung is the failure’s response, and what is the failure’s kind?

Answer. The re-observe succeeded on the cross-modal, and the temporal check’s failure is a new failure — the jump’s kind. The failure’s kind is the temporal’s, and the rung is the re-observe again, with the temporal check’s failure’s source’s diagnosis.

Worked-example ladder.

WE-L.C.a (fully worked). The stack’s cross-modal check fails at $d_M^2 = 18.3$, and the re-observe’s pose’s $d_M^2$ is 4.2. Compute the ladder’s rung and the failure’s kind.

Solution. The re-observe’s $d_M^2$ is below the threshold, so the re-observe’s rung succeeds. The failure’s kind is the pose’s transient’s (the occlusion’s or the reflection’s), and the ladder’s outcome is the pose’s delivery with the re-observe’s value.

WE-L.C.b (completion). The re-observe’s $d_M^2$ is 14.1, above the threshold. What is the next rung, and what is the failure’s kind?

Completion. The re-observe’s failure triggers the degrade’s rung, and the failure’s kind is the pose’s persistent’s. The degrade’s pose’s covariance is inflated to the sources’ disagreement’s scale, and the consumer’s tolerance’s check is the 3.3.3 affordance’s robustness’s test.

WE-L.C.c (faded). Design the trigger’s mapping’s table for the five checks and the three rungs.

Faded step. Cross-modal → re-observe; temporal → re-observe; freshness → degrade; frame → re-observe with drift’s correction; provenance → abstain.

Completion. The trigger’s mapping is the failure’s kind’s function, and the mapping’s design’s choice is the 4.2.4 lesson’s, not the stack’s.

WE-L.C.d (contrastive near-miss). A stack’s freshness’s failure triggers the re-observe, and the re-observe’s latency makes the freshness’s failure worse. Why it fails: the re-observe’s latency increases the age, and the freshness’s failure is the age’s existing problem; the re-observe’s rung is the wrong trigger for the freshness’s kind. The fix: the trigger’s mapping’s freshness → degrade, per the table.

Comparative table: the ladder’s rungs.

Design principle P53.3. Trigger the ladder by the failure’s kind, not by the check’s identity; the freshness’s failure is the age’s, and the re-observe’s rung is the age’s increase.

Consolidation checkpoint 3

(a) Three capabilities. You can name the four failure modes and their detection signals. You can lay out the ladder’s three rungs and their claims. You can map the triggers to the rungs.

(b) Transfer probe. The frame’s check fails with a 4 cm disagreement. Which rung, and what is the failure’s kind? Answer: the re-observe with the drift’s correction; the failure’s kind is the frame’s, and the re-observe’s value is the corrected pose. The degrade’s rung is the next if the re-observe’s disagreement persists.

(c) Remediation rule. If you cannot state why the freshness’s failure triggers the degrade rather than the re-observe, re-read §L.C.3 before Part D.

Part D — Latency and scheduling

§L.D.1 — The budget’s arithmetic

The 4.1.1 §L.B.3’s perception budget was 30 ms. The budget’s arithmetic is the sources’ latencies’ sum, with the sharing’s structure.

The RGB-D estimator’s latency. The 2.1.2 inference is 30 ms on the deployment’s hardware — the budget’s full allocation. The estimator’s inference is the stack’s dominant cost.

The point-cloud registration’s latency. The 2.1.4 registration is 8 ms on the downsampled cloud. The registration’s pipeline includes the cloud’s construction from the depth’s image, which is 4 ms, so the registration’s total is 12 ms.

The SLAM’s latency. The 2.1.5 relative-pose’s estimate is 3 ms per frame; the estimate is amortized over the two frames’ comparison, and the per-frame’s cost is 1.5 ms.

The kinematic prior’s latency. The 1.1.1 forward-kinematics’s evaluation is under 1 ms.

The adaptation’s latency. The 3.4.4 mapping is applied to the training’s images, not the deployment’s; the deployment’s mapping’s latency is zero, per 3.4.4 §L.A.3.

The checks’ latency. The four checks’ arithmetic is under 1 ms.

The scheduler’s latency. The scheduler’s holding and dispatch is under 1 ms.

The sum is $30 + 12 + 1.5 + 1 + 0 + 1 + 1 = 46.5$ ms, which exceeds the 30 ms budget. The sum’s structure is the resolution: the RGB-D estimator’s 30 ms and the point cloud’s 12 ms are not on the same frame’s critical path if the scheduler runs them on alternating frames — the estimator on the even frames, the registration on the odd frames — and the fused pose’s per-frame’s age is the one-frame’s latency, not the sum’s.

The scheduler’s design is the budget’s constraint’s answer.

§L.D.2 — The scheduling’s design

The scheduler’s design has three levels, and each is a different cadence.

The per-frame’s cadence. Every frame is timestamped and the checks’ freshness’s data is recorded. The per-frame’s cost is the timestamp and the check’s arithmetic, under 1 ms.

The per-source’s cadence. The RGB-D estimator runs on every frame at 10 Hz — 30 ms of the 100 ms period — and the point-cloud registration runs on every other frame at 5 Hz — 12 ms every 200 ms. The alternating’s cadence is the cross-modal check’s cadence: the check compares the two sources’ estimates at the frames where both exist, and the check’s period is 200 ms.

The per-cycle’s cadence. The fused pose is delivered at the consumer’s request’s cadence, which is the 3.3.1 reasoning’s 10 Hz. The scheduler’s hold is the pose’s cache between the source’s completion and the consumer’s request, and the hold’s age is the freshness’s data.

The scheduling’s trade is the estimator’s frame’s cadence against the registration’s. The estimator’s 10 Hz is the value’s precision’s cadence; the registration’s 5 Hz is the check’s cadence, and the check’s 200 ms period is the failure’s detection’s latency. A faster registration’s cadence — every frame — would be the check’s better latency and the budget’s 12 ms’s addition, which the 30 ms’s budget does not allow.

The scheduler’s worst case is the frame where the estimator’s inference spikes to 50 ms. The stack’s hold is the previous frame’s pose, and the freshness’s age is 100 ms; the consumer’s tolerance is the 3.3.1 reference’s 50 ms, and the freshness’s failure fires. The degrade’s rung is the response, and the degraded pose’s covariance’s inflation is the 3.3.3 affordance’s robustness’s test.

§L.D.3 — The tail’s management

The budget’s number is the 95th percentile’s latency, not the mean’s, and the tail’s management is the stack’s design’s choice.

The sources’ tail. The estimator’s latency’s distribution is heavy-tailed because the inference’s time depends on the image’s content. The 95th percentile is about 45 ms, against the mean’s 30 ms; the tail’s 15 ms’s excess is the scheduler’s margin’s source.

The scheduler’s budget. The budget’s 30 ms is the mean’s allocation, and the tail’s 45 ms is the 95th’s. The scheduler’s hold is the mechanism that absorbs the tail: the pose is held for up to 50 ms (the 3.3.1 reference’s tolerance), and the tail’s 45 ms is within the hold.

The consumer’s tolerance. The 3.3.1 reference’s 50 ms and the 2.3.1 grasp’s 20 ms are the consumers’ tolerances, and the stack’s 95th percentile’s age is the two tolerances’ comparison. The 95th’s 45 ms passes the reasoning’s 50 ms and fails the grasp’s 20 ms; the scheduler’s separate cadences for the two consumers — the reasoning’s 10 Hz and the grasp’s 5 Hz with the dedicated estimator’s pass — is the design’s answer.

The degradation’s policy. The scheduler’s policy is the priority’s: the grasp’s request is the higher priority, and the estimator’s pass for the grasp’s pose preempts the reasoning’s. The preemption’s cost is the reasoning’s age’s rise, and the rise’s bound is the 50 ms’s tolerance.

The tail’s management’s metric is the 4.1.3 suite’s replay’s 95th percentile, and the 4.2.4 recovery’s design’s input is the tail’s exceedance’s pattern.

Self-explanation 2. Why is the tail’s management a scheduling’s problem rather than an optimization’s?

Answer. Because the tail’s source is the content’s dependence — the small object’s inference is slower — and the optimization’s average’s reduction does not reduce the small object’s time; the scheduler’s hold and the preemption’s priority are the mechanisms that absorb the content’s variation, and the optimization is the mean’s.

Concept check. The stack’s 95th percentile’s age is 48 ms and the reasoning’s tolerance is 50 ms. Is the stack’s budget satisfied?

Answer. The 48 ms is within the 50 ms, and the budget is satisfied for the reasoning. The grasp’s tolerance is 20 ms, and the 48 ms fails it; the stack’s budget is the consumers’ tolerances’ conjunction, and the grasp’s preemption’s design is the requirement.

Worked-example ladder.

WE-L.D.a (fully worked). The RGB-D estimator’s mean is 30 ms with a 95th’s 45 ms. The reasoning’s tolerance is 50 ms and the grasp’s is 20 ms. Compute each consumer’s margin at the mean and at the 95th.

Solution. The reasoning’s margin at the mean is 20 ms and at the 95th is 5 ms; the grasp’s margin at the mean is $-10$ ms and at the 95th is $-25$ ms. The grasp’s margin is negative at both, and the stack’s design must preempt and dedicate a pass for the grasp.

WE-L.D.b (completion). The dedicated pass’s latency is the estimator’s 30 ms with a 95th’s 45 ms, and the grasp’s request’s cadence is 5 Hz. Compute the grasp’s margin at the dedicated pass and state the preemption’s cost.

Completion. The dedicated pass’s margin at the mean is $-10$ ms and at the 95th is $-25$ ms; the margin is still negative, because the estimator’s latency itself is above the grasp’s tolerance. The preemption does not help the estimator’s latency’s fact, and the grasp’s pose is the estimator’s at its own latency; the consumer’s tolerance’s 20 ms is below the estimator’s mean, and the estimator’s optimization is the requirement.

WE-L.D.c (faded). Design the scheduler’s hold’s bound from the reasoning’s tolerance and the estimator’s tail.

Faded step. The hold’s bound is the reasoning’s 50 ms minus the estimator’s 95th’s 45 ms, which is 5 ms; the hold’s slack is the 5 ms.

Completion. The hold’s bound’s 5 ms is the margin’s small, and the estimator’s tail’s reduction is the hold’s enlargement’s prerequisite; the stack’s design’s both the tail’s optimization and the hold’s bound are required.

WE-L.D.d (contrastive near-miss). A team’s stack’s mean is 28 ms and its 95th is 65 ms; the team reports the mean and concludes the budget is satisfied. Why it fails: the 95th’s 65 ms exceeds the reasoning’s 50 ms, and the tail’s failures are the deployment’s. The team’s mean’s margin is the tail’s disguise. The fix: the 95th’s report and the tail’s management, per §L.D.3.

Comparative table: the scheduling’s cadences.

Design principle P53.4. Budget the 95th percentile and schedule the sources’ cadences; the mean’s margin is the tail’s disguise, and the content’s variation is the scheduler’s problem.

Consolidation checkpoint 4

(a) Three capabilities. You can compute the sources’ latency’s sum and the alternating cadence’s per-frame’s age. You can state the scheduler’s three cadences. You can compute the consumer’s margin at the mean and the 95th.

(b) Transfer probe. The estimator’s 95th is 45 ms and the reasoning’s tolerance is 50 ms. Is the stack’s budget satisfied for the reasoning? Answer: yes, with a 5 ms margin; the margin is small, and the estimator’s tail’s reduction is the margin’s enlargement’s prerequisite.

(c) Remediation rule. If you cannot state why the 95th rather than the mean is the budget’s number, re-read §L.D.3 before Part E.

Part E — The running case

§L.E.1 — The stack’s assembly

The running case’s stack, assembled from the parts.

The contract. The five fields of §L.A.1, with the running case’s values: the cube’s pose in the world frame $W$, the covariance’s $6 \times 6$, the freshness’s bound at the reasoning’s 50 ms and the grasp’s 20 ms, the frame’s $W$‘s definition, and the provenance’s three sources’ list.

The fusion. The late fusion of the RGB-D pose, the point-cloud pose, and the adaptation’s aligned pose, with the inverse-variance’s weights and the fused covariance’s $6 \times 6$. The frame’s comparison of the SLAM’s relative pose and the kinematic prior.

The checks. The cross-modal’s Mahalanobis at the 12.59 threshold and the correlated sources’ effective degrees of freedom’s recalibration; the temporal’s plausibility’s bound at the arm’s maximum velocity’s 10 cm/s and the observation’s period’s 100 ms, so 1 cm plus the noise’s allowance; the freshness’s two tolerances; the frame’s drift’s bound at the 2.1.5 §L.E.2’s 5 mm.

The ladder. The triggers’ mapping’s table, with the cross-modal → re-observe, the temporal → re-observe, the freshness → degrade, the frame → re-observe with the drift’s correction, and the provenance → abstain.

The scheduler. The estimator’s 10 Hz and the registration’s 5 Hz’s alternating cadence; the reasoning’s 10 Hz hold’s 5 ms’s slack; the grasp’s preemption’s dedicated pass and the estimator’s tail’s reduction’s requirement.

§L.E.2 — The integration’s evidence

The stack’s evidence is the 4.1.3 suite’s replay’s four checks’ statistics, the 95th percentile’s age, and the ladder’s rung’s distribution.

The checks’ statistics. The suite’s 400 episodes’ replay’s cross-modal’s $d_M^2$‘s distribution’s 95th percentile at the nominal’s 3.8, below the 12.59 threshold; the temporal’s jump’s 95th at 0.6 cm, below the 1 cm bound; the freshness’s age’s 95th at 48 ms, within the reasoning’s 50 ms; the frame’s disagreement’s 95th at 2 mm, below the 5 mm bound.

The 95th percentile’s age. The suite’s replay’s 95th at 48 ms, within the reasoning’s tolerance and above the grasp’s; the grasp’s dedicated pass’s 95th at 45 ms, above the grasp’s 20 ms; the estimator’s optimization’s requirement is the 4.2.4 lesson’s input.

The ladder’s rung’s distribution. The suite’s replay’s ladder’s use is 12 re-observes, 3 degrades, and 1 abstain across the 400 episodes. The rung’s distribution’s acceptance is the 4.2.4 recovery’s design’s input, and the 1 abstain’s episode is the register’s entry.

§L.E.3 — The register’s entries

The perception’s register’s four entries.

The register’s entries are the 3.3.4 §L.E.3’s five fields, and the residuals are the perception’s acceptance’s residuals.

The pose’s failure’s residual is the occlusion’s persistence — the object is behind the divider and no re-observe helps; the 3.3.3 §L.D.2’s semantic layer’s check is the fallback.

The frame’s drift’s residual is the drift’s unmonitored axis — the roll’s drift is not in the check’s three axes; the 3.4.4 §L.C.3’s loop’s monitor is the complement.

The freshness’s age’s residual is the tail’s exceedance — the estimator’s 99th exceeds the hold’s bound; the 4.2.4 §L.D’s recovery’s policy is the complement.

The provenance’s absence’s residual is the logging’s gap — a source’s entry is missing due to a logging’s failure; the 4.2.4 §L.E’s audit is the complement.

Self-explanation 3. Why is the occlusion’s persistence a residual rather than a mitigation’s complete coverage?

Answer. Because the re-observe’s latency cannot resolve an object that is behind the divider, and the semantic layer’s check is the fallback’s mechanism, not the perception’s; the residual is what the perception’s stack does not cover, and the fallback is the 4.2.3 reasoning’s design’s.

Worked-example ladder.

WE-L.E.a (fully worked). The suite’s replay’s cross-modal’s $d_M^2$‘s 95th is 3.8, and the threshold is 12.59. Compute the margin and state whether the threshold’s calibration is sound.

Solution. The margin is $12.59 - 3.8 = 8.79$, which is 70% of the threshold; the nominal’s $d_M^2$ is far below the threshold, and the threshold’s calibration is sound for the suite’s distribution. The margin’s large is the check’s power’s cost: the sources’ agreement’s 3.8 is the nominal’s, and the failure’s 18.3 is the fail’s, and the check’s separation is the 14.5’s gap.

WE-L.E.b (completion). The suite’s replay’s freshness’s age’s 95th is 48 ms and the reasoning’s tolerance is 50 ms. Compute the margin and state the acceptance.

Completion. The margin is 2 ms, which is within the reasoning’s tolerance and very small; the acceptance is marginal, and the estimator’s tail’s reduction is the margin’s enlargement’s requirement. The register’s freshness entry’s residual is the tail’s exceedance, which is the 5% of episodes above the 95th.

WE-L.E.c (faded). Write the register’s entry for the pose’s failure with the occlusion’s persistence’s residual.

Faded step. Failure: the occlusion; probability: moderate; consequence: the grasp’s miss; mitigation: the cross-modal’s check and the re-observe; residual: the occlusion’s persistence.

Completion. The residual’s acceptance is the 3.3.3 §L.D.2’s semantic layer’s check’s fallback, and the fallback is the 4.2.3 reasoning’s design’s.

WE-L.E.d (contrastive near-miss). A team’s stack’s 95th age is 48 ms and the reasoning’s tolerance is 50 ms; the team reports the acceptance’s margin at the mean’s 30 ms and the 20 ms’s margin. Why it fails: the mean’s margin is the tail’s disguise, and the 95th’s 2 ms margin is the actual margin. The fix: the 95th’s report and the tail’s management, per §L.D.3.

Comparative table: the stack’s evidence.

Design principle P53.5. Assemble the stack with its contract, its fusion, its checks, its ladder, and its schedule; the evidence is the four checks’ statistics and the 95th’s age, and the register’s residuals are the stack’s acceptance.

Consolidation checkpoint 5

(a) Three capabilities. You can state the running case’s stack’s five components. You can read the evidence’s table and the margins. You can write the register’s four entries with residuals.

(b) Transfer probe. The freshness’s 95th is 48 ms and the threshold is 50 ms. Is the check’s calibration sound? Answer: the 2 ms margin is small, and the check’s false-alarm rate is the 5% of episodes above the 95th, which the check will flag as failures; the check’s calibration is tight, and the estimator’s tail’s reduction is the requirement.

(c) Remediation rule. If you cannot state why the register’s residual is the perception’s acceptance, re-read §L.E.3 before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: the priority’s preemption, the check’s axes, and the rung’s acceptance’s loose.

The lab

Task. Implement the perception stack’s contract, its late fusion, and its cross-modal check for the running case’s cube, and run the suite’s replay’s four checks’ statistics with a fault injector. Measure the fused covariance’s reduction over the single source and the check’s separation between the nominal and the faulted distributions.

Conditions. Python 3.10+, NumPy. The cube’s true pose is $T_W^{Obj} = (R_z(\pi/4), [0.5, 0.2, 0.3])$; the RGB-D estimator’s noise is the 2.1.2 covariance; the point cloud’s noise is the 2.1.4 covariance with a 3 mm bias on 5% of frames. Seed all RNGs.

Controls. (i) The single-source RGB-D; (ii) the late fusion of the two sources; (iii) the cross-modal’s check with the independence’s threshold; (iv) the check with the correlated sources’ recalibrated threshold.

Metrics.

Pre-registered hypotheses.

H1. The fused covariance’s $x$-axis is below the single source’s 0.0004, because the two sources’ inverse-variance fusion reduces the variance.

H2. The cross-modal’s nominal’s 95th is below 12.59 and the faulted’s 95th is above, because the 3 mm bias on the 5% of frames is the check’s signal.

H3. The correlated sources’ recalibrated threshold reduces the false alarm rate below the independence’s, because the correlated sources’ degrees of freedom are fewer.

H4. The check’s separation’s ratio is at least 2, and the check’s power is the sources’ independent errors’ count.

Break-it sequence.

1. Disable the cross-modal check silently. The faulted frames’ pose is delivered as the correct, and the downstream grasp misses. Silent: the checks’ logs are empty.
2. Set the threshold to the independence’s degrees of freedom silently. The correlated sources’ nominal’s $d_M^2$ fires the check on the nominal’s 15% of frames. Silent: the threshold looks correct.
3. Report the mean’s age silently. The 95th’s age is the deployment’s failure’s, and the mean’s margin hides it. Silent: the mean’s number is plausible.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The contract’s five fields are each a value with a check. (2) The fusion’s covariance is computed per axis and the reduction’s magnitude is reported. (3) The checks’ thresholds are calibrated on the suite’s replay and the false alarm rate is measured. (4) The ladder’s rung’s distribution is reported, not only the successes. (5) The 95th’s age is the budget’s number and the mean is the context.

A runnable core of the fusion and the cross-modal check, seeded and annotated:

# lab_4_2_1.py — perception stack: fusion, cross-modal check, covariance reduction
import numpy as np
np.random.seed(0)

# Cube's true pose (x, y, z, roll, pitch, yaw)
TRUE = np.array([0.5, 0.2, 0.3, 0.0, 0.0, np.pi/4])

# Source covariances: RGB-D (2.1.2) and point cloud (2.1.4)
SIGMA_RGB = np.diag([0.0004, 0.0004, 0.0009, 0.0001, 0.0001, 0.0003])
SIGMA_PC  = np.diag([0.0016, 0.0016, 0.0025, 0.0004, 0.0004, 0.0010])

def sample_source(true, sigma, bias=None, n=1):
    x = np.random.multivariate_normal(true, sigma, size=n)
    if bias is not None:
        x += bias
    return x

def fuse(x1, x2, s1, s2):
    """Inverse-variance fusion (Eq. B.2), per-axis since the covariances are diagonal."""
    w1 = np.diag(s1) ** -1
    w2 = np.diag(s2) ** -1
    fused_var = 1.0 / (w1 + w2)
    fused_mean = fused_var * (w1 * x1 + w2 * x2)
    return fused_mean, np.diag(fused_var)

def cross_modal(x1, x2, s1, s2):
    """Mahalanobis distance (Eq. B.3)."""
    d = x1 - x2
    S = s1 + s2
    return float(d @ np.linalg.inv(S) @ d)

N = 400
nominal = []
faulted = []
for i in range(N):
    x1 = sample_source(TRUE, SIGMA_RGB)[0]
    bias = np.array([0.003, 0, 0, 0, 0, 0]) if i % 20 == 0 else None
    x2 = sample_source(TRUE, SIGMA_PC, bias=bias)[0]
    nominal.append(cross_modal(x1, x2, SIGMA_RGB, SIGMA_PC))
    if bias is not None:
        faulted.append(cross_modal(x1, x2, SIGMA_RGB, SIGMA_PC))

print(f"nominal $d_M^2$ 95th = {np.percentile(nominal, 95):.2f}")
print(f"faulted $d_M^2$ 95th = {np.percentile(faulted, 95):.2f}")

# Fused covariance's reduction
_, fused_var = fuse(TRUE, TRUE, SIGMA_RGB, SIGMA_PC)
print(f"RGB-D x-var = {SIGMA_RGB[0,0]:.4f}  fused x-var = {fused_var[0]:.4f}")
print(f"reduction = {1 - fused_var[0]/SIGMA_RGB[0,0]:.1%}")

# Contract's five fields
contract = {
    "pose": TRUE.tolist(),
    "covariance_diag": fused_var.tolist(),
    "freshness_ms": 48,
    "frame": "W (table surface, arm base)",
    "provenance": ["2.1.2 RGB-D", "2.1.4 point cloud", "3.4.4 aligned"],
}
for k, v in contract.items():
    print(f"  {k}: {v}")

Line 5 is the cube’s true pose; line 8–9 the two sources’ covariances. Line 11–14 sample a source with an optional bias; line 16–22 the inverse-variance fusion; line 24–28 the cross-modal check. Lines 30–38 run the suite’s replay with the 5% faulted frames; lines 40–42 the check’s separation; lines 44–46 the fused covariance’s reduction; lines 48–56 the contract’s five fields.

Cheat sheet

Exercises with solutions

E 4.2.1.1. State the contract’s five requirements and the failure mode each guards.

S 4.2.1.1. Pose (the wrong value), covariance (the wrong uncertainty), freshness (the staleness), frame (the ambiguity), provenance (the unattributable). The lesson: the value alone is one of five, and the other four are the integration’s.

E 4.2.1.2 (transfer). The 2.1.2 estimator’s covariance is 0.0004 in $x$, and the true error’s variance is 0.0010. Which requirement fails, and what is the consequence?

S 4.2.1.2. The covariance’s correctness; the Mahalanobis check passes when it should fail, and the fused covariance inherits the understatement. The lesson: the covariance’s correctness is the check’s validity’s premise.

E 4.2.1.3. Compute the fused covariance’s $x$-axis for $\Sigma_1 = 0.0004$ and $\Sigma_2 = 0.0016$.

S 4.2.1.3. $(2500 + 625)^{-1} = 0.00032$. The lesson: the reduction is 20%, less than the naive average’s 50%, because the smaller variance dominates.

E 4.2.1.4 (diagnostic). A team disables the cross-modal check and the grasp misses on 5% of the episodes. Diagnose and fix.

S 4.2.1.4. The pose’s failure is undetected; the check is the detection’s mechanism, and its absence makes the failure silent. The fix is the check’s re-enabling with the calibrated threshold. The lesson: the checks are the runtime’s failure detectors, not the debugging’s tools.

E 4.2.1.5. Name the four failure modes and their detection signals.

S 4.2.1.5. Pose (cross-modal’s $d_M^2$, temporal’s jump), frame (SLAM-prior’s disagreement), freshness (age’s timestamp), provenance (completeness’s check). The lesson: each failure’s signal is the requirement’s violation’s measure.

E 4.2.1.6 (transfer). The freshness’s failure fires and the re-observe’s latency makes the age worse. Diagnose and fix.

S 4.2.1.6. The trigger’s mapping is wrong; the freshness’s failure is the age’s, and the re-observe’s rung increases it. The fix is the degrade’s rung, per §L.C.3. The lesson: the trigger’s mapping is by the failure’s kind, not the check’s identity.

E 4.2.1.7 (diagnostic). The stack’s 95th age is 48 ms and the reasoning’s tolerance is 50 ms. Is the budget satisfied?

S 4.2.1.7. Yes, with a 2 ms margin; the margin is small and the tail’s exceedance is the register’s residual. The lesson: the budget’s number is the 95th, and the margin’s small is the acceptance’s tight.

E 4.2.1.8. State why the frame’s fusion is a comparison rather than an average.

S 4.2.1.8. Because the frame’s sources are the drift’s detectors, and the average hides the disagreement. The lesson: the comparison’s output is the drift’s signal, and the average would remove it.

E 4.2.1.9 (transfer). The suite’s replay’s ladder’s distribution is 12 re-observes, 3 degrades, 1 abstain. What does the 1 abstain’s episode’s register’s entry’s residual state?

S 4.2.1.9. The occlusion’s persistence or the provenance’s gap, depending on the episode’s kind; the residual is the perception’s stack’s non-coverage, and the 3.3.3 §L.D.2’s semantic layer’s check is the fallback. The lesson: the register’s residual is the stack’s acceptance’s non-coverage.

Exit test

You are ready for Lesson 4.2.2 when you can, without notes:

1. State the contract’s five requirements and the failure mode each guards.
2. Explain the single-source’s ceiling’s four failures.
3. Name the seven sources and their best requirements.
4. State the three fusion levels and choose by the attribution’s requirement.
5. Compute the inverse-variance fusion and the fused covariance.
6. Run the cross-modal’s Mahalanobis check and calibrate its threshold.
7. State why the frame’s fusion is a comparison.
8. Name the four failure modes and their detection signals.
9. Lay out the ladder’s three rungs and their claims.
10. Map the triggers to the rungs by the failure’s kind.
11. Compute the 95th’s margin from the sources’ budgets.
12. State the scheduler’s three cadences and the hold’s bound.
13. Assemble the running case’s stack’s five components.
14. Read the evidence’s table and the register’s residuals.
15. State why the 95th’s age, not the mean’s, is the budget’s number.

Cross-references. 4.1.1 supplies the interface contract and the 30 ms perception budget restated in §L.A.1 and §L.D.1. 4.1.3 supplies the suite and the replay’s calibration restated in §L.B.3 and §L.E.2. 2.1.1 supplies the projection restated in §L.A.2. 2.1.2 supplies the pose estimator and its covariance restated in §L.A.2 and §L.B.2. 2.1.4 supplies the point-cloud registration restated in §L.A.3. 2.1.5 supplies the kinematic prior and the SLAM restated in §L.A.3 and §L.B.1. 3.4.4 supplies the adaptation and the monitor restated in §L.B.4 and §L.D.3. Lesson 4.2.2 (the policy layer) consumes the contract’s outputs. Lesson 4.2.3 (the reasoning layer) consumes the scene’s serialization built from them. Lesson 4.2.4 (safety and recovery) consumes the failure ladder. Module 4.3 uses the checks as its diagnostic’s basis.

Ready. The next lesson to write is 4.2.2 — Policy Layer. Say “next” to generate it.