Lesson 3.4.4 — Real-to-Sim Adaptation

Week 30

Prerequisite lessons: 3.4.1 (the reality gap as a spectrum), 3.4.2 (domain randomization), 3.4.3 (system identification), 3.2.1 (latent world models), 3.1.4 (VLA fine-tuning), 3.3.4 (safety and constraints on LLM outputs).

Prior results used in this lesson. From 3.4.1: the four divergence axes, the twin-track’s tail metric, and the spectrum whose far end this lesson occupies. From 3.4.2: the randomization families and the performance-robustness curve whose third regime is the conservatism cost. From 3.4.3: the identified parameters as the simulator’s nominal values and the structure-error allowance as the range’s width. From 3.2.1: the ELBO’s reconstruction term and the latent-space training pattern. From 3.1.4: the shadow deployment, the decision conjunction, and the rollback triggers. From 3.3.4: the residual-risk register. Each is restated at the point of use.

What this lesson hands forward. The adaptation-family taxonomy, the cycle-consistency objective, the distribution-alignment measurement, and the closed adaptation loop. Module 4.2 (building the full-stack agent) uses the adaptation as the perception stack’s final calibration step. Module 4.3 (iteration and failure analysis) uses the adaptation’s cycle as its diagnostic template. Module 4.4 (the master’s defense) uses the residual-risk register’s adaptation entries.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, and divider are unchanged; this lesson takes the 3.4.1 §L.E’s perception axis — the dominant 45% of the sim-to-real tail — and fits a mapping from the simulated observation to the real one, so that the policy trained on the mapped observation transfers to the real cell.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) state why adaptation is the method for the unmodelled axes and why it is often the first method; (2) place the four adaptation families — pixel translation, feature alignment, residual correction, and differentiable rendering — on the two-axis spectrum; (3) write the cycle-consistency objective and explain both of its directions; (4) state the adversarial and reconstruction objectives’ roles and their failure modes; (5) design the closed adaptation loop and its acceptance gate; (6) measure the distribution alignment with a two-sample test and interpret the result; (7) diagnose the five adaptation pitfalls from their signatures; (8) state the mapping’s own error and the policy’s trust in it; (9) assemble the running case’s adaptation cycle with data, training, and validation; (10) write the register’s adaptation entry with its residual.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 3.4.1 §L.B.3, the axis decomposition attributed 45% of the running case’s tail to the perception axis. What does that attribution imply about the other methods’ sufficiency?

Answer. It implies that the identification and the randomization narrow the dynamics and actuation axes and leave the perception axis’s 45% untouched; the 3.4.1 §L.D.2’s sequence puts the perception’s method first because it is the dominant contributor and it is the only method that narrows an unmodelled axis. Used in §L.A.1 to justify adaptation’s role.

Q2. In 3.4.2, the randomization’s appearance family trained a policy across a distribution of rendered textures. Why was that a randomization and not an adaptation?

Answer. Because the randomization samples the appearance and trains the policy to be robust across the samples; it does not fit a mapping from the simulated appearance to the real one. The adaptation fits the mapping, and the randomization’s family is the interim measure the adaptation replaces. Used in §L.A.2 to distinguish the two methods.

Q3. In 3.4.3, the identification’s intervals were the statistical uncertainty and the structure-error allowance widened them. What is the adaptation’s analogue of the structure-error allowance?

Answer. The mapping’s own residual — the error the mapping makes on the real cell’s images that the policy then trusts. Used in §L.B.2 to name the mapping’s error’s role.

Q4. In 3.3.4, the residual-risk register’s entries had a residual column that recorded what the mitigation did not cover. What is the register’s entry for the adaptation’s mapping’s error?

Answer. Failure: the mapping’s residual misaligns the task-relevant statistics; probability from the alignment test; consequence is the policy’s actions being chosen for the wrong observation; mitigation is the alignment measurement; residual is the mapping’s error’s own drift. Used in §L.E.3 to write the entry.

Part A — What adaptation is

§L.A.1 — The unmodelled axis’s method

The 3.4.1 §L.B.4 result was that a modelled axis with an assumed parameter widens the gap, because the assumption’s error is trusted. The perception axis is the extreme case: the rendering’s realism is not a parameter that identification can fit, because it is a high-dimensional property of the renderer’s textures, lighting, and noise model, and the renderer’s configuration exposes only a few of its dimensions.

Real-to-sim adaptation is the method that addresses this case. Its idea is to leave the simulator’s physics alone and fit a mapping between the simulator’s observation and the real observation, so that the policy trained on the mapped observation sees the real cell’s distribution.

The formal statement is: find a mapping $g_\omega: \mathcal{O}_{sim} \to \mathcal{O}_{real}$ (or the inverse) such that the distribution $g_\omega \circ P_{sim}(o)$ is close to $P_{real}(o)$ in the task-relevant statistics. The mapping’s parameters $\omega$ are learned from data — a set of real observations and a set of simulated observations, not necessarily paired — and the policy is trained on the mapped distribution.

The method’s scope is the 3.4.1 §L.A.3’s three mechanisms’ third: the perception mismatch, which is present at $t = 1$ and is not action-proportional. The method’s cost is the real data’s size, which is the 3.4.1 §L.D.1’s second axis’s high end.

Self-explanation 1. Why is adaptation the method for the unmodelled axis rather than for the identified ones?

Answer. Because the identified axes have parameters that identification can estimate; the unmodelled axis has no parameter to estimate, and the mapping is the representation of the axis’s effect without modelling the axis itself.

§L.A.2 — Adaptation versus randomization

The two methods are adjacent on the 3.4.1 §L.D.1 spectrum, and they are often confused. The distinction is precise.

Randomization samples the perception’s appearance from a distribution and trains the policy to be robust across the samples. The policy sees a distribution that the designer believes contains the real cell’s appearance, and its robustness is the ability to perform across the distribution’s support. The method’s cost is the 3.4.2 §L.D.1’s curve’s third regime: the conservatism.

Adaptation fits a mapping from the simulated appearance to the real one and trains the policy on the mapped observations. The policy sees the mapping’s output, which is intended to be a sample from the real cell’s appearance’s distribution, and its performance is the ability to perform on that sample. The method’s cost is the mapping’s error and the real data’s size.

The two methods’ costs are different kinds. The randomization’s conservatism is a performance cost that is paid on every action; the adaptation’s mapping’s error is an alignment cost that is paid once but is a new source of divergence. The 3.4.2 §L.E.2’s placeholder was the randomization as the interim measure, and the adaptation is the replacement; the two are the spectrum’s two ends of the perception axis’s method, and the choice is by the real data’s availability.

The 3.4.2 §L.D.1’s curve’s third regime applies to the adaptation as well: an adaptation whose mapping is over-fit to its real data — a mapping trained on 100 images that memorizes them — generalizes poorly to the real cell’s new images, and its policy’s performance is the over-fit’s. The 3.4.3 §L.D.3’s held-out validation is the adaptation’s as well.

§L.A.3 — The mapping’s two directions

The mapping can be fitted in either direction, and the direction is a design choice with consequences.

Sim-to-real. $g: \mathcal{O}_{sim} \to \mathcal{O}_{real}$ maps the simulated observation to the real one. The policy is trained on $g(o_{sim})$, so its inputs are intended to be real-like. The advantage is that the policy’s test-time inputs are the real images directly, with no mapping at inference; the disadvantage is that the policy’s training’s inputs are the mapping’s outputs, whose error is what the policy sees.

Real-to-sim. $h: \mathcal{O}_{real} \to \mathcal{O}_{sim}$ maps the real observation to the simulated one. The policy is trained on the simulator’s observations as usual, and at deployment the real image is mapped by $h$ before the policy sees it. The advantage is that the policy is trained on the simulator’s distribution, which is what the simulator produces naturally; the disadvantage is that the mapping is in the inference’s path, so its error is on the critical path and its latency is the deployment’s.

The running case uses the sim-to-real direction: the 3.1.4 policy is fine-tuned on the mapped observations, and the deployment’s inputs are the real camera’s images without a mapping. The choice is by the latency: the sim-to-real direction puts the mapping in the training loop, where the latency is affordable, rather than in the inference loop, where the 3.2.3 planner’s 20 ms budget is the constraint.

Concept check. A team uses the real-to-sim direction and the mapping’s latency is 15 ms per frame. What does this do to the deployment’s budget, and what is the alternative?

Answer. It consumes 15 of the 100 ms control period’s budget, leaving 85 ms for the policy’s inference and the planner; the alternative is the sim-to-real direction, which moves the mapping to the training loop and leaves the deployment’s inputs unmapped. [displaces M3]

Worked-example ladder.

WE-L.A.a (fully worked). A mapping’s alignment reduces the perception axis’s contribution from 45% to 15% of the tail, and the tail’s other contributions are unchanged at 35% (dynamics), 15% (actuation), and 5% (timing). Compute the tail’s total reduction if the baseline tail was 4.5 cm.

Solution. The baseline tail is the sum of the contributions times the baseline; the perception’s reduction is $0.45 \times 4.5 = 2.03$ cm to $0.15 \times 4.5 = 0.68$ cm, a reduction of 1.35 cm. The new tail is $4.5 - 1.35 = 3.15$ cm, which is the 3.4.4 §L.E.2’s expected value.

WE-L.A.b (completion). The mapping’s residual adds 0.5 cm to the tail because the policy trusts the mapping’s error. Recompute the tail and state whether it is within the 3 cm tolerance.

Completion. The tail is $3.15 + 0.5 = 3.65$ cm, which is above the 3 cm tolerance. The completion shows the mapping’s error’s cost: the adaptation’s benefit is the alignment’s 1.35 cm, and the mapping’s residual’s cost is 0.5 cm, and the net is above the tolerance if the dynamics is not also narrowed.

WE-L.A.c (faded). Derive the sim-to-real direction’s training objective for a policy trained on the mapped observations, and state the mapping’s role. Faded step. The policy’s loss is $\mathbb{E}_{o_{sim} \sim P_{sim}}[L(\pi(g(o_{sim})), a_{sim})]$, where $a_{sim}$ is the simulator’s action and $g$ is the mapping. Completion. The mapping is a fixed preprocessing during the policy’s training, and the policy’s gradient flows through it; the mapping’s error is therefore in the policy’s gradient’s path, which is why the mapping’s accuracy matters.

WE-L.A.d (contrastive near-miss). A team’s mapping is trained to produce beautiful, realistic images, and the policy trained on them fails on the real cell. Why it fails: the mapping’s objective was the perceptual quality, not the task-relevant statistics’ alignment; the cube’s edge’s precise position is a low-contrast feature that the perceptual loss does not weight. The fix: the alignment’s objective is the task-relevant statistics, measured by the 2.1.2 pose estimator’s consistency, not the perceptual quality.

Comparative table: adaptation versus randomization.

Design principle P49.1. Adaptation is for the axes that cannot be modelled; it fits a mapping where identification would fit a parameter, and the mapping’s error is the new source of divergence the deployment must control.

Consolidation checkpoint 1

(a) Three capabilities. You can state why adaptation is the unmodelled axis’s method. You can distinguish adaptation from randomization by their costs. You can state the two directions and choose between them by latency.

(b) Transfer probe. A cell’s perception contribution is 20% and the dynamics is 60%. Should the adaptation be the first method? Answer: no; the dynamics is the dominant axis and identification is the cheaper method, so the sequence starts with identification and the adaptation follows if the perception’s residual is the binding one.

(c) Remediation rule. If you cannot state why the sim-to-real direction moves the mapping off the critical path, re-read §L.A.3 before Part B.

Part B — The four adaptation families

§L.B.1 — Pixel-level translation

The pixel-level translation family fits a mapping between the two image spaces. The canonical methods are the generative adversarial ones.

CycleGAN (Zhu et al. 2017, tier A, S) learns two mappings $g: \mathcal{O}_{sim} \to \mathcal{O}_{real}$ and $h: \mathcal{O}_{real} \to \mathcal{O}_{sim}$ with the cycle-consistency objective:

plus two adversarial losses that make each mapping’s output indistinguishable from its target domain. The cycle-consistency is what makes the mapping invertible without paired data; the adversarial losses are what make the outputs realistic.

RetinaGAN (Bousmalis et al. 2018, tier A, S) is the robotics-specific variant. It adds a retina loss that preserves the object detection’s outputs through the mapping, so the mapping does not change what the detector sees. The addition is the recognition that a cycle-consistent mapping can change the task-relevant content — the cube’s pose — while preserving the appearance, and the retina loss is the constraint that the content is preserved.

The family’s strength is that it needs no paired data: a set of simulated images and a set of real images, unpaired, suffice. Its weakness is the adversarial training’s instability and the mode collapse, and its evaluation’s difficulty: a mapping that produces beautiful images can misalign the task-relevant features, and the 3.4.3 §L.D.3’s held-out validation is the only check.

§L.B.2 — Feature-level alignment

The feature-level alignment family fits the mapping in the policy’s feature space rather than in the pixel space. The canonical method is the domain-adversarial training (Ganin et al. 2016, tier A, S):

where $\theta_e$ is the feature encoder, $\theta_c$ the task head, and $\theta_d$ a domain discriminator trained to distinguish the simulated features from the real ones. The encoder’s minimization of the discriminator’s loss makes the features domain-invariant, so the task head — trained on the simulated features — applies to the real features.

The family’s strength is that it acts where the policy’s decisions are made: the features are the policy’s inputs, and their alignment is the alignment that matters. Its weakness is that it needs the policy’s architecture — the encoder and the task head — to be trainable together with the discriminator, which is the 3.1.4 fine-tuning’s structure, and its stability depends on the discriminator’s balance.

For the running case, the feature-level alignment is the second stage: the 3.1.4 VLA’s vision encoder is the $e$, the action head is the $c$, and the discriminator is trained on the encoder’s features from the simulated and the real images. The alignment is applied to the 3.4.1 §L.B.2’s perception axis’s residual after the pixel-level mapping’s first stage.

§L.B.3 — Residual correction

The residual correction family fits a correction to a simulator’s prediction rather than a mapping between the observations. The method’s form is

where $r_\psi$ is a small correction network trained to minimize the discrepancy between the corrected simulated observations and the real ones.

The family’s strength is that the correction’s magnitude is small, so the correction’s error is small, and the base simulator’s structure is preserved. Its weakness is that it requires paired data — the same scene rendered in sim and observed in real — which the running case can produce by placing the object at a known pose and rendering it in the simulator at that pose. The pairing is the family’s cost, and the correction’s small magnitude is its benefit.

SimOpt (Chebotar et al. 2019, tier A, S) is the family’s robotics representative: a learned residual model that maps the simulated state and action to the real one, trained on the real system’s data, and the policy is trained on the corrected simulation. The residual’s form is the sim-to-real’s action-space analogue, and the correction’s small magnitude is the 3.4.1 §L.B.4’s argument for why a correction is safer than a mapping: a small error’s effect is small, while a mapping’s error is the policy’s input’s.

§L.B.4 — Differentiable rendering

The differentiable rendering family makes the simulator itself differentiable with respect to the scene’s parameters — the textures, the lighting, the camera’s pose — and fits those parameters by gradient descent on the image-space discrepancy (Kato et al. 2018, tier A, S; Loper & Black 2014, tier A, S for the earlier differentiable renderer).

The family’s strength is that it fits the physical parameters rather than a black-box mapping, so the fitted values are interpretable and the extrapolation is principled. Its weakness is the rendering’s cost — a differentiable renderer is slower than a rasterizer by orders of magnitude — and the local minima in the image-space loss.

For the running case, the differentiable rendering is the third stage, applied to the textures and lighting that the pixel-level mapping’s first stage could not represent. Its cost is the reason it is the third stage: the first two stages’ cheaper methods narrow the gap first, and the differentable rendering’s parameter fit is the residual’s.

Self-explanation 2. Why is the residual correction family’s error smaller than the pixel mapping family’s, given the same data?

Answer. Because the correction is constrained to be small — its form is a correction to a good simulator — while the mapping is unconstrained in magnitude; the constraint is the error’s bound, and the bound is the family’s benefit.

Concept check. A cell’s policy is a VLA with a frozen vision encoder. Which family applies?

Answer. The pixel-level translation, because the feature-level alignment requires the encoder to be trainable; the frozen encoder is the constraint that selects the family. [displaces M2]

Worked-example ladder.

WE-L.B.a (fully worked). A mapping’s cycle-consistency loss is 0.02 in the simulated direction and 0.03 in the real direction. Compute the total and state the interpretation.

Solution. The total is 0.05; the two directions’ losses are similar, which indicates the mapping is close to invertible on both domains. A large asymmetry — one direction’s loss much larger — would indicate the mapping is collapsing one domain’s content, which is the mode collapse’s signature.

WE-L.B.b (completion). The real direction’s loss rises to 0.15 while the simulated’s stays at 0.02. State the diagnosis and the fix. Completion. The asymmetry indicates the mapping is losing the real domain’s content — the cube’s features are being replaced by the simulated appearance’s — which is the retina loss’s target. The fix is the retina loss or the feature-level alignment’s second stage.

WE-L.B.c (faded). Derive the domain-adversarial objective’s equilibrium and state the discriminator’s accuracy at the equilibrium. Faded step. At the equilibrium, the discriminator’s accuracy is 0.5 — the features are indistinguishable — and the encoder’s features are domain-invariant. Completion. The 0.5 accuracy is the alignment’s success criterion, and the 3.4.4 §L.D.1’s two-sample test is its measurement.

WE-L.B.d (contrastive near-miss). A learner uses the pixel-level translation with a cycle-consistency loss alone, no retina loss, and finds that the mapped images are beautiful and the cube’s pose is wrong. Why it fails: the cycle-consistency preserves the domain’s content only up to the mapping’s information loss; the cube’s precise pose is a low-contrast feature that the mapping can change without changing the cycle’s loss. The fix: the retina loss, which adds the detector’s outputs’ preservation as a constraint.

Comparative table: the four families.

Design principle P49.2. Choose the family by what the policy’s input is: the pixel’s mapping when the policy’s encoder is frozen, the feature’s alignment when it is trainable, the residual’s correction when paired data exists.

Consolidation checkpoint 2

(a) Three capabilities. You can write the cycle-consistency objective. You can state the domain-adversarial objective and its equilibrium. You can choose a family from the policy’s architecture.

(b) Transfer probe. The simulator’s renderer is differentiable and the textures are parameterized. Which family, and why not the pixel translation? Answer: the differentiable rendering, because the physical parameters are identifiable and the fitted values extrapolate; the pixel translation’s black-box mapping does not extrapolate and its adversarial training is unstable.

(c) Remediation rule. If you cannot state why the retina loss is necessary, re-read §L.B.1 before Part C.

Part C — Training the mapping and the loop

§L.C.1 — The data’s collection

The adaptation’s data has two sets: a simulated set and a real set. The sets are collected with three design choices.

Coverage. The real set must cover the deployment’s distribution — the objects, the poses, the lighting, and the camera’s poses — or the mapping’s alignment is on the training set’s distribution, not the deployment’s. The 3.1.3 §L.D’s diversity rules apply.

Size. The mapping’s training needs enough real data for the mapping’s capacity. A small mapping — the residual correction’s — needs hundreds of pairs; a large mapping — the pixel translation’s — needs thousands of unpaired images. The 3.4.2 §L.B.4’s sample-complexity argument is the bound.

Pairing. The residual correction and the differentiable rendering need paired data; the pixel translation and the feature alignment do not. The pairing is produced by the known-pose procedure: the object is placed at a measured pose, the real image is taken, and the simulator renders at the same pose.

The running case’s real set is the 3.1.3 §L.E.1’s 4,000 episodes’ images, which cover the cell’s objects and poses. The simulated set is the renderer’s output on the same poses, which the simulator produces as part of the policy’s training’s rollouts.

§L.C.2 — The training’s stability

The adversarial families’ training is unstable, and the stability’s mechanisms are the design’s.

The discriminator’s balance. A discriminator that is too strong gives the encoder no gradient; one that is too weak gives a meaningless objective. The standard remedy is the gradient penalty (Gulrajani et al. 2017, tier A, S) or the spectral normalization (Miyato et al. 2018, tier A, S), both of which bound the discriminator’s Lipschitz constant and stabilize the equilibrium.

The cycle’s weight. The cycle-consistency’s weight relative to the adversarial’s sets the trade between the content’s preservation and the appearance’s realism. A large cycle’s weight preserves content at the cost of realism; a small one gives realistic but content-changed images. The running case’s weight is set so that the 2.1.2 pose estimator’s consistency is preserved, which is the retina loss’s operational definition.

The mapping’s capacity. A mapping with too much capacity memorizes the training set and fails on the held-out; one with too little cannot represent the appearance’s difference. The capacity is chosen by the 3.4.3 §L.D.3’s held-out validation.

§L.C.3 — The closed loop

The adaptation is not a one-time preprocessing; it is a closed loop, because the real cell drifts and the mapping’s residual grows.

The loop’s stages are the 3.1.4 §L.A.1’s seven phases applied to the mapping.

Stage 1: the baseline’s alignment. The mapping’s alignment is measured on a held-out real set, by the 3.4.4 §L.D.1’s two-sample test.

Stage 2: the policy’s training. The policy is fine-tuned on the mapped observations, with the 3.1.4 composite loss.

Stage 3: the twin-track’s re-validation. The 3.4.1 §L.C.2’s twin-track is re-run with the adapted policy.

Stage 4: the shadow’s deployment. The 3.1.4 §L.A.1’s shadow deployment runs the adapted policy beside the incumbent.

Stage 5: the monitor’s alignment. The alignment is re-measured on the deployment’s real images, and a significant degradation triggers the mapping’s re-fit.

The loop’s stage 5 is the drift’s detection, and its re-fit is the 3.4.4 §L.D.3’s continuation. The loop’s total cost is the mapping’s training plus the policy’s fine-tuning plus the shadow’s duration; the running case’s is about two days of cell time, which is the 3.1.3 collection’s budget’s fraction.

Self-explanation 3. Why must the loop re-fit the mapping rather than freeze it?

Answer. Because the real cell’s appearance drifts — the lighting changes, the table’s surface wears, the camera’s calibration drifts — and the frozen mapping’s residual grows with the drift; the 3.4.2 §L.D.3’s drift argument applies to the mapping’s coordinates.

Concept check. The mapping’s alignment test passes on the training set and fails on the deployment’s images. What is the cause, and what is the fix?

Answer. The real set’s coverage is the training set’s distribution and not the deployment’s; the fix is the real set’s expansion to the deployment’s distribution, per §L.C.1’s coverage. [displaces M4]

Worked-example ladder.

WE-L.C.a (fully worked). The mapping’s alignment’s two-sample statistic is 0.02 on the held-out real set and 0.15 on the deployment’s first week’s images. Compute the drift’s magnitude and state the decision.

Solution. The statistic’s ratio is $0.15/0.02 = 7.5$, which is a significant degradation; the alignment’s threshold is the held-out’s statistic’s confidence interval’s upper bound, and 0.15 is above it. The decision is the mapping’s re-fit on the deployment’s images, which is the loop’s stage 5.

WE-L.C.b (completion). The re-fit’s statistic is 0.04 on the deployment’s images and 0.03 on the held-out. State whether the loop’s iteration is successful. Completion. Both statistics are within the threshold, so the re-fit is successful; the held-out’s small rise (0.02 to 0.03) is the re-fit’s cost, which is the mapping’s capacity being spent on the deployment’s distribution.

WE-L.C.c (faded). Design the alignment’s measurement for the running case and state its sample size. Faded step. The measurement is the two-sample test on the 3.1.2’s feature space, with 500 real and 500 simulated images per test. Completion. The sample size is the test’s power’s requirement; the 3.4.4 §L.D.1’s statistic’s variance is $1/n$, so 500 gives a standard deviation of 0.045, which resolves a 0.1 difference.

WE-L.C.d (contrastive near-miss). A team trains the mapping once and deploys it for a month without re-fitting. The policy’s success falls from 88% to 72%. Why it fails: the mapping’s residual drifted, and the policy’s inputs are misaligned; the drift is the 3.4.2 §L.D.3’s mechanism in the mapping’s coordinates. The fix: the loop’s stage 5’s monitor and re-fit.

Comparative table: the loop’s stages.

Design principle P49.3. The adaptation is a loop, not a preprocessing; the real cell drifts and the mapping’s residual grows, and the loop’s monitor is the drift’s detector.

Consolidation checkpoint 3

(a) Three capabilities. You can design the real set’s coverage, size, and pairing. You can state the adversarial training’s stability mechanisms. You can lay out the closed loop’s five stages.

(b) Transfer probe. The mapping’s alignment passes on the held-out and the deployment’s images are unavailable. What is the risk, and what is the mitigation? Answer: the coverage gap; the mitigation is the shadow deployment’s images as a proxy for the deployment’s distribution, and the loop’s monitor as the early warning.

(c) Remediation rule. If you cannot state why the loop re-fits rather than freezes, re-read §L.C.3 before Part D.

Part D — Evaluation and pitfalls

§L.D.1 — The alignment’s measurement

The alignment is measured by a two-sample test in a feature space. The feature space is the policy’s encoder’s output — the 3.1.2’s vision features for the running case’s VLA — and the two samples are the mapped simulated features and the real features.

The statistic is the maximum mean discrepancy (MMD; Gretton et al. 2012, tier A, S):

where $k$ is a kernel and the expectations are over the two samples. The statistic is zero when the distributions are identical and grows with the divergence; its threshold is set from the held-out sample’s confidence interval.

The MMD’s advantage over the discriminator’s accuracy is that it is a statistic with a known sampling distribution, so its threshold is a hypothesis test’s, not a training’s heuristic. The test’s power depends on the sample size; the 3.4.4 §L.C.3’s 500-image sample gives a standard deviation of 0.045, which resolves a 0.1 difference at a reasonable confidence.

The MMD’s limit is the feature space’s choice. A feature space that is invariant to the task-relevant statistics — the 3.1.2’s encoder trained on the simulated data — is the space the alignment should be in, but its invariance is a property the alignment’s own training produced. The 3.3.4 §L.B.2’s semantic layer’s check is the complement: the alignment’s MMD is the perception’s check, and the affordance’s check is the task’s.

§L.D.2 — The composition’s evaluation

The adapted policy’s evaluation is the 3.4.1 §L.C.2’s twin-track, and its acceptance is the 3.4.1 §L.C.1’s tail against the tolerance. The composition has two error sources: the mapping’s residual and the policy’s capability.

The two sources are separable by a controlled comparison.

Mapping’s error alone. The policy is evaluated on the real cell with the mapping’s ground-truth correction — the real images directly, without the mapping — and the difference is the mapping’s error’s cost. The comparison’s baseline is the policy trained on the real images, which is the 3.1.3 collection’s data, if it exists.

Policy’s capability alone. The policy is evaluated on the mapped simulated images with the ideal mapping — the real images’ distribution’s features — and the difference is the policy’s capability’s cost. The comparison’s baseline is the policy trained on the simulator’s images without adaptation.

The two comparisons’ difference is the interaction: the mapping’s error’s effect on the policy’s behaviour. A policy that is brittle to the mapping’s error has a large interaction, and the fix is the policy’s training’s robustness — the 3.4.2’s randomization of the mapping’s noise — not the mapping’s improvement.

§L.D.3 — The five pitfalls

Mode collapse. The mapping produces a narrow set of outputs and fails on the held-out. Signature: the cycle-consistency’s real-direction loss rises and the mapped images look similar. Fix: the retina loss, the gradient penalty, or the feature-level alignment.

Over-fitting. The mapping memorizes the real set. Signature: the MMD is low on the held-out and high on the deployment’s. Fix: the real set’s expansion, per 3.4.3 §L.D.3’s held-out discipline.

The mapping’s drift. The real cell’s appearance changes. Signature: the monitor’s MMD rises over time. Fix: the loop’s re-fit, per §L.C.3.

The content’s change. The mapping aligns the appearance and changes the task-relevant content. Signature: the 2.1.2 pose estimator’s consistency falls, or the 3.3.3 affordance check fails after the mapping. Fix: the retina loss or the pose’s preservation’s auxiliary loss.

The interaction’s cost. The policy is brittle to the mapping’s error. Signature: the controlled comparison’s interaction term is large. Fix: the policy’s training’s randomization of the mapping’s noise.

The five pitfalls are distinguished by where the failure appears: the mode collapse in the mapping’s training, the over-fitting in the held-out-deployment gap, the drift in the monitor’s time series, the content’s change in the pose’s consistency, and the interaction in the controlled comparison.

Self-explanation 1. Why is the interaction term’s presence a policy problem rather than a mapping problem?

Answer. Because the interaction measures the policy’s sensitivity to the mapping’s error; a policy that is robust to the error’s distribution has a small interaction, and the fix is the policy’s robustness, not the mapping’s accuracy.

Concept check. The MMD is 0.02 on the held-out and 0.12 on the deployment’s; the pose’s consistency is 0.98. Which pitfall, and what is the fix?

Answer. The over-fitting or the drift; the pose’s consistency is preserved, so the content’s change is not the cause, and the deployment’s MMD’s rise is either the held-out’s coverage or the drift. The fix is the real set’s expansion and the loop’s re-fit. [displaces M4]

Worked-example ladder.

WE-L.D.a (fully worked). The adapted policy’s twin-track tail is 2.8 cm, within the 3 cm tolerance. The mapping’s MMD is 0.03 on the held-out and 0.05 on the deployment’s. State the acceptance and the register’s entry.

Solution. The tail is within tolerance and the MMD is within the threshold on both the held-out and the deployment’s; the acceptance passes. The register’s entry is the mapping’s drift: the deployment’s MMD’s 0.05 is a rise from the held-out’s 0.03, and the rise’s monitor’s threshold is the entry’s residual.

WE-L.D.b (completion). The deployment’s MMD rises to 0.15 after three weeks. Apply the loop’s stage 5. Completion. The rise is above the threshold, so the monitor triggers the re-fit; the re-fit’s cost is the mapping’s training plus the policy’s fine-tuning, and the re-validation is the twin-track’s re-run.

WE-L.D.c (faded). Write the controlled comparison’s interaction term’s definition and its interpretation. Faded step. The interaction is the difference between the adapted policy’s real performance and the sum of the ideal-mapping and the no-mapping’s performances. Completion. The term is the policy’s sensitivity to the mapping’s error, and its fix is the policy’s robustness.

WE-L.D.d (contrastive near-miss). A team’s mapping’s MMD is 0.01 on the held-out and the policy’s twin-track tail is 4.5 cm. The team concludes the mapping is excellent and the policy is the problem. Why it fails: the MMD measures the mapping’s alignment in the feature space, and the tail measures the composition; the mapping’s alignment can be perfect and the policy’s capability poor. The conclusion is half right: the mapping is aligned, and the policy’s capability is the next analysis’s target. The fix: the controlled comparison separates the two.

Comparative table: the pitfalls and their signatures.

Design principle P49.4. Measure the mapping’s alignment and the composition’s tail separately; a perfect mapping with a brittle policy fails, and an aligned mapping’s residual is not the policy’s capability.

Consolidation checkpoint 4

(a) Three capabilities. You can write the MMD’s definition and its threshold’s source. You can separate the mapping’s error from the policy’s capability. You can diagnose each pitfall from where it appears.

(b) Transfer probe. The MMD is 0.01 and the pose’s consistency is 0.80. Which pitfall? Answer: the content’s change; the mapping aligned the appearance and changed the task-relevant content, and the pose’s consistency’s fall is the signature.

(c) Remediation rule. If you cannot state why the interaction is a policy problem, re-read §L.D.2 before Part E.

Part E — The running case

§L.E.1 — The mapping’s design

The running case’s adaptation is the 3.4.1 §L.B.3’s 45% perception axis, and its design is a two-stage fit.

Stage 1: pixel-level translation. A RetinaGAN-style mapping from the renderer’s images to the camera’s images, trained on the 3.1.3 dataset’s 4,000 episodes’ images as the real set and the renderer’s output on the same poses as the simulated set. The retina loss preserves the 2.1.2 pose estimator’s outputs.

Stage 2: feature-level alignment. The 3.1.4 VLA’s vision encoder is fine-tuned with the domain-adversarial objective, with the discriminator on the encoder’s features. The stage’s purpose is the residual that the pixel-level mapping’s cycle could not align — the cube’s edge’s precise position and the depth’s noise’s statistics.

The stages’ order is by cost: the pixel-level’s adversarial training is the cheaper and the larger’s benefit, and the feature-level’s fine-tuning is the smaller and the residual’s. The running case does not use the residual correction (no paired data’s cost is justified) or the differentiable rendering (the renderer is not differentiable), and the two omitted families are the 3.4.1 §L.D.2’s sequence’s options if the two stages’ residual exceeds the tolerance.

§L.E.2 — The cycle and its gates

The closed loop’s five stages, with the running case’s gates.

Baseline alignment. The MMD on the held-out real set; the gate is below the 0.05 threshold.

Policy training. The 3.1.4 composite loss with the gradient-norm balance; the gate is the 3.1.4 §L.D.1’s offline lower bound on the three axes.

Twin-track. The 3.4.1 §L.C.2’s twin-track with the adapted policy; the gate is the tail within the 3 cm tolerance. The expected tail from 3.4.1 §L.E.2’s sequence and §L.A.1’s arithmetic is about 2.8 cm, with the dynamics and actuation’s identification from 3.4.3’s intervals.

Shadow. The 3.1.4 §L.A.1’s shadow with the intervention rate within the incumbent’s two points.

Monitor. The alignment’s MMD on the deployment’s images, with the threshold at the held-out’s upper confidence bound; a rise above the threshold triggers the re-fit.

The gates’ order is the 3.1.4 §L.D.3’s conjunction’s, and the adaptation’s entry is the baseline alignment’s gate, which is the loop’s unique addition.

§L.E.3 — The register and the hand-off

The register’s adaptation entry.

The register’s entries are the 3.3.4 §L.D.3’s five fields, and the residuals are the deployment’s acceptance.

The hand-off to Phase 4 is the adapted policy as the capstone’s perception’s baseline and the loop as the capstone’s iteration’s template. The 4.1.1 lesson’s capstone’s scoping uses the adapted policy’s twin-track’s tail as its starting point, and the 4.3 lesson’s iteration uses the loop’s five stages as its experiment’s design.

Self-explanation 2. Why is the interaction’s entry a residual rather than a mitigation?

Answer. Because the policy’s robustness to the mapping’s error is a property of the policy’s training, and the randomization’s coverage is the distribution it was trained on; the interaction’s residual is the error’s distribution outside that training’s coverage.

Worked-example ladder.

WE-L.E.a (fully worked). The adapted policy’s twin-track tail is 2.8 cm, the MMD is 0.04, and the shadow’s intervention rate is 4% versus the incumbent’s 3%. Compute the acceptance.

Solution. The tail is within 3 cm, the MMD is below 0.05, and the intervention’s difference is within the two-point tolerance; the acceptance’s conjunction passes on all three gates, and the loop’s stage 5 is the monitor’s ongoing check.

WE-L.E.b (completion). The monitor’s MMD rises to 0.12 after two weeks. Compute the decision and the re-fit’s scope. Completion. The rise is above the threshold, so the decision is the mapping’s re-fit on the deployment’s images; the re-fit’s scope is stage 1’s pixel-level mapping, and if the feature-level’s MMD’s residual is still high, stage 2’s re-fit follows.

WE-L.E.c (faded). Write the register’s entry for the interaction’s residual’s measurement. Faded step. The measurement is the controlled comparison’s interaction term on the held-out real set; the entry’s probability is the term’s magnitude’s distribution’s tail. Completion. The entry’s residual is the interaction’s distribution’s tail beyond the held-out’s coverage, and the mitigation is the policy’s randomization’s re-training.

WE-L.E.d (contrastive near-miss). A team deploys the adapted policy without the monitor’s MMD and finds the success falls from 88% to 70% over a month. Why it fails: the mapping’s drift was unmonitored, and the policy’s inputs were silently misaligned; the failure’s signature is the MMD’s rise, which no one measured. The fix: the monitor’s MMD and the loop’s re-fit, per §L.E.2.

Comparative table: the running case’s adaptation’s gates.

Design principle P49.5. Deploy with the loop’s monitor and the register’s drift entry; a mapping without a monitor is a silent failure’s instrument.

Consolidation checkpoint 5

(a) Three capabilities. You can state the two-stage design and its order. You can apply the loop’s five gates. You can write the register’s four entries.

(b) Transfer probe. The twin-track’s tail is 2.9 cm and the shadow’s intervention is 6% versus the incumbent’s 3%. Is the acceptance met? Answer: no; the intervention’s difference is three points, above the two-point tolerance, and the shadow’s gate fails even though the twin-track passes.

(c) Remediation rule. If you cannot state why the monitor’s MMD’s threshold is the held-out’s upper bound, re-read §L.E.2 before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: capacity, discriminator’s imbalance, and lighting’s change.

The lab

Task. Train a cycle-consistent mapping between a synthetic “simulated” domain and a “real” domain with a content-changing corruption, measure the MMD on the held-out, and demonstrate the retina loss’s role in preserving the content.

Conditions. Python 3.10+, PyTorch 2.x, CPU sufficient. The two domains are 32×32 images of a disc whose position encodes a “pose”; the “real” domain adds a texture and a position jitter. The mapping is a small convolutional network. Seed all RNGs.

Controls. (i) The cycle-consistency alone; (ii) the cycle-consistency plus the retina loss; (iii) the feature-level alignment; (iv) the MMD’s measurement on the held-out and the deployment’s sets.

Metrics.

Pre-registered hypotheses.

H1. The cycle-consistency alone has a low cycle’s loss and a high pose’s consistency’s error, because the mapping changes the position while preserving the cycle.

H2. The retina loss reduces the pose’s error to below 1 pixel at a small cost in the cycle’s loss.

H3. The feature-level alignment’s MMD is comparable to the pixel-level’s on the held-out, because both align the feature’s distribution.

H4. The MMD’s held-out-deployment gap is the over-fitting’s signature, and its magnitude is the register’s entry.

Break-it sequence.

1. Remove the retina loss silently. The pose’s error rises while the images look realistic. Silent: the cycle’s loss is low.
2. Train the mapping on too few real images silently. The held-out MMD is low and the deployment’s is high. Silent: the training’s metrics are good.
3. Freeze the mapping and shift the “real” domain silently. The MMD rises over the deployment’s images. Silent: the mapping’s training’s metrics are unchanged.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The pose’s consistency is measured through the mapping, not assumed. (2) The MMD is reported on two sets, so the gap is visible. (3) The retina loss’s cost is reported alongside its benefit. (4) The break-it results are a table. (5) The register’s entries have residuals.

A runnable core, seeded and annotated:

# lab_3_4_4.py — cycle-consistent mapping with retina loss
import numpy as np, torch, torch.nn as nn
torch.manual_seed(0); np.random.seed(0)

IMG, N = 32, 2000

def make_sim(n=N):                                    # sim: disc at random pos
    imgs, poses = [], []
    for _ in range(n):
        x, y = np.random.randint(8, 24, 2)
        img = np.zeros((1, IMG, IMG), dtype=np.float32)
        img[0, x-2:x+3, y-2:y+3] = 1.0
        imgs.append(img); poses.append((x, y))
    return torch.tensor(np.stack(imgs)), torch.tensor(poses, dtype=torch.float32)

def make_real(n=N):                                   # real: disc with texture
    imgs, poses = [], []
    for _ in range(n):
        x, y = np.random.randint(8, 24, 2)
        img = 0.1 * np.random.rand(1, IMG, IMG).astype(np.float32)
        img[0, x-2:x+3, y-2:y+3] = 1.0
        imgs.append(img); poses.append((x, y))
    return torch.tensor(np.stack(imgs)), torch.tensor(poses, dtype=torch.float32)

class Mapping(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(),
                                 nn.Conv2d(16, 16, 3, padding=1), nn.ReLU(),
                                 nn.Conv2d(16, 1, 3, padding=1), nn.Sigmoid())
    def forward(self, x): return self.net(x)

def pose_of(img):                                     # centroid as the "pose"
    g = img.detach().sum(dim=1)[:, 0]                # sum over channels
    ys = torch.arange(IMG).float()
    px = (g.sum(dim=1) * 0 + (g * ys).sum(dim=1) / g.sum(dim=1).clamp(min=1e-6))
    py = (g * ys).sum(dim=1) / g.sum(dim=1).clamp(min=1e-6)
    return torch.stack([px, py], dim=1)

sim_imgs, sim_poses = make_sim()
real_imgs, real_poses = make_real()
g, h = Mapping(), Mapping()
opt_g = torch.optim.Adam(g.parameters(), lr=1e-3)
opt_h = torch.optim.Adam(h.parameters(), lr=1e-3)

for step in range(2000):                              # training
    s = sim_imgs[np.random.randint(N, size=8)]
    r = real_imgs[np.random.randint(N, size=8)]
    opt_g.zero_grad(); opt_h.zero_grad()
    cyc_s = ((h(g(s)) - s) ** 2).mean()              # Eq. (B.1) sim direction
    cyc_r = ((g(h(r)) - r) ** 2).mean()              # Eq. (B.1) real direction
    pose_s = ((pose_of(g(s)) - pose_of(s)) ** 2).mean()  # retina loss
    loss = cyc_s + cyc_r + 0.5 * pose_s
    loss.backward(); opt_g.step(); opt_h.step()

with torch.no_grad():
    held = real_imgs[:200]
    mapped = g(held)
    print("held-out pose error:", ((pose_of(mapped) - pose_of(held)) ** 2).mean().item())

def mmd(x, y, sigma=1.0):                            # Eq. (D.1)
    def k(a, b):
        d = (a[:, None, :] - b[None, :, :]).pow(2).sum(-1)
        return torch.exp(-d / (2 * sigma ** 2))
    return k(x, x).mean() + k(y, y).mean() - 2 * k(x, y).mean()

with torch.no_grad():
    mapped = g(real_imgs[:300]).flatten(1)
    real = real_imgs[:300].flatten(1)
    print("MMD mapped-vs-real:", mmd(mapped, real).item())

Line 6–13 builds the “simulated” domain: a disc at a random position. Lines 15–24 builds the “real” domain: the same disc with a texture background. Lines 26–33 are the mapping, a small convolutional network. Lines 35–40 compute the pose as the disc’s centroid, which is the “retina” signal. Lines 46–60 train the two directions with the cycle-consistency loss and the retina loss. Lines 62–65 measure the held-out pose’s error. Lines 67–73 implement the MMD and measure the mapped-versus-real alignment.

Cheat sheet

Exercises with solutions

E 3.4.4.1. State why adaptation is the method for the unmodelled axis and why it is often the first method.

S 3.4.4.1. Because the unmodelled axis has no parameter to estimate, and the mapping is the axis’s effect’s representation; it is often the first method because the perception axis is the dominant contributor when the rendering’s realism is the largest uncertainty. The lesson: the method’s position is the axis’s, not the method’s sophistication.

E 3.4.4.2 (transfer). A cell’s policy has a frozen vision encoder and its perception’s contribution is 50%. Which family and why?

S 3.4.4.2. The pixel-level translation, because the feature-level alignment requires the encoder to be trainable; the frozen encoder selects the family. The lesson: the policy’s architecture is the family’s constraint.

E 3.4.4.3. Write the cycle-consistency objective and explain both directions’ roles.

S 3.4.4.3. $L = |h(g(s)) - s| + |g(h(r)) - r|$; the first direction preserves the simulated content through the round trip, the second the real’s. The lesson: the cycle is the pairing-free alternative to the paired loss, and its cost is that the content is preserved only up to the mapping’s information.

E 3.4.4.4 (diagnostic). A mapping’s images look realistic and the cube’s pose is wrong. Diagnose and propose two fixes.

S 3.4.4.4. The content’s change, because the cycle’s loss preserves the content only up to the mapping’s information; the fixes are the retina loss (the detector’s outputs’ preservation) and the pose’s auxiliary loss. The lesson: realism is not content; the two are different objectives.

E 3.4.4.5. Compute the MMD’s threshold from a held-out sample’s statistic of 0.03 and a standard deviation of 0.01, at a 95% confidence.

S 3.4.4.5. The threshold is $0.03 + 1.645 \times 0.01 = 0.046$. The lesson: the threshold is a statistical test’s, not a heuristic’s.

E 3.4.4.6 (transfer). A policy’s twin-track tail is within tolerance and the deployment’s MMD is 0.15. What is the risk, and what is the fix?

S 3.4.4.6. The over-fitting or the drift: the mapping’s alignment on the held-out is not the deployment’s, and the policy’s inputs are misaligned. The fix is the real set’s expansion and the loop’s re-fit. The lesson: the held-out’s MMD is not the deployment’s, and the monitor is the gap’s detector.

E 3.4.4.7 (diagnostic). The controlled comparison’s interaction is 1.5 cm and the mapping’s error’s cost is 0.5 cm. Diagnose and propose the fix.

S 3.4.4.7. The policy is brittle to the mapping’s error; the interaction is larger than the error’s direct cost, which is the signature. The fix is the policy’s training’s randomization of the mapping’s noise. The lesson: the interaction is a policy problem, not a mapping problem.

E 3.4.4.8. State why the closed loop re-fits rather than freezes the mapping.

S 3.4.4.8. Because the real cell’s appearance drifts and the mapping’s residual grows; the frozen mapping’s misalignment is the silent failure’s instrument. The lesson: the loop is the drift’s detector, and the monitor is its mechanism.

E 3.4.4.9 (transfer). Write the register’s four adaptation entries with their residuals.

S 3.4.4.9. Drift (residual: the unmonitored states); over-fitting (residual: the deployment’s coverage); content’s change (residual: the unmonitored features); interaction (residual: the error’s unrepresented distribution). The lesson: the register’s residuals are what the deployment’s acceptance is of.

Exit test

You are ready for Module 4.1 when you can, without notes:

1. State why adaptation is the unmodelled axis’s method.
2. Distinguish adaptation from randomization by their costs.
3. State the two directions and choose by latency.
4. Write the cycle-consistency objective and explain its directions.
5. State the retina loss’s role and its failure if absent.
6. Write the domain-adversarial objective and its equilibrium.
7. Choose the family from the policy’s architecture.
8. Design the real set’s coverage, size, and pairing.
9. State the adversarial training’s stability mechanisms.
10. Lay out the closed loop’s five stages and their gates.
11. Write the MMD’s definition and its threshold’s source.
12. Separate the mapping’s error from the policy’s capability.
13. Diagnose the five pitfalls from where they appear.
14. State the running case’s two-stage design and its order.
15. Write the register’s four adaptation entries with residuals.

Cross-references. 3.4.1 supplies the four axes, the tail metric, and the spectrum restated in §L.A.1 and §L.E.2. 3.4.2 supplies the randomization families and the curve’s third regime restated in §L.A.2. 3.4.3 supplies the identified parameters and the structure-error allowance restated in §L.E.2. 3.2.1 supplies the reconstruction term restated in §L.C.2. 3.1.4 supplies the shadow deployment and the conjunction restated in §L.C.3 and §L.E.2. 3.3.4 supplies the register restated in §L.E.3. Module 4.1 uses the adapted policy as the capstone’s perception’s baseline. Module 4.3 uses the loop’s five stages as its experiment’s design.

Ready. The next lesson to write is 4.1.1 — Defining the Capstone. Phase 4 is the synthesis and mastery phase, and its lessons are marked proposed in the curriculum index. Say “next” to begin Phase 4, or say “skip to X” to revisit an earlier lesson.