Lesson 3.4.1 — The Reality Gap as a Spectrum

Week 27

Prerequisite lessons: 3.3.4 (safety and constraints on LLM outputs), 3.2.1 (latent world models), 2.2.2 (model-based RL and MBPO), 2.3.3 (contact-rich manipulation), 2.3.4 (in-hand and dexterous manipulation), 3.1.4 (VLA fine-tuning).

Prior results used in this lesson. From 3.2.1: the coverage-gap ratio, which is the same measurement this lesson applies to the sim-to-real divergence. From 2.2.2: the short-rollout result, which is the same compounding-error argument that makes long simulated rollouts diverge. From 2.3.3: the contact parameters — friction, stiffness, delay — that are the gap’s dominant axes. From 2.3.4: the domain-randomization ranges set from identification uncertainty. From 3.1.4: the shadow-deployment protocol, which is the measurement instrument the gap analysis uses. Each is restated at the point of use.

What this lesson hands forward. The four-axis decomposition of the gap, the divergence metrics, and the spectrum of transfer methods from system identification through domain randomization to real-to-sim adaptation. Lesson 3.4.2 (domain randomization) uses the four axes as its randomization dimensions. Lesson 3.4.3 (system identification) uses the axes as its identification targets. Lesson 3.4.4 (real-to-sim adaptation) uses the spectrum’s far end as its starting point. Module 4.3 uses the divergence metrics as its experiment templates.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, and divider are unchanged; this lesson asks where the simulated running case diverges from the real one, how much each divergence costs, and how to choose among the transfer methods that the next three lessons will detail.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) state why the reality gap is a distribution shift and what follows for evaluation; (2) name the four divergence axes and one measurement per axis; (3) write the divergence metrics — mean, tail, and coverage ratio — and compute each from twin-track data; (4) state the sim-to-real divergence’s three cost mechanisms (dynamics mismatch, perception mismatch, and action-scale mismatch); (5) place system identification, domain randomization, and real-to-sim adaptation on the spectrum with their data costs and guarantees; (6) state the spectrum’s two axes — the degree of physical modelling and the degree of real data — and why they are not the same axis; (7) argue from a measurement which method is appropriate for the running case; (8) state the three preconditions for a transfer method to be evaluable; (9) design a twin-track experiment with per-axis divergence reporting; (10) compute the coverage ratio’s threshold from the deployment’s tolerance.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 3.2.1, the coverage-gap ratio was the held-out error on one task divided by the held-out error on the training task, with a threshold of 2–3. What does the ratio measure, and what is its limitation?

Answer. It measures how much worse the model is on a distribution it did not see, relative to one it did. Its limitation is that it is a ratio, not an absolute; a model can have a low ratio and an unacceptable absolute error. Used in §L.C.1 to define the sim-to-real divergence metric, which has the same structure and the same limitation.

Q2. In 2.3.3, the contact parameters were friction, stiffness, and delay. Which of the three is hardest to identify from data, and why?

Answer. The contact stiffness, because it is observable only during the brief impact and depends on the contact geometry, which changes as the object moves. Used in §L.B.2 to rank the axes by identification difficulty.

Q3. In 2.3.4, the domain-randomization ranges were set from the identification uncertainty, not from arbitrary numbers. What happens if the range is narrower than the uncertainty?

Answer. The policy is trained on a narrower distribution than the deployment’s, so it can be systematically wrong on the deployment; the robustness is a claim about the training range, not about the real robot. Used in §L.D.2 to state the relationship between identification and randomization.

Q4. In 3.1.4, the shadow deployment measured the candidate on the deployment distribution without risking the cell. What is the shadow’s analogue for a simulator?

Answer. The simulator’s shadow is the twin-track evaluation: the same policy run in sim and on the real cell, with the same states and the same instruction, and the divergence between the two measured. Used in §L.C.2.

Part A — The gap as a distribution shift

§L.A.1 — Why “gap” is a metaphor

The phrase “reality gap” suggests a single distance between two places. The reality is more specific, and the specificity matters because each axis has a different remedy.

A simulator is a distribution $P_{sim}(s'|s,a)$ over transitions. The real world is a distribution $P_{real}(s'|s,a)$. The gap is the divergence between them, and the divergence is not a scalar: it is a function of the state $s$ and the action $a$, and it has different magnitudes along different directions of the state-action space.

The first consequence is that the gap is not a defect to be eliminated. Even a perfect physics engine would have a gap, because the real cell’s friction, mass, and actuator delays are not the values in any configuration file. The gap is a distribution shift, and the simulator is one sample from the space of plausible simulators.

The second consequence is that the gap’s measurement is a distributional question. The right question is not “how big is the gap?” but “on which states and actions does the sim’s prediction differ from the real cell’s, and by how much relative to the deployment’s tolerance?” The 3.2.1 §L.D.3 coverage ratio is the same question, asked of a learned model instead of a physics engine.

The third consequence is that the gap’s remedy is not “more fidelity”. Fidelity on the modelled axes is necessary; fidelity on the unmodelled axes is impossible; and the transfer methods of Part D are the methods for making a policy robust to the parts of the gap that cannot be modelled.

Self-explanation 1. Why does a simulator’s gap persist even when the simulator’s physics is “correct”?

Answer. Because the physics is correct only up to the parameters, and the parameters — friction, mass, delay — are not the real cell’s values; correctness of the model’s form does not imply correctness of its parameters.

§L.A.2 — The twin-track evaluation

The gap is measured by the twin-track evaluation: the same policy, the same initial state, and the same instruction are run in the simulator and on the real cell, and the two trajectories are compared.

The comparison has three components.

State divergence. At each step $t$, the distance between the simulated state $s_t^{sim}$ and the real state $s_t^{real}$. The state is the observable state — the joint positions, the end-effector pose, and the object pose from the 2.1.2 estimator — not the simulator’s hidden state, which the real cell does not have.

Action divergence. The distance between the simulated action and the real action at the same nominal state. If the policy is deterministic, the actions should be identical at identical states; a difference indicates that the states have diverged, not that the policy differs.

Outcome divergence. The difference in the task’s outcome — success or failure, placement error, peak force — between the two tracks.

The twin-track’s design requires three preconditions, and each is a place where the evaluation can be invalid.

Same initial state. The real cell’s initial state is measured, not assumed, and the simulator is initialized to that state. A simulator initialized to its own nominal state measures a different trajectory.

Same instruction. The instruction is the same string, and the policy’s language path is the same. A 3.1.4 fine-tuned policy’s language path is part of the comparison.

Same control frequency. The action is interpreted at the same frequency, and the 3.1.4 §L.C.2 action-scale convention is asserted. A frequency mismatch produces a divergence that is not the gap’s.

The twin-track’s cost is the real cell’s time. A 100-episode twin-track is 100 real episodes, which is the 3.1.3 collection’s cost; the evaluation is expensive, and its design must be efficient.

§L.A.3 — The gap’s three costs

The divergence has three mechanisms, and each has a different signature and a different remedy.

Dynamics mismatch. The simulator’s transition differs from the real cell’s, and the policy’s plan diverges over time. The signature is a state divergence that grows with the trajectory’s length, with the growth rate set by the 2.2.2 compounding-error argument. The remedy is identification (the parameters’ true values) or randomization (robustness over their range).

Perception mismatch. The simulator’s images and depth differ from the real camera’s — rendering, noise, and lighting — and the policy’s observation encoder sees a different input. The signature is a state divergence that is not explained by the dynamics, because the policy’s actions are different at the same true state. The remedy is rendering fidelity, domain randomization of the appearance, or a real-to-sim adaptation of the observation model.

Action-scale mismatch. The simulator’s actuators respond differently from the real ones — the delay, the gear ratio, the saturation — and the same action produces a different motion. The signature is a divergence that is proportional to the action’s magnitude and that does not compound with the trajectory’s length. The remedy is actuator identification or a calibration of the action space.

The three mechanisms are separable in the twin-track’s data. A divergence that grows with $t$ is dynamics; one that is present at $t=1$ is perception or action-scale; one that is proportional to the action is action-scale. The 3.3.4 §L.B fault-classification logic applies: each mechanism has a signature, and the signature is what assigns it.

Concept check. The twin-track’s state divergence is 1 mm at $t=1$ and 5 cm at $t=20$, with the growth roughly linear. Which mechanism dominates?

Answer. Dynamics mismatch, because the divergence grows with the trajectory’s length; perception and action-scale would be present at $t=1$ and roughly constant. [displaces M1]

Worked-example ladder.

WE-L.A.a (fully worked). A twin-track has 50 episodes. The mean state divergence at $t=10$ is 2 cm in the real track and 1.8 cm in the sim track, and the outcome success rates are 0.90 and 0.94. Compute the divergences and state which is the larger.

Solution. The state divergence between the tracks at $t=10$ is $|2 - 1.8| = 0.2$ cm, which is small relative to the absolute errors; the outcome divergence is 0.04. The small state divergence and the larger outcome divergence indicate that the outcome is more sensitive than the state, which is common in contact tasks; the outcome is the deployment’s metric.

WE-L.A.b (completion). The state divergence at $t=1$ is 3 mm and at $t=20$ is 2 cm, with the growth super-linear. Classify the mechanism. Completion. The super-linear growth indicates dynamics with feedback, the 3.2.2 §L.D.3 exponent above 1; the mechanism is dynamics mismatch, and the remedy is identification or randomization.

WE-L.A.c (faded). Design the twin-track’s initial state measurement and state what the simulator must be told. Faded step. Measure the joint positions, the end-effector pose, and the cube’s pose from 2.1.2; set the simulator’s state to those values. Completion. The simulator’s hidden state — velocities and contact state — is not directly measurable, and its initialization is an assumption; the assumption is a source of divergence that the twin-track cannot separate from the gap.

WE-L.A.d (contrastive near-miss). A learner runs the twin-track with the simulator initialized to its nominal state and the real cell at a different pose. The divergence is 8 cm at $t=1$ and the learner concludes the dynamics is wrong. Why it fails: the initial states differ, and the divergence is the initial-state difference, not the dynamics. The fix: initialize the simulator to the measured state and re-run.

Comparative table: the three mechanisms.

Design principle P46.1. The gap is a distribution shift, not a defect; measure it on the axes and by the mechanisms, and let the measurement choose the remedy.

Consolidation checkpoint 1

(a) Three capabilities. You can state why the gap persists even with correct physics. You can design a twin-track evaluation with its three preconditions. You can classify a divergence by its mechanism from its signature.

(b) Transfer probe. The twin-track’s divergence is 5 mm at $t=1$ and 6 mm at $t=20$. Which mechanism? Answer: perception or action-scale, because the divergence is present early and does not grow; the discriminating test is whether it scales with the action’s magnitude.

(c) Remediation rule. If you cannot state why the initial-state initialization matters, redo WE-L.A.d before Part B.

Part B — The four divergence axes

§L.B.1 — The axes and their models

The divergence’s four axes are the sources that the sim-to-real literature has converged on, and each has a formal model that the identification of 3.4.3 and the randomization of 3.4.2 will use.

Dynamics. The rigid-body and contact dynamics: masses, inertias, friction coefficients, restitution, and the contact model’s stiffness and damping. The formal model is the 2.3.3 §L.B contact law $F_n = \max(0, k_c\phi + b_c\dot\phi)$ with the parameters $(m, \mu, k_c, b_c, e)$.

Perception. The camera’s rendering, noise, and calibration: the lighting, the textures, the sensor noise, and the intrinsics. The formal model is the 2.1.1 projection with the intrinsics $\mathbf{K}$ and the distortion coefficients, plus an additive noise model on the image.

Actuation. The motors’ response: the delay, the gear ratio, the saturation, and the back-EMF. The formal model is a first-order lag with a delay $\tau_d$, a saturation limit, and a gain.

Timing. The controller’s and the simulator’s clocks: the control period, the physics timestep, and the jitter. The formal model is a nominal period $T$ with a jitter distribution.

The four axes are not independent. A friction mismatch manifests as a dynamics divergence and a contact-stiffness divergence; a timing mismatch manifests as an actuation divergence and a contact divergence. The axes are a decomposition for measurement, not a claim of independence.

§L.B.2 — The running case’s axis values

The running case’s simulated values and their identification uncertainty, restated from 3.2.1 and 2.3.3 where the values were used.

Dynamics. The cube’s mass 0.1 kg with $\pm 20$ uncertainty; the friction coefficient 0.5 with $\pm 0.2$; the contact stiffness $10^5$ N/m with $\pm 0.5 \times 10^5$; the restitution 0.3 with $\pm 0.1$. The mass and friction are the two that dominate the grasp’s success.

Perception. The camera’s intrinsics are calibrated with the 2.1.1 procedure, and the residual is small; the rendering’s textures are synthetic and the real cell’s are not; the sensor noise is modelled as Gaussian with a variance that the real camera’s noise does not match. The perception axis’s uncertainty is the largest of the four, because the rendering’s realism is a qualitative property that the calibration does not capture.

Actuation. The arm’s motors have a 10 ms delay with $\pm 10$ ms uncertainty, a gear ratio that the simulator models nominally, and a saturation at 40 N that the 2.3.3 force limit is set below.

Timing. The simulator runs at 2 ms and the controller at 1 ms; the real controller’s jitter is a few hundred microseconds, and the simulator has none. The timing axis is the smallest, because the real controller’s jitter is small relative to the control period.

The ranking is the lesson’s operational output: the dynamics axis’s uncertainty is moderate and identifiable; the perception axis’s is large and only partly identifiable; the actuation and timing axes’ are small. The 3.4.2–3.4.4 lessons will use the ranking to choose their methods, and the running case’s ranking is what §L.E uses.

Self-explanation 2. Why is the perception axis’s uncertainty harder to identify than the dynamics axis’s?

Answer. Because the dynamics parameters are scalars with observable effects on the trajectories, while the perception’s realism is a high-dimensional property with no single observable; the calibration measures the intrinsics and not the rendering’s realism.

§L.B.3 — The axis-divergence decomposition

The twin-track’s divergence can be decomposed by axis with a controlled experiment per axis.

Dynamics. Hold the perception and the actuation fixed by using the real camera and the real actuator in the loop, and vary the simulated dynamics’ parameters. The divergence’s sensitivity to each parameter is the axis’s contribution.

Perception. Hold the dynamics and the actuation fixed by using the identified dynamics and the real actuator, and swap the simulated camera for the real one. The divergence’s change is the perception’s contribution.

Actuation. Hold the dynamics and the perception fixed, and swap the simulator’s actuator model for the identified one. The divergence’s change is the actuation’s contribution.

Timing. The timing’s contribution is measured by the divergence’s sensitivity to the control period; it is usually small and is measured last.

The decomposition’s cost is four twin-track runs, each with one axis varied and the others held. The runs are the 3.4.3 identification’s data, and their design is the same as the 2.2.2 MBPO’s model-error decomposition.

§L.B.4 — Fidelity’s double edge

The fourth misconception is that more fidelity always narrows the gap. It does not, and the mechanism is worth stating precisely.

A simulator that models an axis with a parameter that is identified narrows the gap on that axis. A simulator that models an axis with a parameter that is assumed widens the gap on that axis, because the assumption is a point estimate whose error is not represented. An unmodelled axis has an uncertainty that the policy can be made robust to; a modelled axis with a wrong parameter has an error that the policy trusts.

The running case’s contact stiffness is modelled at $10^5$ N/m, and the real cell’s is $5 \times 10^4$ to $1.5 \times 10^5$. The simulator’s assumption is the nominal value, and a policy trained on it can be brittle to the range. The 3.4.2 domain randomization represents the range; the 3.4.3 identification narrows it. Neither is “more fidelity”; both are better representation of the uncertainty.

Worked-example ladder.

WE-L.B.a (fully worked). The dynamics axis’s divergence is 2 cm at $t=20$ with the nominal parameters, and 1 cm when the mass is set to its true value. Compute the mass’s contribution and the remaining divergence.

Solution. The mass’s contribution is $2 - 1 = 1$ cm, half the total; the remaining 1 cm is the other parameters’ and the unmodelled axes’. The decomposition shows that the mass is the dominant single parameter, and the 3.4.3 identification should target it first.

WE-L.B.b (completion). The perception axis’s divergence is 3 cm at $t=20$ with the synthetic camera and 1.2 cm with the real camera. Compute the perception’s contribution and compare it to the dynamics’. Completion. The perception’s contribution is 1.8 cm, larger than the dynamics’ 1 cm after the mass correction; the perception axis dominates, and the 3.4.4 real-to-sim adaptation should target it.

WE-L.B.c (faded). Design the controlled experiment for the actuation axis and state the confound it must control. Faded step. Vary the actuator model with the identified dynamics and the real camera; the confound is the contact’s response to the actuator’s delay, which changes the dynamics’ effect. Completion. The confound is controlled by keeping the contact’s parameters fixed and varying only the delay.

WE-L.B.d (contrastive near-miss). A learner raises the simulator’s contact stiffness to the real cell’s measured value and finds the policy’s success falls. Why it fails: the measured value is a point estimate, and the real cell’s stiffness varies across contacts; the policy trained on the point estimate is brittle to the variation. The fix: randomize over the identified range, not the point estimate.

Comparative table: the running case’s axes.

Design principle P46.2. Fidelity on an identified axis narrows the gap; fidelity on an assumed axis widens it, because the assumption’s error is trusted rather than represented.

Consolidation checkpoint 2

(a) Three capabilities. You can name the four axes and their formal models. You can rank the running case’s axes by uncertainty. You can design the axis-decomposition experiment.

(b) Transfer probe. The dynamics axis’s divergence is 2 cm, the perception’s is 1.8 cm, and the actuation’s is 0.3 cm. Which axis’s method should run first? Answer: the dynamics, because it is the largest identifiable axis and identification is the cheapest method; the perception’s method is second and is more expensive.

(c) Remediation rule. If you cannot state why a modelled-but-wrong axis is worse than an unmodelled one, re-read §L.B.4 before Part C.

Part C — Measuring the gap

§L.C.1 — The divergence metrics

Three metrics summarize the twin-track’s data, and each answers a different question.

Mean divergence. The average state divergence across the episodes at a fixed horizon $t$,

The mean answers “how far off is the sim on average?” and is the least informative of the three, because it hides the tail.

Tail divergence. The 95th percentile of the state divergence at the same horizon. The tail answers “how far off is the sim in the worst cases?” and is the metric that the deployment’s tolerance is compared to, because the deployment fails in the tail, not at the mean.

Coverage ratio. The 3.2.1 §L.D.3 ratio restated: the task’s outcome divergence on the sim divided by the outcome divergence on the real cell, or equivalently the error ratio between the tracks. The ratio answers “how much does the sim overstate the policy’s performance?” and is the metric that the acceptance uses.

The three are computed on the same twin-track and reported together. A mean of 2 mm with a tail of 5 cm is a different system from a mean of 1 cm with a tail of 1.5 cm, and the mean alone does not distinguish them.

§L.C.2 — The twin-track protocol

The protocol’s steps are the 3.1.4 shadow deployment’s structure applied to the sim.

Step 1: policy freeze. The policy is the deployed checkpoint, not a training checkpoint; the sim’s evaluation is of the deployment’s candidate.

Step 2: initial state matching. Each episode’s initial state is measured on the real cell and set in the simulator, per §L.A.2.

Step 3: paired execution. The same instruction and the same initial state are run in both tracks; the real track’s actions are logged and the sim track’s actions are the policy’s outputs at the sim’s states.

Step 4: divergence computation. The per-step state divergence, the outcome divergence, and the coverage ratio are computed.

Step 5: axis attribution. The divergence is attributed to the axes by the §L.B.3 decomposition, using the four controlled runs.

Step 6: tolerance comparison. The tail divergence is compared to the deployment’s tolerance — for the running case, the 16-pixel 3.2.2 §L.E.1 tolerance, converted to metres by the projection.

The protocol’s output is a table with the axes’ contributions, the metrics, and the tolerance comparison. That table is the input to the transfer-method choice of §L.E.

§L.C.3 — What the metrics do not say

The metrics have three limits, and each is a place where the evaluation can be over-read.

The sim is one sample. The twin-track measures the divergence between this simulator and this cell. The divergence’s extrapolation to a different cell or a different simulator is an assumption.

The policy is one sample. The divergence is the policy-sim pair’s, not the sim’s. A different policy — more robust, more conservative — can have a smaller divergence from the same sim, because the policy’s actions avoid the states where the sim is wrong. The 3.4.2 randomization’s value is that it produces such a policy, and the twin-track after randomization measures a different pair.

The real cell is one sample. The real cell’s parameters vary with time and with the object; the twin-track’s measurement is at one time with one object. The 3.1.4 monitor’s drift detection is the mechanism that catches the temporal variation, and the twin-track’s repetition is the mechanism that catches the object variation.

The limits do not invalidate the metrics; they specify what the metrics are evidence for. The metrics are evidence about this policy on this sim and this cell, and the deployment’s generalization is an extrapolation that the certification’s 3.3.4 §L.D.3 register records.

Self-explanation 3. Why can a robust policy have a smaller twin-track divergence than a brittle one, from the same simulator?

Answer. Because the divergence is the policy’s trajectory’s, and a robust policy chooses actions that keep the trajectory near the states the sim models well; a brittle policy explores the states where the sim is wrong.

Concept check. The mean divergence is 2 mm and the tail is 8 cm. Is the sim adequate for a deployment with a 1 cm tolerance?

Answer. No, because the tail exceeds the tolerance; the mean is irrelevant to the deployment’s worst cases, and the tail is the metric the acceptance uses. [displaces M1]

Worked-example ladder.

WE-L.C.a (fully worked). A twin-track of 50 episodes gives state divergences at $t=15$ of mean 1.2 cm and 95th percentile 4.5 cm. The deployment’s tolerance is 3 cm. Compute the acceptance and state the failure.

Solution. The tail exceeds the tolerance by 1.5 cm, so the sim’s predictions at $t=15$ are outside the deployment’s tolerance in the worst 5% of cases. The acceptance fails, and the fix is the axis whose contribution dominates the tail.

WE-L.C.b (completion). The axis decomposition shows the perception axis contributes 60% of the tail. Which transfer method, and why? Completion. The perception axis’s uncertainty is large and only partly identifiable, so the 3.4.4 real-to-sim adaptation is the method; the 3.4.2 randomization of appearance is a complement, not a substitute.

WE-L.C.c (faded). Compute the coverage ratio from a sim success rate of 0.94 and a real success rate of 0.82. Faded step. The ratio is $0.94/0.82 = 1.15$, or the error ratio $(1-0.82)/(1-0.94) = 3.0$. Completion. The two ratios answer different questions: the success ratio says the sim is 15% optimistic, and the error ratio says the sim’s failures are a third of the real ones. The error ratio is the 3.2.1 coverage ratio’s form and the one the threshold uses.

WE-L.C.d (contrastive near-miss). A team reports the mean divergence only, and the deployment fails in the tail. Why it fails: the mean is dominated by the typical case, and the tail is where the deployment’s tolerance is exceeded. The fix: report the tail, and use it for the acceptance.

Comparative table: the divergence metrics.

Design principle P46.3. Accept on the tail, report the mean, and state the policy-sim pair the metrics describe; a mean without a tail is a metric that hides the failure.

Consolidation checkpoint 3

(a) Three capabilities. You can compute the mean, tail, and coverage ratio. You can run the twin-track protocol’s six steps. You can state the metrics’ three limits.

(b) Transfer probe. A policy has a mean divergence of 3 mm, a tail of 6 mm, and a coverage ratio of 1.1. Is it acceptable for a 1 cm tolerance? Answer: yes, the tail is within the tolerance and the coverage ratio is close to 1; the remaining risk is the metrics’ limits, not the numbers.

(c) Remediation rule. If you cannot state why the coverage ratio and the success ratio answer different questions, redo WE-L.C.c before Part D.

Part D — The transfer-method spectrum

§L.D.1 — The spectrum’s two axes

The transfer methods differ along two axes, and the axes are not the same.

The degree of physical modelling. How much of the gap the method tries to represent in the simulator’s physics. System identification raises it by fitting the parameters; domain randomization raises it by representing the parameters’ range; real-to-sim adaptation raises it by fitting the observation model.

The degree of real data. How much real data the method consumes. Identification consumes a small identification dataset; randomization consumes little or none; adaptation consumes a dataset large enough to fit the observation model, which is the largest.

The spectrum’s methods are placed by the pair, and the pair is the choice’s structure.

System identification (the 3.4.3 lesson) is high modelling and low real data: it fits the simulator’s parameters to a small real dataset and uses the fitted simulator unchanged. Its guarantee is that the gap on the identified axes is narrowed to the fit’s residual; its cost is the identification dataset’s size, which is small.

Domain randomization (the 3.4.2 lesson) is high modelling as a range and minimal real data: it samples the parameters from the identification’s uncertainty intervals and trains the policy to be robust across them. Its guarantee is robustness over the sampled range; its cost is the policy’s conservatism and the 3.4.2 lesson’s performance hit.

Real-to-sim adaptation (the 3.4.4 lesson) is high modelling of the observation and high real data: it fits a model that maps the simulated observation to the real one, or vice versa, and the policy is trained on the adapted observation. Its guarantee is that the perception axis is narrowed; its cost is the real dataset’s size.

The three methods are not exclusive. A complete pipeline identifies the dynamics, randomizes over the identified range, and adapts the perception — the three together, in that order, are the 3.4.2–3.4.4 lessons’ sequence.

§L.D.2 — Choosing by measurement

The choice is made from the §L.C metrics and the §L.B axis ranking, and the rule is a sequence of questions.

Is the dynamics axis the dominant contributor to the tail? If yes, run identification first; it is the cheapest method and it narrows the axis to the fit’s residual. If the residual is still above the tolerance, randomize over the identified range.

Is the perception axis the dominant contributor? If yes, the options are rendering fidelity (expensive, and the realism’s measurement is qualitative), appearance randomization (cheap, and it trades conservatism for robustness), or adaptation (expensive in data, and the only method that narrows the unmodelled axis). The choice is by the real data’s availability: with a small dataset, randomize; with a large one, adapt.

Is the actuation axis the dominant contributor? If yes, identify the delay and the gain; the axis is small and identifiable, and the identification is the cheapest method for it.

Is the timing axis the dominant contributor? If yes, the fix is the simulator’s timestep and the controller’s jitter model; the axis is small and the fix is a configuration change.

The rule’s sequence is by cost, and its output is the method or the method’s combination. The sequence is the running case’s §L.E’s structure.

Self-explanation 1. Why is identification the first method in the sequence?

Answer. Because it is the cheapest and it narrows the axis to its fit’s residual, which may be below the tolerance without the need for the more expensive methods.

Concept check. The dynamics axis’s residual after identification is 0.5 cm and the tail’s tolerance is 1 cm. Is randomization needed?

Answer. No for the dynamics axis, because the residual is within tolerance; the randomization would be a cost without a need. If the perception axis dominates the remaining tail, the randomization is for perception, not dynamics. [displaces M3]

§L.D.3 — The spectrum’s far end

Real-to-sim adaptation is the spectrum’s far end, and the fourth misconception names it as the last resort. The correction is that it is the first resort when the gap’s dominant axis is unmodelled, because it is the only method that narrows an unmodelled axis.

The running case’s perception axis is the example. The rendering’s realism is not a parameter that the identification fits; it is a property of the renderer’s textures, lighting, and noise model that the simulator’s configuration does not expose. The adaptation fits a mapping from the simulated observation to the real one — a learned image transformation, or a feature-level alignment — and the policy trained on the adapted observation sees the real distribution’s statistics.

The adaptation’s cost is the real data’s size. The mapping’s fit needs enough real images to cover the appearance’s variation, which is the 3.1.3 collection’s diversity. The cost is the reason the method is placed last in the sequence when the other methods suffice, and the reason it is first when they do not.

Worked-example ladder.

WE-L.D.a (fully worked). The tail’s contributions are dynamics 40%, perception 50%, actuation 10%. The tolerance is exceeded by 50%. Which methods, in what order?

Solution. The perception axis dominates, so adaptation is the first method for it; the dynamics is second, and identification is cheap and reduces the 40%; the actuation’s 10% is identified last. The order is adaptation, identification, actuation identification — the perception’s method is first because its axis is the largest and it is the only method that narrows it.

WE-L.D.b (completion). The perception’s contribution falls to 20% after adaptation, and the dynamics is now 70%. Recompute the order. Completion. The dynamics is now dominant, and identification is the next method; the sequence re-orders as the measurements change, which is why the choice is iterative.

WE-L.D.c (faded). State the identification’s residual’s definition and its measurement. Faded step. The residual is the divergence after the fit minus the divergence with the true parameters, measured on a held-out identification dataset. Completion. The residual is the axis’s lower bound for the identification method, and it is what the randomization’s range is set from.

WE-L.D.d (contrastive near-miss). A team’s dynamics axis is dominant and they start with real-to-sim adaptation because it is the “most advanced” method. The adaptation narrows the perception axis, which is not dominant, and the tail remains above tolerance. Why it fails: the method was chosen by its sophistication, not by the measurement; the dynamics axis was untouched. The fix: the choice is by the axis ranking, not by the method’s reputation.

Comparative table: the transfer methods.

Design principle P46.4. Choose the transfer method by the axis that dominates the tail, not by the method’s sophistication; the cheapest method that meets the tolerance is the right one.

Consolidation checkpoint 4

(a) Three capabilities. You can place each method on the two-axis spectrum. You can apply the choice’s sequence by measurement. You can state why adaptation is first when perception dominates.

(b) Transfer probe. The actuation axis dominates the tail. Which method, and why not randomization? Answer: identification, because the axis is small and identifiable and the identification is the cheapest; the randomization’s robustness is unnecessary when the parameter can be measured.

(c) Remediation rule. If you cannot state why the choice is by the axis ranking, re-read §L.D.2 before Part E.

Part E — The running case

§L.E.1 — The measurement

The running case’s twin-track has 50 paired episodes, the initial states matched per §L.A.2, and the metrics of §L.C.1 computed at the grasp’s horizon $t=15$.

The tail’s contributions, from the four controlled runs of §L.B.3.

Dynamics contributes 35% of the tail, with the mass and friction the dominant parameters.

Perception contributes 45%, with the rendering’s textures and the depth noise the dominant sources.

Actuation contributes 15%, with the delay the dominant parameter.

Timing contributes 5%, with the jitter negligible.

The coverage ratio is 1.2 by the success ratio and 3.0 by the error ratio; the tail is 4.5 cm against a 3 cm tolerance, so the acceptance fails and the methods are needed.

§L.E.2 — The choice

The sequence of §L.D.2 gives the running case’s plan.

Step 1: adaptation for perception. The perception axis dominates, and the rendering’s realism is unmodelled. The 3.4.4 real-to-sim adaptation fits an image-level mapping on the 3.1.3 dataset’s images, and the policy is retrained on the adapted images. The expected contribution’s reduction is 45% to 15%, which brings the tail to about 2.6 cm, within the 3 cm tolerance.

Step 2: identification for dynamics. The dynamics axis is the next largest, and the identification of the 3.4.3 lesson fits the mass and friction from the twin-track’s trajectories. The expected reduction is 35% to 10%, and the tail falls to about 2.0 cm, with margin.

Step 3: identification for actuation. The actuation axis’s delay is identified and the simulator’s actuator model is updated. The reduction is 15% to 5%, and the tail falls to about 1.7 cm.

Step 4: no randomization. The dynamics’ residual is within tolerance after the identification, so the 3.4.2 randomization is not needed for the dynamics. It may be needed for the perception’s residual appearance variation, and that is the 3.4.4 lesson’s decision.

The sequence’s cost is the adaptation’s real dataset, which the 3.1.3 collection already has, and the identification’s small dataset, which the twin-track supplies. The total is the 3.4.2–3.4.4 lessons’ sequence with the running case’s numbers.

§L.E.3 — The evaluation’s re-run and the acceptance

The methods are followed by a re-run of the twin-track, and the acceptance is the §L.C.1’s tail against the tolerance.

The re-run’s expected tail is 1.7 cm, within the 3 cm tolerance with a margin of 1.3 cm. The coverage ratio’s expected value is 1.05 by the success ratio and 1.5 by the error ratio; the error ratio’s residual is the adaptation’s fit’s and the identification’s residual’s, and it is the 3.3.4 §L.E.3 register’s entry.

The acceptance is the 3.3.4 conjunction with the sim-to-real entry: the twin-track’s tail is within tolerance, the coverage ratio is within the threshold, and the register lists the residuals. The 3.1.4 shadow deployment is the real cell’s confirmation, and the two evaluations together are the deployment’s evidence.

Self-explanation 2. Why is the twin-track’s re-run not a substitute for the shadow deployment?

Answer. Because the twin-track measures the sim’s divergence from the real cell at the twin-track’s states, and the deployment visits states the twin-track does not; the shadow deployment is the measurement at the deployment’s distribution.

Worked-example ladder.

WE-L.E.a (fully worked). After the adaptation and the identification, the tail is 2.0 cm and the tolerance is 3 cm. Compute the margin and state whether the acceptance passes.

Solution. The margin is 1 cm, or 33% of the tolerance; the acceptance passes on the tail. The coverage ratio’s error-ratio value is 1.5, within the threshold’s 3, so the acceptance passes on both metrics.

WE-L.E.b (completion). The adaptation’s expected reduction does not materialize; the tail falls only to 3.8 cm after step 1. Recompute the sequence. Completion. The perception axis’s residual is larger than expected, so the adaptation’s fit is insufficient; the next option is the appearance randomization of the perception axis, and the sequence re-orders as the measurements change.

WE-L.E.c (faded). Write the register’s entry for the sim-to-real residual after the full sequence. Faded step. Failure: the sim’s residual divergence exceeds the tolerance in the tail. Probability: from the twin-track’s tail. Consequence: the grasp fails in the deployment’s worst cases. Mitigation: the twin-track’s re-run and the shadow deployment. Residual: the unmodelled states. Completion. The entry’s residual is the states the twin-track did not visit, which is the deployment’s distribution’s coverage gap.

WE-L.E.d (contrastive near-miss). A team runs the sequence and the tail is within tolerance, and they skip the shadow deployment. The cell fails on the first day because the deployment’s objects differ from the twin-track’s. Why it fails: the twin-track’s states are the twin-track’s distribution, not the deployment’s; the shadow deployment is the measurement at the deployment’s distribution. The fix: the shadow deployment is the deployment’s evidence, and the twin-track is the sim’s evidence; both are needed.

Comparative table: the running case’s sequence.

Design principle P46.5. Measure the gap, choose by the dominant axis, re-measure, and confirm with the shadow deployment; the twin-track is the sim’s evidence and the shadow is the cell’s.

Consolidation checkpoint 5

(a) Three capabilities. You can state the running case’s axis contributions and tail. You can apply the choice’s sequence and its expected reductions. You can state why the twin-track and the shadow are both needed.

(b) Transfer probe. The identification’s residual is 1.5 cm and the tolerance is 3 cm, but the perception’s contribution is still 40% of the tail. Is the sequence complete? Answer: no; the identification narrowed the dynamics axis, and the perception’s contribution is still the largest, so the adaptation or the appearance randomization is the next method.

(c) Remediation rule. If you cannot state why the twin-track’s states are not the deployment’s, re-read WE-L.E.d before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: correlated parameters, rigid mapping, and shadow omission.

The lab

Task. Build a synthetic twin-track with four axes, inject a known parameter mismatch on each, and measure each axis’s contribution to the divergence metrics. Then apply the choice’s sequence and verify the tail’s reduction.

Conditions. Python 3.10+, NumPy. The synthetic “simulator” is a 2-D point-mass with a contact-like nonlinearity; the “real cell” is the same model with perturbed parameters. Seed all RNGs.

Controls. (i) The nominal twin-track; (ii) each axis perturbed alone; (iii) all axes perturbed; (iv) the sequence’s identification applied to the dynamics axis.

Metrics.

Pre-registered hypotheses.

H1. The all-axes perturbation has a tail at least three times the nominal, and the four single-axis perturbations’ tails sum to roughly the all-axes tail.

H2. The dynamics perturbation’s signature is a growth with $t$, and the perception perturbation’s is a constant offset.

H3. The identification reduces the dynamics axis’s contribution by at least 40%, and the tail falls correspondingly.

H4. The sequence’s re-run’s tail is within the tolerance that the nominal twin-track meets.

Break-it sequence.

1. Perturb the perception axis silently. The policy’s actions differ at the same true state, and the dynamics is unchanged. Silent: the sim’s physics is correct.
2. Perturb the actuation delay silently. The action-scale mismatch is proportional to the action’s magnitude. Silent: the delay looks like a nominal value.
3. Skip the initial-state matching silently. The divergence at $t=1$ is the initial-state difference. Silent: the policy is the same.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The four axes are perturbed one at a time, so each contribution is measured. (2) The signatures are reported per axis, not only the magnitudes. (3) The identification’s reduction is measured against the axis’s contribution. (4) The break-it results are a table. (5) The re-run’s tail is compared to the nominal twin-track’s, not to an absolute.

A runnable core, seeded and annotated:

# lab_3_4_1.py — synthetic twin-track with four axes
import numpy as np
np.random.seed(0)

T, N = 15, 50

# The "real cell": point mass with contact-like nonlinearity
def real_step(s, a, p):
    x, v = s
    contact = p["k"] * max(0, x - 0.5) ** 2          # contact spring
    $v_new$ = v + (a - p["b"] * v - contact) * 0.05
    return np.array([x + $v_new$ * 0.05, $v_new$])

# The "simulator": same form, perturbed parameters
def sim_step(s, a, p, perception_scale=1.0, delay_gain=1.0):
    x, v = s
    contact = p["k"] * max(0, x - 0.5) ** 2
    $v_new$ = v + (a * delay_gain - p["b"] * v - contact) * 0.05
    return np.array([(x + $v_new$ * 0.05) * perception_scale, $v_new$])

REAL_P = {"k": 8.0, "b": 0.3}
SIM_P  = {"k": 8.0, "b": 0.3}

def run_twin(real_p, sim_p, perception=1.0, delay=1.0):
    $s_r$ = np.array([0.0, 0.0]); $s_s$ = $s_r$.copy()
    ds = []
    for t in range(T):
        a = -0.5 * $s_s[0$] - 0.2 * $s_s[1$]            # simple policy
        $s_r$ = $real_step(s_r$, a, real_p)
        $s_s$ = $sim_step(s_s$, a, sim_p, perception, delay)
        ds.append(np.linalg.$norm(s_r$ - $s_s$))
    return np.array(ds)

def metrics(ds):
    return ds[-1].mean(), np.percentile(ds[-1], 95)

nominal = run_twin(REAL_P, SIM_P)
print("nominal mean/tail:", metrics(nominal))

for name, kw in [
    ("dynamics k", dict(sim_p={"k": 12.0, "b": 0.3})),
    ("dynamics b", dict(sim_p={"k": 8.0, "b": 0.5})),
    ("perception", dict(perception=1.05)),
    ("actuation",  dict(delay=1.2)),
]:
    args = dict(real_p=REAL_P, sim_p=SIM_P); args.update(kw)
    ds = run_twin(**args)
    m, t95 = metrics(ds)
    print(f"{name:12s} mean={m:.4f} tail={t95:.4f}")

# Axis decomposition: all axes together
all_ds = run_twin({"k": 8.0, "b": 0.3}, {"k": 12.0, "b": 0.5},
                  perception=1.05, delay=1.2)
print("all-axes mean/tail:", metrics(all_ds))

Line 5 defines the “real cell”: a point mass with a contact spring at $x = 0.5$. Lines 9–13 are the “simulator”: the same form with a perception scale and an actuator gain. Line 15 runs a pair of trajectories with the same policy and the same initial state, and line 16 records the state divergence at each step. Lines 22–24 compute the mean and the tail of the final divergence. Lines 26–34 perturb each axis alone and print its metrics. Lines 37–38 run the all-axes case, which is the twin-track’s input to the axis decomposition.

Cheat sheet

Exercises with solutions

E 3.4.1.1. State why the reality gap is a distribution shift and what follows for evaluation.

S 3.4.1.1. The simulator’s transition and the real cell’s are two distributions, and the gap is their divergence, which depends on the state and the action. It follows that the evaluation is a distributional question — on which states, by how much, relative to the tolerance — not a scalar. The lesson: the gap’s remedy is axis-specific, and a scalar summary hides the axis.

E 3.4.1.2 (transfer). A twin-track’s divergence is 2 mm at $t=1$ and 6 cm at $t=20$, growing super-linearly. Classify and state the remedy.

S 3.4.1.2. Dynamics mismatch with feedback, the 3.2.2 exponent above 1; the remedy is identification or randomization. The lesson: the growth’s shape is the signature, and the signature chooses the remedy.

E 3.4.1.3. Compute the mean, tail, and coverage ratio from a twin-track with divergences at $t=15$ of mean 1.4 cm and 95th percentile 4.2 cm, and sim/real success rates 0.94 and 0.82.

S 3.4.1.3. Mean 1.4 cm, tail 4.2 cm, error ratio $(1-0.82)/(1-0.94) = 3.0$. The tail exceeds a 3 cm tolerance, so the acceptance fails; the error ratio is at the threshold’s edge. The lesson: the tail and the ratio are both needed, and the tolerance decides.

E 3.4.1.4 (diagnostic). A team raises the simulator’s contact stiffness to the measured value and the policy’s success falls. Diagnose and state the fix.

S 3.4.1.4. The measured value is a point estimate, and the real stiffness varies; the policy trained on the estimate is brittle. The fix is to randomize over the identified range, not the point estimate. The lesson: a modelled axis with a wrong parameter is worse than an unmodelled one, because the error is trusted.

E 3.4.1.5. Place system identification, domain randomization, and real-to-sim adaptation on the two-axis spectrum.

S 3.4.1.5. Identification: high modelling, low real data. Randomization: high modelling as a range, minimal real data. Adaptation: high observation modelling, high real data. The lesson: the methods differ on both axes, and the choice is by the measurement, not by the method’s position.

E 3.4.1.6 (transfer). The dynamics axis dominates the tail and the perception’s is second. Which method first, and why?

S 3.4.1.6. Identification first, because the dynamics is identifiable and the method is the cheapest; if the residual is within tolerance, the perception’s method is not needed. The lesson: the sequence is by cost and by the axis ranking, and the cheapest method that meets the tolerance is the right one.

E 3.4.1.7 (diagnostic). A policy’s twin-track tail is within tolerance and the deployment fails on the first day. Name the cause and the fix.

S 3.4.1.7. The twin-track’s states are not the deployment’s distribution; the fix is the 3.1.4 shadow deployment, which measures at the deployment’s distribution. The lesson: the twin-track is the sim’s evidence and the shadow is the cell’s.

E 3.4.1.8. Design the axis-decomposition experiment for the perception axis and state its confound.

S 3.4.1.8. Hold the identified dynamics and the real actuator, and swap the simulated camera for the real one. The confound is the camera’s mounting and calibration, which must be held fixed. The lesson: the decomposition is a controlled experiment, and each axis’s confounds are its neighbours’.

E 3.4.1.9 (transfer). A team’s coverage ratio is 1.1 by the success ratio and 4.0 by the error ratio. Which is the acceptance’s metric, and why?

S 3.4.1.9. The error ratio, because it measures the sim’s overstatement of failures, which is what the 3.2.1 coverage ratio’s threshold uses; the success ratio is a reporting metric. The lesson: the two ratios answer different questions, and the acceptance uses the error’s.

Exit test

You are ready for Lesson 3.4.2 when you can, without notes:

1. State why the reality gap is a distribution shift.
2. State the twin-track’s three preconditions and why each matters.
3. Name the three divergence mechanisms and their signatures.
4. Name the four axes and their formal models.
5. Rank the running case’s axes by uncertainty and state why perception is hardest.
6. Design the axis-decomposition experiment.
7. State fidelity’s double edge and its mechanism.
8. Compute the mean, tail, and coverage ratio from twin-track data.
9. State the metrics’ three limits.
10. Place the three methods on the two-axis spectrum.
11. State the choice’s sequence and its cost ordering.
12. State why adaptation is first when perception dominates.
13. Argue the running case’s method sequence from the measurements.
14. State why the twin-track and the shadow are both needed.
15. Write the register’s sim-to-real entry with its residual.

Cross-references. 3.2.1 supplies the coverage-gap ratio restated in §L.C.1. 2.2.2 supplies the compounding-error argument restated in §L.A.3. 2.3.3 supplies the contact parameters restated in §L.B.2. 2.3.4 supplies the randomization-from-uncertainty rule restated in §L.D.2. 3.1.4 supplies the shadow deployment restated in §L.C.2 and §L.E.3. 3.3.4 supplies the register restated in §L.E.3. Lesson 3.4.2 uses the four axes as its randomization dimensions. Lesson 3.4.3 uses the axes as its identification targets. Lesson 3.4.4 uses the spectrum’s far end as its starting point.

Ready. The next lesson to write is 3.4.2 — Domain Randomization. Say “next” to generate it.