Lesson 3.4.2 — Domain Randomization

Week 28

Prerequisite lessons: 3.4.1 (the reality gap as a spectrum), 2.3.3 (contact-rich manipulation), 2.3.4 (in-hand and dexterous manipulation), 3.2.1 (latent world models), 3.1.4 (VLA fine-tuning), 2.2.4 (offline-to-online fine-tuning), 3.3.4 (safety and constraints on LLM outputs).

Prior results used in this lesson. From 3.4.1: the four divergence axes, the identification-uncertainty intervals, the performance-robustness trade named as the spectrum’s second axis, and the twin-track’s tail metric. From 2.3.3: the contact law’s parameters $m, \mu, k_c, b_c$ and the grasp force budget. From 2.3.4: the randomization ranges set from identification uncertainty and the teacher–student distillation pattern. From 3.2.1: the ELBO’s reconstruction and KL balance, the same training machinery used here. From 3.1.4: the shadow deployment and the monitor’s trough criteria. Each is restated at the point of use.

What this lesson hands forward. The randomization-family taxonomy, the range-from-uncertainty rule, the sample-complexity arithmetic, and the performance-robustness curve. Lesson 3.4.3 (system identification) supplies the ranges this lesson randomizes over. Lesson 3.4.4 (real-to-sim adaptation) uses the family taxonomy as its baseline. Module 4.3 uses the performance-robustness curve as its ablation template.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, and divider are unchanged; this lesson takes the 3.4.1 §L.B.2 identification intervals — cube mass $0.1 \pm 0.02$ kg, friction $\mu = 0.5 \pm 0.2$, contact stiffness $k_c = 10^5 \pm 0.5\times 10^5$ N/m, actuator delay $10 \pm 10$ ms, and the perception’s appearance uncertainty — and asks how to sample them during training so that the resulting policy transfers.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) state what randomization does to the training distribution and why it buys robustness; (2) name the three randomization families and one representative of each; (3) state the range-from-uncertainty rule and apply it to the running case’s five parameters; (4) choose a distribution family for each parameter from its support and its effect; (5) compute the sample-complexity exponent from the number of randomized dimensions; (6) write the automatic-domain-randomization curriculum and its expansion rule; (7) plot and interpret the performance-robustness curve; (8) diagnose the five pitfalls from their signatures; (9) state the correlated-parameter problem and two mitigations; (10) assemble the running case’s randomization schedule with a budget and a coverage argument.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 3.4.1 §L.B.2, the running case’s identification intervals were mass $0.1 \pm 0.02$ kg, friction $0.5 \pm 0.2$, contact stiffness $10^5 \pm 0.5\times10^5$ N/m, and delay $10 \pm 10$ ms. Which of the four has the largest relative uncertainty, and which has the largest absolute uncertainty?

Answer. The delay has the largest relative uncertainty (100%), and the contact stiffness has the largest absolute uncertainty ($0.5 \times 10^5$ N/m). Used in §L.B.1 to argue that the relative uncertainty sets the range’s shape.

Q2. In 3.4.1 §L.B.4, fidelity on an identified axis narrows the gap and fidelity on an assumed axis widens it. How does randomization respond to that distinction?

Answer. Randomization replaces the assumed point estimate with a distribution over the identified interval, so the simulator’s error on that axis is represented rather than trusted. Used in §L.A.2 to define randomization’s mechanism.

Q3. In 2.3.3, the grasp force budget at the $45°$ misalignment with a 5 N grip was 1.67 N required and a margin of 3.0. How does an uncertain friction coefficient change that budget?

Answer. The required force scales as $1/\mu$, so a friction drop from 0.5 to 0.3 raises the requirement from 1.39 N to 2.32 N; the 5 N grip’s margin falls from 3.6 to 2.2. Used in §L.E.1 to compute the running case’s grasp-force randomization.

Q4. In 2.3.4, the domain-randomization ranges for the dexterous hand were set to the identification intervals, not wider, and the lesson argued that widening beyond the intervals costs performance without buying robustness. What is the mechanism?

Answer. A wider range makes the policy’s behaviour robust against perturbations the real robot does not exhibit, so the policy is more conservative than the deployment requires, and the conservatism is a cost paid for robustness that is never exercised. Used in §L.D.1 to derive the performance-robustness curve.

Part A — What randomization does

§L.A.1 — The training-distribution change

Domain randomization (DR) replaces the simulator’s fixed parameters with a distribution and trains the policy on samples from that distribution. Formally, if the simulator’s parameters are $\phi \in \Phi$ and the nominal simulator is $P_{sim}(s'|s,a;\phi_0)$, the randomized simulator is

where $p(\phi)$ is the randomization distribution. The policy is trained on $P_{DR}$ instead of $P_{sim}(\cdot;\phi_0)$, so its optimization problem changes from “maximize return on the nominal simulator” to “maximize expected return over the parameter distribution”.

The change has a precise effect on the policy. A policy trained on the nominal simulator specializes to the parameter’s value and is brittle to deviations. A policy trained on $P_{DR}$ must perform well across the distribution’s support, and the only way to do that is to choose actions whose success is not conditional on a parameter’s exact value. That is what “robustness” means operationally: the policy’s action at a state is chosen to be good under a set of plausible dynamics, not under one.

The change also has a precise cost. The randomized problem is harder than the nominal one, because the policy must satisfy a set of conditions simultaneously rather than one; the optimum’s value is lower, and the learning needs more samples. The 3.4.1 §L.D.1 spectrum’s second axis — the degree of real data — is the resource that the cost is measured in, and the performance-robustness curve of Part D is the trade’s shape.

Self-explanation 1. Why does training on $P_{DR}$ produce robustness rather than merely producing a policy that is good on average?

Answer. Because the policy is a function of the observation, and the observation does not reveal the parameter; to be good across the distribution, the policy must choose actions that are good for every parameter value consistent with the observation, which is robustness.

§L.A.2 — Why it works, and when it does not

Randomization works because the real robot’s parameters are unknown and the identification’s interval is the best available knowledge. The training distribution’s support is the interval, and a policy that performs across the interval performs on the real robot if the real robot’s parameters lie inside.

The condition is the whole argument, and it is where the method fails. If the real parameter is outside the interval, the policy’s behaviour is undefined — it is out of the training distribution — and the robustness guarantee does not apply. The 3.4.1 §L.C.3 coverage ratio’s limit is the same statement: the metrics describe the training distribution’s support, and the deployment’s parameter is an extrapolation.

The method’s three failure conditions are therefore:

Range mis-specification. The identification’s interval is wrong — too narrow, and the real parameter is outside; too wide, and the policy is over-conservative. §L.B.1 treats this.

Irrelevant parameters. The randomized set includes parameters whose variation does not affect the task; the randomization’s cost is paid without a benefit. §L.C.3 treats this.

Distribution mis-specification. The randomization’s shape is wrong — a uniform distribution where the parameter is log-normal — and the samples concentrate where the real robot is unlikely to be. §L.B.2 treats this.

The fourth source of failure is not randomization’s but the deployment’s: the real robot’s parameters can drift after training (a worn gripper’s friction falls), and the drift is outside the training distribution even though the training’s range was correct at training time. The 3.1.4 monitor and the 3.3.4 register are the mechanisms that catch drift, and the randomization’s re-training is the response.

§L.A.3 — The three families

The randomization literature has three families, distinguished by which 3.4.1 axis they randomize.

Visual randomization (Tobin et al. 2017, tier A, S) randomizes the perception axis: textures, lighting, camera pose, and noise. The original result trained a policy on a distribution of rendered scenes with randomized textures and lighting and transferred it to a real cell without any real images. The family’s cost is the rendering’s compute; its limit is that a randomization of appearance does not randomize the geometry the task depends on.

Dynamics randomization (Peng et al. 2018, tier A, S; OpenAI et al. 2018, tier A, S) randomizes the dynamics axis: masses, inertias, friction, contact stiffness, and actuator gains. The Dactyl result is the canonical demonstration — a dexterous hand policy trained across a wide distribution of friction and mass transferred to a real hand and reoriented a cube. The family’s cost is the sample complexity of the multi-dimensional parameter space; its limit is that the correlated parameters’ joint distribution is not captured by independent sampling.

Temporal randomization (Chebotar et al. 2019, tier A, S) randomizes the timing and actuation axes: delays, control periods, and jitter. It is the smallest family and the most targeted: a policy that is robust to a delay range is robust to the real actuator’s delay, and the randomization is cheap because the delay is a one-dimensional parameter.

A fourth family, sometimes separated, is disturbance randomization: external forces and torques applied at random times, which is a dynamics randomization of the environment rather than the robot. The running case’s contact events are disturbances, and §L.E uses this family for the divider contact.

The families compose. The running case’s design randomizes all four, with the 3.4.1 axis ranking determining the ranges’ widths and the sample budget’s allocation.

Concept check. A team randomizes the texture’s colour and the cube’s friction. Which family is each, and which axis does each belong to?

Answer. The texture’s colour is visual randomization on the perception axis; the friction is dynamics randomization on the dynamics axis. The two families target different axes, and a policy robust to one is not automatically robust to the other. [displaces M1]

Worked-example ladder.

WE-L.A.a (fully worked). The nominal simulator has friction $\mu = 0.5$ and the identification’s interval is $[0.3, 0.7]$. Compute the training distribution’s effect on the required grip force at the $45°$ misalignment.

Solution. From 2.3.3 Eq. (B.5), the required force at friction $\mu$ and misalignment $45°$ is $m(g+a)/(2\mu\cos 45°)$. At $\mu = 0.7$: $0.1 \times 9.81/(2 \times 0.7 \times 0.7071) = 0.99$ N. At $\mu = 0.3$: $0.1 \times 9.81/(2 \times 0.3 \times 0.7071) = 2.31$ N. The policy must choose a grip force above 2.31 N to succeed across the interval, versus 1.39 N on the nominal $\mu = 0.5$.

WE-L.A.b (completion). The standard 5 N grip gives a margin of 3.6 at the nominal friction and 2.2 at the interval’s worst case. Is the margin sufficient, and what does the completion say about the randomization’s cost?

Completion. The 2.2 margin is above the 3.4.1 threshold’s 1.5, so the grip is sufficient. The cost is that the policy’s grip is set for the interval’s worst case rather than the nominal, so its peak force is higher than the nominal policy’s, and the 3.3.4 force limit’s headroom is reduced. The completion shows that randomization’s conservatism shows up as a force cost, not only as a success-rate cost.

WE-L.A.c (faded). Derive the visual randomization’s training-loss expression for a pixel-wise reconstruction objective, and state why the objective is the same as 3.2.1’s. Faded step. The loss is $\mathbb{E}_{\phi \sim p}[|o_{sim}(\phi) - \hat{o}|^2]$; the expectation over the appearance distribution is the ELBO’s reconstruction term with the appearance as a latent. Completion. The objective is the same as 3.2.1’s because the appearance is a nuisance variable, and marginalizing it is what the reconstruction term does.

WE-L.A.d (contrastive near-miss). A team randomizes the cube’s mass over $[0.01, 1.0]$ kg “to be safe”. The trained policy’s success rate on the real 0.1 kg cube is 60%, versus 90% for a policy trained over $[0.08, 0.12]$. Why it fails: the wider range’s samples are concentrated in the masses the real cube is unlikely to have, and the policy’s behaviour on the real mass is a corner of the training distribution where the samples are sparse. The fix: the range is the identification’s interval, not a safety margin; a wider range is not “safer”.

Comparative table: the three families.

Design principle P47.1. Randomization trades peak performance for robustness over a range; the range is the identification’s interval, and a range wider than that is a cost, not a safety margin.

Consolidation checkpoint 1

(a) Three capabilities. You can state the training-distribution change and its cost. You can name the three families and their axes. You can compute the grip force’s range from the friction’s interval.

(b) Transfer probe. A team randomizes the lighting and the mass. Which family is each, and which axis does each belong to? Answer: the lighting is visual on the perception axis; the mass is dynamics on the dynamics axis. The two target different axes and are not substitutes.

(c) Remediation rule. If you cannot state why a wider range is not “safer”, re-read §L.A.2 before Part B.

Part B — Distribution and range

§L.B.1 — The range-from-uncertainty rule

The range’s source is the 3.4.1 §L.B.2 identification’s interval. The rule is:

where $\hat{\phi}_i$ is the identified value and $\sigma_i$ is its identification’s standard deviation. The factor 2 is a 95% interval’s half-width; the factor’s choice is the rule’s only freedom, and its justification is the identification’s confidence, not a safety margin.

The rule’s two failures are the 3.4.1 §L.B.4 double edge’s two directions.

Too narrow. The real parameter is outside the interval and the policy’s behaviour is undefined. The signature is a deployment failure on the real parameter with a success rate that the twin-track did not predict.

Too wide. The policy is conservative against perturbations the real robot does not have, and the conservatism is a cost. The signature is a lower success rate on the real parameter than a narrower range would have given, with the 3.4.1 §L.D.1 performance-robustness curve’s optimum at a narrower range.

The rule’s operational statement: the range is the identification’s interval, not the safety’s. The two are different, and the confusion is the second misconception.

§L.B.2 — The distribution family

The distribution $p(\phi)$ is chosen from the parameter’s support and its effect on the task. Three families cover the running case’s parameters.

Uniform. For a parameter whose support is bounded and whose values are equally plausible: the friction’s interval $[0.3, 0.7]$ and the contact stiffness’s $[0.5\times10^5, 1.5\times10^5]$. The uniform is the default because it is the maximum-entropy distribution over the interval.

Log-uniform. For a positive parameter whose relative uncertainty is large and whose effect is multiplicative: the delay’s interval $[0, 20]$ ms is not log-uniform because its lower bound is zero, but a delay’s interval $[1, 20]$ ms would be. The log-uniform assigns equal probability to each decade, which is the right prior when the parameter’s order of magnitude is more uncertain than its exact value.

Gaussian. For a parameter whose support is unbounded and whose uncertainty is symmetric: a measurement error. The Gaussian is the default when the identification’s own noise is Gaussian and the parameter’s physical bound is far from the mean. The Gaussian’s tails are the risk: a sample three standard deviations out is a plausible value for the training and an implausible value for the real robot, and the sample’s rarity in the batch means the policy is under-trained there.

The running case’s choices are in §L.E.1, but the principle is stated here: the distribution is chosen from the parameter’s support and its effect’s form, not from the convenience of a sampling call. A uniform over a bounded interval, a log-uniform over a ratio scale, and a Gaussian over a measurement are the three defaults, and a parameter whose effect is multiplicative and whose order of magnitude is uncertain is the log-uniform’s case.

Self-explanation 2. Why is the uniform distribution the default over a bounded interval?

Answer. Because it is the maximum-entropy distribution over the interval, so it is the least informative choice given only the bounds, which is what the identification supplies.

§L.B.3 — Correlated parameters

The randomization’s samples are usually drawn independently across parameters, which assumes the parameters are independent. They are not. The friction and the contact stiffness are correlated by the materials: a rubber pad has higher friction and lower stiffness than a metal pad, and an independent sample of (high friction, high stiffness) is a pair that no material exhibits.

The independent sampling’s consequence is a policy that is robust to a distribution the real robot does not have, and the 3.4.1 §L.B.4 double edge applies: the unrepresented correlation is an error the policy trusts.

Two mitigations are available.

Parameterize by the physical cause. Randomize the material’s identity — rubber, plastic, metal — and compute the friction and the stiffness from the material’s model. The correlation is then represented by the material’s distribution, and the sample is physically consistent.

Fit the joint distribution. Identify the parameters’ joint from the identification’s data, and sample from the joint rather than independently. The fit needs enough identification data to estimate the correlation, which is the 3.4.3 lesson’s target.

The running case’s material choice is a two-level parameter — the rubber pads’ material and the cube’s plastic — and the correlation is mild; the independent sampling’s error is small. The dexterous hand’s case is stronger, which is why 2.3.4 §L.D.3’s randomization was over the material’s identity rather than over the friction and stiffness independently.

§L.B.4 — Sample complexity

Each randomized parameter adds a dimension to the training distribution, and the sample complexity grows with the dimension. The precise growth depends on the policy’s capacity and the task’s smoothness, but the worst-case bound is exponential:

where $d$ is the number of randomized dimensions and $\epsilon$ is the required resolution in each dimension. For $d = 5$ and $\epsilon = 0.1$, the bound is $10^5$; for $d = 10$, it is $10^{10}$. The bound is pessimistic — the policy is usually smooth in the parameters, so the growth is polynomial in practice — but its structure is the reason the randomized set must be chosen, not enumerated.

The practical arithmetic is the one the lab uses: the number of environments times the episodes per environment times the steps per episode is the sample budget, and the budget must be at least the effective sample complexity. The running case’s budget is the 2.2.4 100,000 transitions plus the 3.1.3 4,000 episodes; at 50 steps per episode, that is $200{,}000$ steps, which supports a $d$ of about 5 with a moderate resolution — which is the running case’s count. The completion in WE-L.B.c below makes this concrete.

Worked-example ladder.

WE-L.B.a (fully worked). The friction’s interval is $[0.3, 0.7]$ and the mass’s is $[0.08, 0.12]$. Compute the uniform distributions’ densities and the probability that a sample has friction below 0.4 and mass above 0.11.

Solution. The friction’s density is $1/0.4 = 2.5$ per unit; the mass’s is $1/0.04 = 25$ per unit. The joint probability, under independence, is $(0.1/0.4) \times (0.01/0.04) = 0.25 \times 0.25 = 0.0625$, or 6.25%.

WE-L.B.b (completion). The friction’s distribution is changed to a log-uniform over the same interval. Compute the probability that a sample is below 0.4, and state the effect on the training’s coverage.

Completion. The log-uniform’s density is proportional to $1/\mu$, so the CDF is $\log(\mu/0.3)/\log(0.7/0.3)$. At $\mu = 0.4$: $\log(1.333)/\log(2.333) = 0.2877/0.8473 = 0.34$. The probability is 34% versus the uniform’s 25%, so the log-uniform concentrates more mass near the lower bound, which is where the grip force’s requirement is highest. The completion shows the distribution’s choice is a coverage decision.

WE-L.B.c (faded). Compute the sample budget for $d = 5$ and $\epsilon = 0.2$ under the exponential bound, and compare it with the running case’s 200,000 steps.

Faded step. $N \sim 0.2^{-5} = 3{,}125$. Completion. The bound is well below the running case’s budget, so the budget is sufficient for $d=5$ and the exponential bound is not the binding constraint at this resolution. The completion shows the bound is pessimistic for smooth tasks and the operational arithmetic is the budget’s.

WE-L.B.d (contrastive near-miss). A team randomizes ten parameters with independent uniform distributions and trains for the running case’s budget. The policy’s success on the real cell is 55%, versus 90% for a policy trained on three parameters. Why it fails: the ten-dimensional distribution’s samples are too sparse for the budget, and the policy’s capacity is spent on the six unimportant parameters. The fix: the randomized set is the parameters whose variation affects the task’s success, chosen by the 3.4.1 §L.B.3 sensitivity analysis.

Comparative table: distribution families.

Design principle P47.2. Choose the distribution from the parameter’s support and its effect’s form; choose the range from the identification’s interval; choose the set from the sensitivity analysis.

Consolidation checkpoint 2

(a) Three capabilities. You can apply the range-from-uncertainty rule. You can choose a distribution family for a given parameter. You can compute the sample-complexity bound and compare it with a budget.

(b) Transfer probe. A parameter’s identification gives $0.5 \pm 0.1$ and the parameter is positive with a multiplicative effect. Which distribution, and why? Answer: log-uniform over $[0.3, 0.7]$, because the effect is multiplicative and the relative uncertainty is 20%, which is the regime where the log-uniform’s coverage is better.

(c) Remediation rule. If you cannot state why the Gaussian’s tails are a risk, re-read §L.B.2 before Part C.

Part C — Training and curriculum

§L.C.1 — The sample schedule

The randomization’s samples are drawn per environment, not per step. The standard implementation parallelizes $M$ simulator instances, each initialized with its own parameter sample, and the batch is the $M$ instances’ experience. The parameter is held fixed within an episode and resampled between episodes.

The schedule has two consequences.

The parameter’s variation is between episodes. A policy cannot learn to infer the parameter from the observation within an episode, because the episode’s parameter is fixed; the policy can only learn to be robust across episodes. If the task needs inference — the policy must identify the friction to choose the grip — the parameter must vary within the episode, which is the time-varying randomization of Chebotar et al. 2019.

The batch’s diversity is the number of environments. A batch of $M$ environments has $M$ parameter samples, and the sample’s coverage of the distribution is the batch’s. A small $M$ means the batch’s samples are clustered and the gradient is computed on a narrow slice; a large $M$ means the batch’s coverage is better and the gradient is more representative.

The running case’s $M$ is 4,096, which is the 2.2.4 distributed training’s environment count, and its resampling is per episode.

§L.C.2 — Automatic domain randomization

A fixed range is a design choice made before training; automatic domain randomization (ADR; Akkaya et al. 2019, tier A, S) makes the range a learned quantity. The method maintains a per-parameter range that expands when the policy’s performance on the current range exceeds a threshold, and contracts when the performance falls.

The ADR’s update is, for a parameter $i$ and an expansion rate $\delta$,

where $\text{perf}_i$ is the policy’s success rate on the current range’s boundary samples and $\tau$ is the threshold. The expansion is per-parameter, so each parameter’s range reaches the level the policy can handle, and the final range is the policy’s capability’s, not the designer’s guess.

The ADR’s strengths are two. It removes the range’s design choice, which is the 3.4.1 §L.B.4 double edge’s source. It allocates the policy’s capacity to the parameters that are hard, because a parameter whose range expands slowly is one the policy cannot handle, and the curriculum’s expansion is the diagnostic.

The ADR’s weakness is its cost: the expansion’s boundary samples must be evaluated, which is additional simulator time, and the curriculum’s progression is a hyperparameter that the design must tune. The running case’s ADR is the 3.4.4 lesson’s option, not this lesson’s default; this lesson uses the fixed identification interval, which is the simpler design and the one the range-from-uncertainty rule justifies.

Self-explanation 3. Why does the ADR’s per-parameter expansion allocate capacity to the hard parameters?

Answer. Because a hard parameter’s range expands slowly — the policy fails at the boundary — and the curriculum spends more time on the range where the policy is weak, so the capacity is spent where it is needed.

§L.C.3 — The set selection and the sensitivity analysis

The randomized set is chosen by the 3.4.1 §L.B.3 sensitivity analysis: for each candidate parameter, measure the task’s success rate’s change when the parameter is varied across its interval with the others fixed. The parameters whose variation changes the success rate by more than the task’s tolerance are the ones to randomize.

The running case’s sensitivity analysis gives:

• the friction and the mass: high sensitivity (the grasp’s success is 90% at nominal and 40% at the interval’s worst case);
• the contact stiffness: moderate sensitivity (the chatter’s threshold changes but the grasp’s success does not);
• the delay: low sensitivity (the 2.3.3 controller’s low gain tolerates the range);
• the appearance: high sensitivity on the perception axis (the 3.4.1 §L.B.2’s 45% contribution).

The set is the four parameters whose sensitivity exceeds the tolerance; the stiffness and the delay are randomized over a narrower range as a precaution, and the appearance’s randomization is the 3.4.4 lesson’s adaptation rather than this lesson’s appearance randomization.

The set’s size is the sample-complexity dimension $d$, and the 3.4.1 §L.B.2’s budget is the constraint. The running case’s $d$ is 5 (friction, mass, stiffness, delay, and the appearance’s scale), which is the count the budget supports.

Worked-example ladder.

WE-L.C.a (fully worked). The ADR’s expansion rate is $\delta = 0.01$ per parameter, the threshold $\tau = 0.8$, and the policy’s boundary success is 0.85 at the current range. Compute the expansion and the new range for the friction’s initial $[0.45, 0.55]$.

Solution. The success exceeds the threshold, so both bounds expand: $[0.44, 0.56]$. The expansion continues until the boundary’s success falls below 0.8, at which point the range is the policy’s capability’s.

WE-L.C.b (completion). After ten expansions, the friction’s range is $[0.35, 0.65]$ and the boundary’s success is 0.82. Compute the next expansion and state whether the range is close to the identification interval’s $[0.3, 0.7]$.

Completion. The next range is $[0.34, 0.66]$, and the identification’s $[0.3, 0.7]$ is two more expansions away. The completion shows the ADR’s range converges toward the identification’s from below, and the convergence’s stopping point is the capability’s, not the identification’s — which is the ADR’s diagnostic: if the range stops short, the policy’s capacity is the constraint.

WE-L.C.c (faded). Derive the sensitivity analysis’s statistic and state its test. Faded step. For parameter $i$, the statistic is $\Delta_i = \text{success}(\phi_i^{lo}) - \text{success}(\phi_i^{hi})$; the test is whether $|\Delta_i|$ exceeds the tolerance’s margin. Completion. The statistic is a difference of success rates, and its test is a per-parameter univariate sweep; the interaction terms are the correlated-parameter problem’s, and the joint analysis is the 3.4.3 lesson’s.

WE-L.C.d (contrastive near-miss). A team’s ADR expands the range until the policy’s boundary success is 0.5 and finds the real cell’s success at 0.6. Why it fails: the threshold was set too low, so the range expanded past the policy’s capability and the boundary samples dominate the training’s loss. The fix: the threshold’s value is the policy’s acceptable performance, and 0.5 is below the deployment’s requirement; raise it to the requirement’s level.

Comparative table: fixed range vs. ADR.

Design principle P47.3. The randomized set is chosen by the sensitivity analysis, and the set’s size is the sample budget’s constraint; randomize what the task’s success depends on, not everything.

Consolidation checkpoint 3

(a) Three capabilities. You can state the per-environment sample schedule. You can write the ADR update and interpret its convergence. You can run the sensitivity analysis and select the set.

(b) Transfer probe. A parameter’s sensitivity is 0.02 (a 2% success change) and the task’s tolerance is 5%. Should it be randomized? Answer: no; its variation does not exceed the tolerance, and the randomization would add a dimension to the sample complexity for a benefit the tolerance cannot detect.

(c) Remediation rule. If you cannot state why the ADR’s threshold is the policy’s acceptable performance, re-read WE-L.C.d before Part D.

Part D — The performance-robustness curve and the pitfalls

§L.D.1 — The curve’s shape

The performance-robustness curve is the policy’s success rate as a function of the range’s width, with the range centered on the identification’s mean. The curve has a characteristic shape.

At zero width. The nominal policy’s success on the nominal simulator. It is the curve’s maximum on the training distribution, and it is not a valid deployment estimate because the real parameter is not the nominal one.

At the identification’s width. The policy’s success on the real cell, which is the curve’s operational value. It is lower than the nominal success because the range includes the real parameter’s plausible values.

Beyond the identification’s width. The policy’s success on the real cell falls, because the range’s expansion makes the policy conservative against perturbations the real robot does not have, and the conservatism is a performance cost paid for robustness that is not exercised. The 3.4.1 §L.D.1 spectrum’s second axis is the resource, and the curve’s maximum is at the identification’s width.

The curve’s empirical form is what the running case’s §L.E uses. The lab’s H1–H4 hypotheses test its shape with a synthetic model, and the curve’s three regimes are the values the deployment’s choice uses.

§L.D.2 — The four pitfalls and their signatures

Range mis-specification. The identification’s interval is wrong. Signature: the real cell’s success is below the twin-track’s prediction, and the error is on the 3.4.1 §L.C.1 tail. Fix: more identification data, or a wider range if the identification’s interval was too narrow.

Distribution mis-specification. The randomization’s shape is wrong. Signature: the real cell’s parameter is at the distribution’s edge, and the success there is lower than the interior’s. Fix: the distribution family’s choice, per §L.B.2.

Correlated parameters. The independent sampling produces physically implausible combinations. Signature: the policy’s behaviour on the real cell is better than the twin-track predicts, because the real parameter is in a region the independent sampling under-covered. Fix: the material parameterization, per §L.B.3.

Over-conservatism. The range is wider than the identification’s. Signature: the real cell’s success is lower than a narrower range’s, and the policy’s peak forces are higher. Fix: the range-from-uncertainty rule, per §L.B.1.

The four pitfalls are distinguished by the direction of the twin-track’s error: mis-specification has the real cell worse than predicted, correlated parameters have it better than the independent sampling predicts, and over-conservatism has the narrower range better than the wider. The direction is the signature.

§L.D.3 — The robustness’s boundary

The randomized policy’s robustness is over the training distribution’s support. A perturbation outside the support is out of distribution, and the policy’s behaviour there is undefined.

The boundary’s consequence for deployment is that the randomization is not a guarantee against the real robot’s drift. A gripper’s friction falls as its pads wear, and the fall is outside the training interval after enough wear. The 3.1.4 monitor’s drift detection is the mechanism that catches it, and the 3.3.4 register’s “friction drift” entry is the residual’s record.

The boundary’s consequence for the evaluation is that the twin-track’s coverage argument must include the real parameter’s trajectory, not only its value. The 3.4.1 §L.C.3 metrics’ real-cell sample is one time’s; the drift’s measurement is the 3.4.4 lesson’s real-to-sim adaptation’s feedback loop.

Self-explanation 1. Why is a randomized policy’s behaviour outside its range undefined rather than merely worse?

Answer. Because the policy’s action at a state is a function of the observation alone, and the observation does not reveal the parameter; the policy’s action was chosen for the training distribution’s states, and the out-of-range parameter produces states the training never visited.

Concept check. A policy’s twin-track success is 0.88 and the real cell’s is 0.75. The direction is the real cell worse. Which pitfall’s signature?

Answer. Range or distribution mis-specification — the real parameter is at or beyond the interval’s edge — not correlated parameters, which would have the real cell better than the independent sampling predicted. [displaces M5]

Worked-example ladder.

WE-L.D.a (fully worked). The performance-robustness curve’s three points are (0 width, 0.95), (identification’s width, 0.90), (2x identification’s width, 0.82). Compute the curve’s slope between the first two points and between the last two, and state the interpretation.

Solution. The first slope is $(0.95 - 0.90)/w = 0.05/w$; the second is $(0.90 - 0.82)/w = 0.08/w$. The second slope is steeper, which means the widening beyond the identification’s width costs more than the widening to it — the over-conservatism’s signature. The curve’s maximum is at the identification’s width, and the wider range’s cost is larger than the narrower range’s.

WE-L.D.b (completion). The curve’s third point is measured at 0.88 rather than 0.82. Recompute the slopes and state whether the over-conservatism is present.

Completion. The second slope is $0.02/w$, flatter than the first’s $0.05/w$; the wider range’s cost is smaller, so the over-conservatism is mild and the identification’s width is not sharply optimal. The completion shows the curve’s shape is empirical, and the choice’s sharpness depends on the task’s sensitivity.

WE-L.D.c (faded). Derive the correlated-parameter signature’s direction from the independent sampling’s coverage. Faded step. The real parameter’s correlated pair is in a region the independent sampling under-covers, so the policy’s local behaviour is better than the average’s. Completion. The direction is real cell better than predicted, which is the opposite of the mis-specification’s direction; the two are distinguished by the sign of the twin-track’s error.

WE-L.D.d (contrastive near-miss). A team’s twin-track shows the real cell better than predicted and they widen the range to “capture the real behaviour”. The real cell’s success then falls. Why it fails: the signature was the correlated-parameter problem’s direction, and the fix is the material parameterization, not a wider range. The fix: parameterize by the material and re-sample, per §L.B.3.

Comparative table: the pitfalls and their signatures.

Design principle P47.4. Read the twin-track’s error’s direction; the four pitfalls have different directions and different fixes, and the direction is the diagnostic.

Consolidation checkpoint 4

(a) Three capabilities. You can sketch the performance-robustness curve’s three regimes. You can diagnose each pitfall from the direction of the twin-track’s error. You can state the robustness’s boundary and its consequence for drift.

(b) Transfer probe. The real cell is better than the twin-track predicted. Which pitfall, and why not over-conservatism? Answer: correlated parameters, because the independent sampling under-covered the real parameter’s region; over-conservatism has the real cell worse than a narrower range’s, which is a different comparison.

(c) Remediation rule. If you cannot state why the direction distinguishes the pitfalls, re-read §L.D.2 before Part E.

Part E — The running case

§L.E.1 — The randomization design

The running case’s randomization, from the 3.4.1 §L.B.2 intervals and the 3.4.1 §L.B.3 sensitivity analysis.

The set’s dimension $d = 5$. The sample budget’s 3.1.3 4,000 episodes at 50 steps gives 200,000 steps; the 3.4.1 §L.B.2’s exponential bound at $\epsilon = 0.2$ is about $3 \times 10^3$, well below the budget, and the effective sample complexity is the budget’s constraint, not the bound.

The friction and the mass are the high-sensitivity pair, and the 3.4.1 §L.A.2’s grip-force range is the design’s consequence: the policy’s grip must be above the $0.3$ friction’s 2.31 N requirement, and the standard 5 N grip’s margin at that friction is 2.2, within the 3.4.1 §L.A.2 threshold’s 1.5. The contact stiffness’s moderate sensitivity is why its range is the identification’s full interval rather than a narrower one: the chatter’s threshold is the stiffness’s, and the 2.3.3 controller’s gain is set below the chatter’s frequency for the whole range.

The delay’s low sensitivity is why its range is the full 0-20 ms despite its being the largest relative uncertainty: the 2.3.3 controller’s low gain tolerates the range, and the randomization is a precaution rather than a necessity.

§L.E.2 — The appearance’s handling

The appearance’s scale is the perception axis’s randomization, and its treatment is the 3.4.4 lesson’s. The running case’s design uses a log-uniform over $[0.9, 1.1]$ as a placeholder for the adaptation: the 3.4.4 adaptation fits a mapping from the simulated appearance to the real one, and the placeholder’s range is the adaptation’s residual’s.

The placeholder’s justification is the 3.4.1 §L.E.2’s sequence: the perception axis’s adaptation is first because it dominates the tail, and the appearance’s randomization is the interim measure that the adaptation replaces. The two are not substitutes; the randomization is the interim, and the adaptation is the target.

§L.E.3 — The schedule and the evaluation

The schedule is the 3.4.2 §L.C.1’s: 4,096 parallel environments, per-episode resampling, and the fixed range from the 3.4.1 intervals. The training budget is the 3.1.3 4,000 episodes plus the 2.2.4 100,000 transitions, and the 3.4.1 §L.C.3’s budget arithmetic shows the dimension’s feasibility.

The evaluation is the 3.4.1 twin-track after the randomization, with the 3.4.1 §L.C.1’s tail metric. The expected tail’s reduction from the nominal is the randomization’s measured benefit: the nominal tail was 4.5 cm, the randomization’s expected tail is about 3.2 cm, and the remaining gap is the perception axis’s, which the 3.4.4 adaptation handles.

The evaluation’s acceptance is the 3.4.1 §L.C.1’s tail against the 3 cm tolerance plus the 3.4.1 §L.D.2’s sequence’s remaining methods. The randomization alone’s 3.2 cm is near the tolerance and not within it, so the randomization’s benefit is real and insufficient; the 3.4.4 adaptation is the next method.

Self-explanation 2. Why is the randomization’s expected tail reduction (4.5 to 3.2 cm) not enough for the acceptance?

Answer. Because the tolerance is 3 cm and the tail is 3.2 cm; the difference is 0.2 cm, which is within the metric’s uncertainty, so the acceptance is not met and the remaining method is needed.

Worked-example ladder.

WE-L.E.a (fully worked). The grip force’s required value at the interval’s worst friction is 2.31 N, and the standard grip is 5 N. Compute the margin and the peak force’s cost relative to the nominal.

Solution. The margin is $5/2.31 = 2.16$, above the 1.5 threshold. The nominal’s margin was $5/1.39 = 3.6$, so the randomization’s cost is the margin’s reduction from 3.6 to 2.16, which is the policy’s conservatism paid for the friction’s uncertainty.

WE-L.E.b (completion). The standard grip is reduced to 4 N to lower the peak force, and the friction’s interval is unchanged. Recompute the margin and state whether the acceptance still passes.

Completion. The margin is $4/2.31 = 1.73$, still above 1.5, so the acceptance passes and the peak force’s reduction is 20%. The completion shows the grip’s choice is a trade between the margin’s threshold and the peak force’s cost, and the randomization’s interval sets the required value.

WE-L.E.c (faded). Write the register’s entry for the randomization’s residual after the 3.4.4 adaptation’s interim. Faded step. Failure: the friction drifts below 0.3 as the pads wear. Probability: from the wear’s rate. Consequence: the grip’s margin falls below 1.5. Mitigation: the 3.1.4 monitor and the 3.4.4 feedback. Residual: the drift’s unmonitored states. Completion. The residual is the drift’s states the monitor does not visit, which is the deployment’s distribution’s coverage gap, and the register’s residual column is the entry’s acceptance.

WE-L.E.d (contrastive near-miss). A team randomizes the cube’s mass over $[0.01, 1.0]$ kg “for robustness” and the real cell’s success is 60%. Why it fails: the range was ten times the identification’s, and the policy’s behaviour at the real mass is a sparse corner of the training distribution. The fix: the range is the identification’s interval, and the wider range’s cost is the over-conservatism’s, which the curve’s third regime measures.

Comparative table: the running case’s randomization.

Design principle P47.5. Randomize what the task’s success depends on, over the interval the identification gives, and measure the benefit against the cost’s margin.

Consolidation checkpoint 5

(a) Three capabilities. You can state the running case’s randomization table. You can compute the grip’s margin at the interval’s worst friction. You can state why the randomization alone is insufficient for the acceptance.

(b) Transfer probe. The friction’s interval’s lower bound falls to 0.2 because a new material is used. What happens to the required grip and the margin? Answer: the required force is $0.1 \times 9.81/(2 \times 0.2 \times 0.7071) = 3.47$ N; at 5 N the margin is 1.44, below the 1.5 threshold, so the grip must rise or the range must be narrowed by the material’s identification.

(c) Remediation rule. If you cannot compute the margin at the interval’s worst friction, redo WE-L.E.a before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: budget too small, learning stalls, and drift after training.

The lab

Task. Train a small policy on a synthetic 2-D task with five randomized parameters, plot the performance-robustness curve at three range widths, and run the correlated-parameter test by comparing independent and joint sampling.

Conditions. Python 3.10+, PyTorch 2.x, CPU sufficient. The task is a point-mass reach with a contact-like nonlinearity; the parameters are the mass, the friction, the stiffness, the delay, and the appearance’s scale. 500 episodes per range width. Seed all RNGs.

Controls. (i) Nominal training; (ii) fixed range at the identification’s width; (iii) fixed range at twice the identification’s width; (iv) the joint sampling’s correlated-parameter control.

Metrics.

Pre-registered hypotheses.

H1. The nominal training’s success at the nominal parameter is the highest, and its success at the interval’s worst is the lowest, because the nominal policy is brittle.

H2. The fixed-range training at the identification’s width has a success at the interval’s worst that is at least 30 percentage points above the nominal’s, because the robustness is over the right range.

H3. The twice-width training has a lower success at the nominal and at the interval’s worst than the identification’s-width training, because the over-conservatism costs performance without a benefit.

H4. The joint sampling’s success at the real parameter is higher than the independent sampling’s, because the correlation’s region is under-covered by the independent sampling.

Break-it sequence.

1. Widen the range tenfold silently. The success at the nominal falls and the interval’s worst does not improve. Silent: the training loss looks similar.
2. Use a Gaussian for the friction silently. The tails are physically implausible and the policy’s behaviour at the real friction is under-trained. Silent: the distribution looks normal.
3. Sample independently with a strong correlation silently. The joint region is under-covered. Silent: the marginals are correct.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The performance-robustness curve is plotted at three widths, not one. (2) The interval’s worst’s success is reported alongside the nominal’s, so the cost’s benefit is visible. (3) The correlated-parameter test is a comparison, not an assertion. (4) The break-it results are a table. (5) The distribution family’s choice is justified per parameter.

A runnable core, seeded and annotated:

# lab_3_4_2.py — randomization and the performance-robustness curve
import numpy as np, torch, torch.nn as nn
torch.manual_seed(0); np.random.seed(0)

$D_OBS$, $D_ACT$, $N_EP$, EP_LEN = 4, 1, 500, 50
INTERVAL = {"m": (0.08, 0.12), "mu": (0.3, 0.7),
            "k": (0.5e5, 1.5e5), "d": (0.0, 0.02)}

def sample_params(width_scale=1.0):
    p = {}
    for name, (lo, hi) in INTERVAL.items():
        mid, half = 0.5 * (lo + hi), 0.5 * (hi - lo) * width_scale
        p[name] = np.random.uniform(mid - half, mid + half)
    p["mu"] = max(p["mu"], 0.05)                       # physical floor
    return p

def simulate(p, policy):
    x, v = 0.0, 0.0
    for t in range(EP_LEN):
        a = policy.predict(np.array([x, v, p["m"], p["mu"]]))
        contact = p["k"] * max(0, x - 0.5) ** 2
        v = v + (a - p["mu"] * v - contact * 1e-5) * 0.05
        x = x + v * 0.05
    return float(abs(x - 0.5) < 0.05)                 # success = near target

class Policy(nn.Module):
    def __init__(self, h=64):
        super().__init__()
        self.net = nn.Sequential(nn.$Linear(D_OBS$, h), nn.Tanh(),
                                 nn.Linear(h, h), nn.Tanh(),
                                 nn.Linear(h, $D_ACT$))
    def forward(self, x):
        return self.net(x)
    def predict(self, x):
        with torch.no_grad():
            return self.forward(torch.tensor(x, dtype=torch.float32)).item()

def train(width_scale, epochs=300):
    policy = Policy()
    opt = torch.optim.Adam(policy.parameters(), lr=1e-3)
    for e in range(epochs):
        for _ in $range(N_EP$):
            p = sample_params(width_scale)
            x, v, states, actions = 0.0, 0.0, [], []
            for t in range(EP_LEN):
                a = policy.predict(np.array([x, v, p["m"], p["mu"]]))
                states.append([x, v, p["m"], p["mu"]]); actions.append([a])
                contact = p["k"] * max(0, x - 0.5) ** 2
                v = v + (a - p["mu"] * v - contact * 1e-5) * 0.05
                x = x + v * 0.05
            S = torch.tensor(states, dtype=torch.float32)
            A = torch.tensor(actions, dtype=torch.float32)
            loss = ((policy(S) - A) ** 2).mean()
            opt.zero_grad(); loss.backward(); opt.step()
    return policy

def evaluate(policy, p, n=100):
    return np.mean([simulate(p, policy) for _ in range(n)])

results = {}
for name, scale in [("nominal", 0.0), ("ident", 1.0), ("double", 2.0)]:
    policy = train(scale) if scale > 0 else train(0.05)
    results[name] = {
        "nominal": evaluate(policy, {k: 0.5 * (v[0] + v[1]) for k, v in INTERVAL.items()}),
        "edge":    evaluate(policy, {k: v[0] for k, v in INTERVAL.items()}),
    }
    print(name, results[name])

Line 5 defines the five parameters’ intervals; line 8 samples them with a width scale so the same code trains at three widths. Lines 14–20 simulate an episode with the contact law and the policy’s action. Lines 22–38 define the policy and its training. Lines 40–55 train and evaluate the three controls; the printed nominal and edge successes are the performance-robustness curve’s three points.

Cheat sheet

Exercises with solutions

E 3.4.2.1. State the training-distribution change and its cost.

S 3.4.2.1. The policy is trained on $P_{DR}$, the expectation over the parameter distribution; the cost is that the randomized problem is harder and the optimum’s value is lower, and the learning needs more samples. The lesson: robustness is bought with performance and samples, and the trade’s shape is the curve.

E 3.4.2.2 (transfer). A parameter’s identification gives $0.5 \pm 0.1$. Compute the range-from-uncertainty interval and state whether a safety margin should widen it.

S 3.4.2.2. The interval is $[0.3, 0.7]$. A safety margin should not widen it: the wider range makes the policy conservative against perturbations the real robot does not have, and the cost is paid without a benefit. The lesson: the range is the identification’s, not the safety’s.

E 3.4.2.3. Compute the grip force’s required value at friction 0.3 and at 0.7 with the $45°$ misalignment.

S 3.4.2.3. At 0.3: $0.1 \times 9.81/(2 \times 0.3 \times 0.7071) = 2.31$ N. At 0.7: $0.1 \times 9.81/(2 \times 0.7 \times 0.7071) = 0.99$ N. The lesson: the friction’s interval sets the grip’s range, and the worst case is the requirement.

E 3.4.2.4 (diagnostic). A team randomizes ten parameters and the policy’s success is 55%, versus 90% with three. Diagnose and state the fix.

S 3.4.2.4. The dimension is too large for the sample budget, and the six extra parameters dilute the signal. The fix is the sensitivity analysis’s set selection. The lesson: the set’s size is the sample-complexity’s constraint.

E 3.4.2.5. Compute the ADR’s next range for a friction interval $[0.35, 0.65]$ with $\delta = 0.01$ and boundary success 0.85 above the threshold.

S 3.4.2.5. The range expands to $[0.34, 0.66]$. The lesson: the ADR’s expansion is per-parameter and stops at the capability’s boundary.

E 3.4.2.6 (transfer). The real cell is better than the twin-track predicted. Which pitfall, and why not range mis-specification?

S 3.4.2.6. Correlated parameters, because the independent sampling under-covered the real parameter’s region; range mis-specification has the real cell worse than predicted. The lesson: the direction is the diagnostic.

E 3.4.2.7 (diagnostic). A policy’s peak force is 8 N and the standard grip is 5 N. Diagnose and state the fix.

S 3.4.2.7. The randomization’s conservatism, because the policy sets its grip for the interval’s worst friction and the peak is above the nominal. The fix is to lower the standard grip or narrow the range, with the margin’s threshold as the constraint. The lesson: the conservatism’s cost is a force cost, not only a success cost.

E 3.4.2.8. Plot the performance-robustness curve’s three points and state the regime each represents.

S 3.4.2.8. Zero width: the nominal’s maximum, not a deployment estimate. Identification’s width: the operational value. Twice the width: the over-conservatism’s cost. The lesson: the curve’s maximum is at the identification’s width, and the wider range’s cost is larger than the narrower’s.

E 3.4.2.9 (transfer). The friction’s lower bound falls to 0.2 and the grip’s margin at 5 N falls below 1.5. State the options and the choice.

S 3.4.2.9. Raise the grip to 5.2 N, or narrow the range with a material identification, or change the standard grip’s policy. The choice is the material’s identification, because the lower bound’s fall is the identification’s evidence’s. The lesson: the margin’s threshold is the design’s constraint, and the option that addresses the cause is the one that changes the identification.

Exit test

You are ready for Lesson 3.4.3 when you can, without notes:

1. State the training-distribution change and its cost.
2. Name the three families and their axes.
3. Apply the range-from-uncertainty rule.
4. Choose a distribution family for each of the running case’s parameters.
5. State the correlated-parameter problem and two mitigations.
6. Compute the sample-complexity bound and compare it with a budget.
7. State the per-environment sample schedule.
8. Write the ADR update and interpret its convergence.
9. Run the sensitivity analysis and select the randomized set.
10. Sketch the performance-robustness curve’s three regimes.
11. Diagnose each pitfall from the twin-track’s error’s direction.
12. State the robustness’s boundary and its consequence for drift.
13. Compute the grip’s margin at the interval’s worst friction.
14. State the running case’s randomization table and its costs.
15. Write the register’s drift entry with its residual.

Cross-references. 3.4.1 supplies the four axes, the identification intervals, the twin-track, and the performance-robustness trade restated in §L.A.1 and §L.B.1. 2.3.3 supplies the contact law and the force budget restated in §L.A.1 and §L.E.1. 2.3.4 supplies the randomization-from-uncertainty rule and the teacher–student pattern restated in §L.B.3. 3.2.1 supplies the ELBO’s reconstruction term restated in §L.A.3. 3.1.4 supplies the shadow deployment and the monitor restated in §L.A.2 and §L.D.3. 3.3.4 supplies the register restated in §L.E.3. Lesson 3.4.3 supplies the identification intervals this lesson randomizes over. Lesson 3.4.4 uses the family taxonomy and the appearance’s placeholder as its starting point.

Ready. The next lesson to write is 3.4.3 — System Identification. Say “next” to generate it.