Lesson 3.1.4 — VLA Fine-Tuning: the pipeline and the pitfalls

Week 19

Prerequisite lessons: 3.1.1 (introduction to VLAs), 3.1.2 (VLA architectures), 3.1.3 (data collection and annotation), 2.2.4 (offline-to-online fine-tuning), 2.3.3 (contact-rich manipulation), 2.3.4 (in-hand and dexterous manipulation).

Prior results used in this lesson. From 3.1.1: the VLA interface and the distinction between semantic breadth and motor competence. From 3.1.2: the four-block anatomy, the adaptation ladder, the web-data ablation, gradient-norm balancing, and the LoRA budget. From 3.1.3: the episode schema, the quality gate, the annotation contract, and the episode-level split. From 2.2.4: the seven-phase deploy–collect–fine-tune–monitor loop and the runtime Q-shield. Each is restated at the point of use.

What this lesson hands forward. The fine-tuning protocol, the forgetting audit, the calibration and safety gate, and the rollback criteria. Module 3.2 (world models) assumes a fine-tuned policy exists and asks how to plan with it; Module 4.2 uses the protocol as the policy-layer integration step; Module 4.3 uses the forgetting and calibration audits as experiment templates.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, and divider are unchanged; this lesson takes the pretrained VLA behind the instruction channel and the curated dataset from 3.1.3, and asks how to fine-tune the former on the latter without destroying what the former already knows.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) write the seven-phase VLA fine-tuning pipeline and state each phase’s exit condition; (2) run a baseline audit that measures seen-task, unseen-object, and paraphrase performance before any training; (3) match an adaptation strategy to a change type using the adaptation ladder; (4) compute a forgetting metric and interpret it against the baseline; (5) write the composite fine-tuning loss and balance it by gradient norm; (6) set a replay ratio and justify it from forgetting risk; (7) compute an action-scale shift and state the frequency convention that prevents it; (8) set a Q-shield threshold from offline validation and recalibrate it after fine-tuning; (9) design the shadow deployment and the monitor’s trough criteria; (10) write rollback criteria and the commit decision rule.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 2.2.4, the seven phases were pretrain, safety check, deploy, collect, fine-tune, monitor, and commit or rollback. Which phase produces data that a static offline dataset cannot contain?

Answer. The deploy-and-collect phase produces data on the deployed policy’s own state distribution, including states the policy reaches after its own errors. Used in §L.A.1 to place the collection phase in the VLA pipeline.

Q2. In 3.1.2, the web-data ablation showed that removing web co-fine-tuning degrades unseen-object generalization while seen-task accuracy is preserved. What does that imply about what a fine-tuning run must protect?

Answer. It implies that the semantic path carries generalization and the robot data carries motor competence, so fine-tuning must protect the semantic path while adapting the motor path. Used in §L.B.1 to justify adapters on the backbone and a full head retrain for new action spaces.

Q3. In 3.1.3, the episode-level split is required because adjacent frames are near-duplicates. What is the analogous trap in fine-tuning evaluation?

Answer. Evaluating on frames or episodes from the same collection batch as training overstates success; the evaluation split must be stratified by episode, task, object, operator, and batch. Used in §L.D.1 to define the held-out audit set.

Q4. In 2.3.3, the Q-shield rejects actions whose predicted value falls below a threshold. Why does the threshold depend on the policy that generated the value function?

Answer. Because the value distribution is a property of the policy and the data it visits; a policy change shifts the distribution, so a threshold calibrated on one policy is miscalibrated for another. Used in §L.D.3 to require recalibration after fine-tuning.

Part A — The seven-phase VLA fine-tuning pipeline

§L.A.1 — The pipeline, extended from 2.2.4

The 2.2.4 pipeline was built for a policy that starts from scratch or from a small offline dataset. A VLA starts from a pretrained model with capabilities that the running case’s data did not teach and cannot afford to lose. The seven phases therefore change in content while keeping their order.

Phase 1: baseline audit. Before any training, measure the pretrained VLA on the running case’s held-out audit set. The audit has three axes: seen task with seen object, seen task with unseen object, and seen task with unseen instruction phrasing. The audit is the reference against which every later change is judged, and it is the phase most often skipped.

Phase 2: adaptation strategy selection. Choose what to train: the whole model, the vision-language path with adapters, the action head only, or a combination. The choice follows the adaptation ladder of 3.1.2 §L.C.2 and is restated in §L.B.1.

Phase 3: safety check before deployment. Run the fine-tuned candidate in simulation and on a restrained physical setup against the 2.3.3 force limits and the 2.2.4 Q-shield. The check has a pass condition, not a review meeting.

Phase 4: shadow deployment and collection. Run the candidate in shadow mode beside the incumbent policy, collect its decisions and the states it visits, and record interventions. Shadow mode is the VLA analogue of the 2.2.4 deploy phase and it is what produces data on the candidate’s own distribution.

Phase 5: the fine-tuning run. Train on the curated dataset from 3.1.3 plus the shadow-collected corrections, with the loss of §L.C.1. This is the phase the lesson is named for, and it is one of seven.

Phase 6: monitor. Deploy with the shield active and track success rate, intervention rate, force-limit violations, and latency. The monitor’s criteria are set in §L.D.2.

Phase 7: commit or rollback. Commit if the monitor’s criteria hold for the agreed window; roll back to the audited baseline otherwise. Rollback is a first-class outcome, not a failure of the process.

The order matters because each phase’s exit condition is the next phase’s entry condition. A pipeline that skips the baseline audit cannot distinguish improvement from noise; one that skips the safety check cannot deploy; one that skips shadow mode fine-tunes on the wrong distribution.

§L.A.2 — The baseline audit in detail

The audit is a measurement, and its value depends on being specific.

Seen task, seen object. The pretrained VLA’s success on the running case’s pick-and-place with the cube at the trained pose. This is the number the fine-tuning run must not degrade.

Seen task, unseen object. The same task with the cup, screwdriver, bolt, and sponge. This is the semantic-generalization number that the 3.1.2 web-data ablation says the pretrained model should have and that fine-tuning can destroy.

Seen task, unseen phrasing. The same task with paraphrases outside the training set. This is the language-generalization number that 3.1.3 §L.C.2 says is fragile when phrasing diversity is low.

Latency and power. The end-to-end latency at the deployment power budget, reported at the 95th percentile. Fine-tuning can change the model’s size and therefore its latency, so the audit number is a baseline for the deployment check.

The audit is run on the held-out split from 3.1.3 §L.E.3, with the same stratification. Reporting the three axes separately is what makes the later forgetting analysis diagnostic rather than aggregate.

Self-explanation 1. Why is the baseline audit run before the adaptation strategy is chosen, rather than after?

Answer. Because the strategy is chosen to protect the capabilities the audit shows are strong and to repair the ones it shows are weak; choosing without measuring is guessing.

Concept check. A team runs the baseline audit and finds 92% on seen tasks, 55% on unseen objects, and 80% on unseen phrasings. Which adaptation strategy does this suggest, and why?

Answer. Adapters on the vision-language path with a retrained head, because the semantic path is already strong on unseen objects relative to the phrasing axis, and the phrasing deficit is a fusion-and-language issue that adapters can address without full fine-tuning. [displaces M1]

Worked-example ladder.

WE-L.A.a (fully worked). The audit yields $S_{seen} = 0.92$, $S_{obj} = 0.55$, $S_{phrase} = 0.80$ with 400 test episodes per axis. Compute the 95% confidence half-width for each and decide whether the seen-task number is distinguishable from 0.95.

Solution. Standard error is $\sqrt{p(1-p)/n}$. For $p = 0.92$, $n = 400$: $\sqrt{0.0736/400} = 0.0136$, half-width $1.96 \times 0.0136 = 0.027$. The interval is $[0.893, 0.947]$, which excludes 0.95. For $p=0.55$: $\text{SE} = 0.0249$, half-width 0.049. For $p=0.80$: $\text{SE}=0.020$, half-width 0.039.

WE-L.A.b (completion). The team doubles the test set to 800 episodes per axis. Recompute the seen-task half-width and state whether the conclusion changes. Completion. $\text{SE} = \sqrt{0.0736/800} = 0.0096$, half-width 0.0188, interval $[0.901, 0.939]$, still excluding 0.95. The conclusion is unchanged; the audit is already precise enough for the decision.

WE-L.A.c (faded). Set the audit’s pass condition for a deployment that requires 90% seen-task success. Faded step. Use the lower confidence bound $p - z\sqrt{p(1-p)/n} \ge 0.90$. Completion. At $p=0.92$, $n=400$, the lower bound is 0.893, below 0.90, so the audit does not pass at the required confidence even though the point estimate does. The fix is more episodes or a better model.

WE-L.A.d (contrastive near-miss). A team skips the baseline audit and fine-tunes directly. After training, the seen-task success is 89%. The team cannot tell whether fine-tuning helped, hurt, or did nothing. Why it fails: without a baseline, the post-training number has no reference. The fix: run the audit on the pretrained checkpoint from the same held-out split; it takes hours and it is the only way the result means anything.

Comparative table: the audit axes and what each protects.

Design principle P38.1. Audit before you train; the baseline is the only reference that makes the result interpretable.

Consolidation checkpoint 1

(a) Three capabilities. You can write the seven phases with exit conditions. You can run the three-axis audit and compute confidence intervals. You can read the audit to choose an adaptation strategy.

(b) Transfer probe. The audit shows 60% on unseen objects and 90% on unseen phrasings. Does the strategy change from the one suggested by the earlier example? Answer: yes; the semantic path is weak, so unfreezing the fusion block or co-training with web data moves up the priority list, while the language path needs less adaptation.

(c) Remediation rule. If you cannot compute the lower confidence bound of WE-L.A.c, redo it before Part B.

Part B — Adaptation strategies and forgetting

§L.B.1 — The adaptation ladder, restated and applied

The 3.1.2 adaptation ladder is restated here because the fine-tuning pipeline’s Phase 2 depends on it.

Frozen backbone with a new head. Train only the output module. It is cheapest and safest for forgetting, and it adapts output calibration and small action-space changes. It cannot adapt new representations, so it fails when the new task requires the model to attend to something the pretrained features do not encode.

Low-rank adapters. Train $W + \frac{\alpha}{r}BA$ with rank $r$ on selected matrices. It adapts style, object appearance, and instruction phrasing well. It adapts new motor skills poorly, and a new embodiment is usually out of reach.

High-rank adapters. Raise $r$ and adapt more matrices. It approaches full fine-tuning in capability and cost, and it is the practical compromise when full fine-tuning would forget.

Full fine-tuning. Train all parameters. It adapts everything and forgets everything that the new data does not reinforce. It requires replay or strong regularization to be safe at the running case’s data scale.

The matching rule is by change type: readout changes take a head, calibration and semantics take low-rank adapters, motor-representation changes take high-rank adapters or full fine-tuning with replay, and output-space changes take a reinitialized head regardless of the backbone choice.

§L.B.2 — Catastrophic forgetting, measured

Forgetting is not a vague worry; it is a measured quantity. Define, for each audit axis $i$,

where $S_i^{before}$ is the baseline audit and $S_i^{after}$ is the audit after fine-tuning on the same held-out set. A positive $F_i$ is forgetting; a negative $F_i$ is improvement. The aggregate forgetting is not the mean, because the axes have different costs: forgetting seen-object semantic breadth is recoverable with more web data, while forgetting seen-task competence is a deployment blocker.

The mechanism of forgetting is gradient interference. The fine-tuning gradient on the robot data points in a direction that improves the robot loss and, in general, moves the weights away from the region that supported the pretrained capabilities. The magnitude depends on three factors: the learning rate, the number of steps, and the ratio of new to old data. A small learning rate slows the drift but does not change its direction; replay changes the direction by adding old-data gradients; adapters constrain the direction to a low-rank subspace.

The running case’s data scale makes forgetting severe under full fine-tuning. From 3.1.3, 4,000 episodes is a fraction of a percent of the pretraining corpus, so the new gradient dominates unless the old behaviour is represented in the loss.

§L.B.3 — Replay and its ratio

Replay mixes a sample of the pretraining or audit data into each fine-tuning batch. Its effect is to keep the old gradient present, so the update direction is a compromise between new and old. The ratio is

where $N_{replay}$ and $N_{new}$ are the per-batch counts. At $\rho = 0$ there is no replay and full forgetting risk; at $\rho = 0.5$ the batch is balanced.

The running case cannot replay the pretraining corpus, which is not public. It can replay two substitutes: a held-out sample of the audit data (which preserves the axes the audit measures), and public manipulation data from Open X-Embodiment (O’Connell et al. 2023, tier A, S), which preserves general manipulation competence. The substitute is weaker than the original corpus but far better than nothing, and the ratio should be set from the forgetting budget: if the seen-object axis may lose at most three points, the ratio is raised until a pilot run shows the loss within that budget.

Self-explanation 2. Why does replay change the direction of the update while a smaller learning rate only changes its magnitude?

Answer. Because the replay gradient is added to the new-data gradient before the step, so the combined direction is a weighted sum; a smaller rate scales the same direction down.

Concept check. A pilot fine-tune with $\rho = 0.1$ loses 12 points on the unseen-object axis. The budget is 3 points. What is the next experiment, and what does it test?

Answer. Raise $\rho$ to 0.3 or 0.5 and rerun the pilot; it tests whether the forgetting is replay-limited or whether the adaptation strategy itself is too aggressive. If the loss persists at $\rho = 0.5$, switch to adapters. [displaces M2]

Worked-example ladder.

WE-L.B.a (fully worked). A full fine-tune takes the unseen-object audit from 0.55 to 0.43 and the seen-task audit from 0.92 to 0.94. Compute the forgetting on each axis and state which is the blocker.

Solution. $F_{obj} = 0.55 - 0.43 = +0.12$, a 12-point loss. $F_{seen} = 0.92 - 0.94 = -0.02$, a 2-point gain. The unseen-object loss is the blocker because it is the capability that 3.1.2 says the pretrained model uniquely supplies.

WE-L.B.b (completion). With replay at $\rho = 0.4$, the unseen-object audit is 0.52 and seen-task is 0.93. Compute the forgetting and decide whether the run is acceptable.

Completion. $F_{obj} = 0.03$, within the 3-point budget; $F_{seen} = -0.01$, a small gain. The run is acceptable on both axes, and the remaining decision is whether the 0.93 seen-task number is enough for deployment, which the safety gate tests.

WE-L.B.c (faded). Derive the replay ratio needed to keep the forgetting within a budget $B$ given a measured forgetting rate per unit of new-data exposure. Faded step. Measure $F(\rho)$ at two ratios, fit a linear model $F = a - b\rho$, and solve $a - b\rho \le B$. Completion. The linear model is an approximation; the operational rule is to sweep $\rho \in {0.1, 0.3, 0.5}$ and choose the smallest that meets the budget, because replay costs new-task learning speed.

WE-L.B.d (contrastive near-miss). A team sets $\rho = 0.9$ to “fully protect” the pretrained capabilities and finds that the fine-tuned model barely improves on the new task. Why it fails: the replay gradient dominates, so the update direction is almost entirely old-data, and the new task’s gradient is a small perturbation. The fix: lower $\rho$ to the smallest value that meets the forgetting budget, and use adapters for the rest of the protection.

Comparative table: adaptation strategies and their forgetting.

Design principle P38.2. Measure forgetting on each audit axis, not in aggregate; the axis that blocks deployment is the one the strategy must protect.

Consolidation checkpoint 2

(a) Three capabilities. You can apply the adaptation ladder to a stated change type. You can compute per-axis forgetting. You can set a replay ratio from a forgetting budget.

(b) Transfer probe. The new task requires a 7-DOF arm and a new gripper. Which strategy, and what is the minimum protection? Answer: full fine-tuning with a reinitialized head and replay at a ratio high enough to keep the unseen-object axis within budget; adapters alone cannot supply the output-space change.

(c) Remediation rule. If you cannot explain why replay changes direction and learning rate changes magnitude, re-read §L.B.3 before Part C.

Part C — The training run

§L.C.1 — The composite loss

The fine-tuning loss has three terms:

where $L_{robot}$ is the action loss on the curated dataset, $L_{web}$ is the vision-language loss on web data or replay, and $L_{aux}$ is any auxiliary loss such as the grounding loss from 3.1.3 §L.C.1. The weights are not sample counts; they are gradient-scale weights.

The 3.1.2 gradient-norm rule is restated: choose $\lambda_r$ so that

on a pilot batch, and re-estimate periodically because the robot loss falls faster than the web loss early in training. The rule is a running estimate, not a constant, and the ratio is a real hyperparameter with a measurable effect.

The auxiliary term matters more than its weight suggests. A grounding loss on the 2.1.2 pose targets forces the fusion block to keep the spatial precision that the action loss alone does not require. Without it, fine-tuning on action tokens can degrade the model’s ability to localize the cube even while its success rate holds on the training distribution.

§L.C.2 — The pitfalls, named

Five pitfalls account for most VLA fine-tuning failures, and each has a signature.

Catastrophic forgetting. Signature: seen-task success holds while unseen-object success falls. Diagnosed by the per-axis audit. Fixed by replay and adapters.

Action-scale shift. Signature: the policy moves at the wrong speed after the control frequency or the action range changes. The displacement per step is $\Delta x' = \Delta x \, f / f'$ when the frequency changes from $f$ to $f'$ at fixed per-step action, so a 10 Hz model deployed at 20 Hz moves half as fast per unit time. Fixed by freezing the frequency convention at training and deployment.

Loss imbalance. Signature: both losses fall while seen-task success falls, because the web gradient dominates. Diagnosed by the gradient-norm ratio. Fixed by Eq. (C.2).

Head mismatch. Signature: the model cannot reach the object at all after an embodiment change. Diagnosed by checking the output dimensionality and the action space against the training data. Fixed by reinitializing the head and retraining it, with adapters on the backbone.

Leakage. Signature: offline evaluation is far better than shadow deployment. Diagnosed by comparing the episode-level split against the collection batches. Fixed by re-splitting at the episode level with batch stratification, as in 3.1.3 §L.D.2.

Self-explanation 3. Why is action-scale shift silent in offline evaluation?

Answer. Because offline evaluation replays recorded actions at the recorded frequency; the mismatch appears only when the policy’s actions are executed at a different frequency.

§L.C.3 — The head decision

The action head is where the output space lives, and output spaces do not transfer by adapter. If the new embodiment has a different number of action dimensions, a different gripper range, or a different control frequency convention, the head must be reinitialized and trained on the new data. The backbone adapters then adapt the representation that feeds the head.

The head’s training schedule is separate from the backbone’s. A common recipe trains the head alone for a short warm-up, then unfreezes the adapters with a smaller learning rate. The warm-up prevents the randomly initialized head’s large gradients from distorting the pretrained representation before the head has learned a sensible output scale.

Worked-example ladder.

WE-L.C.a (fully worked). A model trained at 10 Hz with a 2 cm per-step range is deployed at 20 Hz with the same per-step range. Compute the time-domain speed in each case and the scale factor.

Solution. At 10 Hz, 2 cm per step is 20 cm/s. At 20 Hz, 2 cm per step is 40 cm/s, twice as fast. The policy was trained to produce 20 cm/s motions, so at 20 Hz it overshoots by a factor of two; equivalently, if the per-step range is held and the frequency doubled, the displacement per step should halve. The scale factor is $f/f' = 0.5$.

WE-L.C.b (completion). The deployment keeps the 10 Hz convention but the low-level controller interpolates to 1 kHz. Does the scale shift occur? Completion. No, provided the controller interpolates the 10 Hz setpoints rather than treating each setpoint as a 1 kHz command; the convention is the policy’s output rate, not the controller’s rate. The completion shows that the convention is about interpretation, not about the hardware clock.

WE-L.C.c (faded). Set the auxiliary grounding loss weight by matching its gradient norm to the action loss’s on a pilot batch. Faded step. Compute both norms, set $\lambda_a$ to the ratio, and verify that the grounding accuracy on the audit does not fall. Completion. The auxiliary weight is usually small, but its effect on spatial precision is visible in the 2.1.2 pose error, which is the metric to watch.

WE-L.C.d (contrastive near-miss). A learner warms up the head and the adapters together from step zero and finds the seen-task success never recovers to the baseline. Why it fails: the head’s random gradients are large and they propagate into the adapters, moving the representation before the head has a sensible scale. The fix: freeze the adapters during the head warm-up, then unfreeze with a lower rate for the adapters than for the head.

Comparative table: pitfalls and their signatures.

Design principle P38.3. Freeze the interface conventions before training; the action scale, the output dimensionality, and the control frequency are part of the model’s definition, not deployment details.

Consolidation checkpoint 3

(a) Three capabilities. You can write the composite loss and balance it by gradient norm. You can name the five pitfalls and their signatures. You can schedule the head warm-up.

(b) Transfer probe. A fine-tune improves seen-task success from 0.92 to 0.95 but unseen-object success falls from 0.55 to 0.40. Which pitfall, and what is the first experiment? Answer: catastrophic forgetting; raise the replay ratio and rerun the pilot before changing the adaptation strategy.

(c) Remediation rule. If you cannot state why action-scale shift is invisible offline, re-read §L.C.2 before Part D.

Part D — Evaluation, safety, and rollback

§L.D.1 — Offline evaluation and its limits

The fine-tuned candidate is first evaluated on the held-out split from 3.1.3 §L.E.3, with the three audit axes reported separately. The acceptance test is a lower confidence bound, not a point estimate:

where $p_{req}$ is the deployment requirement. The bound makes the acceptance criterion honest at the test set’s size; a point estimate above the requirement with a lower bound below it is not an acceptance.

Offline evaluation has two limits. It is measured on the curated distribution, which is narrower than deployment. And it replays the recorded action frequency and range, so it cannot detect action-scale shift. Both limits are why the safety gate and shadow deployment exist.

§L.D.2 — The safety gate

The safety gate runs the candidate against the physical constraints before any production deployment.

Force limits. The 2.3.3 impedance controller enforces a peak force limit; the candidate’s trajectories are checked in simulation against it, and any trajectory that would exceed it is rejected. The check is not a statistical average; it is a worst-case over the candidate’s evaluation rollouts.

Q-shield compatibility. The candidate’s actions are passed through the 2.2.4 Q-shield. If the rejection rate exceeds a threshold, the candidate is too far out of distribution for the shield to be a meaningful filter, and the run stops.

Recovery. The 1.1.4 state machine’s recovery paths are exercised with the candidate’s failures injected. A candidate whose failures cannot be recovered from is not deployable regardless of its success rate.

§L.D.3 — Shadow deployment, monitoring, and rollback

Shadow deployment. The candidate runs beside the incumbent, receives the same observations, and its actions are logged but not executed. The comparison is between the candidate’s chosen actions and the incumbent’s, and the interventions are recorded as corrective data for the next fine-tuning cycle. Shadow mode is the only phase that measures the candidate on the deployment distribution without risking the cell.

Monitoring. After promotion, the monitor tracks four quantities: success rate, intervention rate, force-limit violations, and 95th-percentile latency. The trough criteria from 2.2.4 apply: a sustained fall in success or a rise in interventions beyond the shadow-measured rate triggers review.

Shield recalibration. The Q-shield threshold is recalibrated on the fine-tuned policy’s validation rollouts, because the value distribution shifted with the policy. The recalibration uses the same procedure as 2.2.4, with the new policy’s rollouts as the calibration set.

Rollback. Rollback is triggered by any of: success below the audit’s lower bound for two consecutive windows; intervention rate above the shadow rate plus a margin; a force-limit violation; or latency above the budget at the 95th percentile. The rollback restores the audited baseline checkpoint, which is why the audit is the first phase and not the last.

Self-explanation 1. Why is the shadow deployment’s intervention rate the right reference for the monitor’s threshold, rather than the offline success rate?

Answer. Because the shadow rate is measured on the deployment distribution, while the offline success rate is measured on the curated one; the monitor must compare like with like.

Concept check. The candidate’s offline lower bound is 0.90, the shadow intervention rate is 4%, and the first production window’s intervention rate is 9%. What is the decision, and why not wait for a second window?

Answer. Roll back or pause, because the production rate is more than double the shadow rate and the discrepancy is itself the signal that the deployment distribution differs from the shadow distribution. The mechanism is that a doubled intervention rate is outside the shadow’s confidence interval at any reasonable window size. [displaces M5]

Worked-example ladder.

WE-L.D.a (fully worked). The candidate scores $\hat{p} = 0.93$ on 400 test episodes. The requirement is $p_{req} = 0.90$. Compute the lower bound and decide acceptance.

Solution. $\text{SE} = \sqrt{0.93 \times 0.07 / 400} = 0.0128$. $p_{lower} = 0.93 - 1.645 \times 0.0128 = 0.909$. This exceeds 0.90, so the candidate is accepted at the 95% one-sided level.

WE-L.D.b (completion). The test set is 200 episodes and $\hat{p} = 0.93$. Recompute and decide. Completion. $\text{SE} = 0.018$, $p_{lower} = 0.93 - 0.0296 = 0.900$, exactly at the requirement. The acceptance is marginal; the honest decision is to enlarge the test set rather than accept on a boundary.

WE-L.D.c (faded). Set the shield recalibration threshold so that the rejection rate on the validation rollouts matches the rejection rate on the pretraining policy’s rollouts. Faded step. Compute the value distribution’s 10th percentile on both sets and set the threshold to the new policy’s percentile. Completion. The match is approximate; the operational rule is to set the threshold so that the rejection rate on a held-out validation set is within one point of the pretraining rate, and to re-check after each fine-tuning cycle.

WE-L.D.d (contrastive near-miss). A team deploys the candidate immediately after offline acceptance and skips shadow mode. On the first day the intervention rate is 15% and the cell is stopped. Why it fails: the offline evaluation was on the curated distribution and could not reveal the deployment shift. The fix: shadow mode, which costs a day and reveals the shift before the cell is at risk.

Comparative table: evaluation stages.

Design principle P38.4. Accept on a lower bound, deploy behind a shield, promote after shadow, and keep the baseline checkpoint for rollback.

Consolidation checkpoint 4

(a) Three capabilities. You can compute the lower confidence bound and apply the acceptance rule. You can set the safety gate’s three checks. You can write the rollback triggers.

(b) Transfer probe. The shadow intervention rate is 5% and production is 6% after one week. Is that a rollback? Answer: not necessarily; the difference is within the shadow’s uncertainty for a week’s window, so the monitor continues and rechecks at the next window. The rollback criterion is a sustained or large divergence, not any difference.

(c) Remediation rule. If you cannot state why offline evaluation cannot detect action-scale shift, re-read §L.D.1 before Part E.

Part E — The running case: the full plan

§L.E.1 — The plan in numbers

The running case starts from an OpenVLA-scale checkpoint, the 3.1.3 dataset of 4,000 episodes with the six-field schema, and the three-axis audit.

Baseline audit. 400 episodes per axis, from the held-out split. Expected numbers from 3.1.1: about 0.90 on seen tasks, 0.55 on unseen objects, 0.80 on unseen phrasings. The audit takes one day including the test harness.

Adaptation strategy. Low-rank adapters on the vision-language path at rank 32, with a retrained action head because the running case’s gripper and action convention differ from the pretraining corpus. The head warm-up is 500 steps with adapters frozen, then 5,000 steps with adapters unfrozen at one-tenth the head’s learning rate.

Replay. A held-out sample of the audit data at $\rho = 0.3$, plus a public manipulation subset for general competence. The ratio is raised if the pilot shows the unseen-object loss exceeding three points.

Loss. The composite of Eq. (C.1), with the gradient-norm balance of Eq. (C.2) re-estimated every 500 steps, and an auxiliary grounding loss on the 2.1.2 pose targets at a weight matched to the action loss’s gradient norm.

Safety gate. Simulation rollouts against the 20 N peak force limit, the Q-shield rejection rate below 15%, and the 1.1.4 recovery paths exercised with injected failures.

Shadow deployment. Two days beside the incumbent, with interventions logged as corrective data. Promotion requires the intervention rate within two points of the incumbent’s and the action divergence below the 2.3.3 impedance controller’s tolerance.

Monitor and rollback. Four quantities tracked; rollback to the audited baseline on any trigger of §L.D.3.

§L.E.2 — The pitfalls in this plan

Each pitfall of §L.C.2 has a specific mitigation in this plan.

Forgetting is mitigated by replay at $\rho = 0.3$ and adapters at rank 32, with the unseen-object axis as the monitored blocker. If the pilot shows more than three points of loss, the ratio rises before the rank does, because replay is cheaper.

Action-scale shift is prevented by writing the frequency convention into the checkpoint metadata and asserting it at load time. The assertion is a one-line check that fails loudly if the deployment frequency differs.

Loss imbalance is handled by the gradient-norm re-estimation. The re-estimation is scheduled, not ad hoc, because the ratio drifts as the robot loss falls.

Head mismatch is handled by the reinitialized head and the warm-up schedule. The head’s output dimensionality is asserted against the 3.1.3 schema at load time.

Leakage is handled by the episode-level split from 3.1.3 and by comparing the offline result against the shadow result; a large gap is the leakage signature.

§L.E.3 — The decision rule

The plan’s output is a decision, and the decision rule is a conjunction.

Commit only if: the offline lower bound meets the requirement on all three axes; the safety gate passes all three checks; the shadow intervention rate is within two points of the incumbent’s; and the monitor’s first window shows no rollback trigger. Any failure means rollback or a new fine-tuning cycle, and the cycle uses the shadow-collected corrections as its new data.

The conjunction is deliberate. A candidate that passes offline but fails shadow is not deployable; one that passes shadow but fails the safety gate is not deployable; one that passes both but fails the monitor’s first window is rolled back. The decision is not a weighted score, because a single failed necessary condition is sufficient to stop deployment.

Self-explanation 2. Why is the decision a conjunction rather than a weighted score?

Answer. Because the conditions are necessary rather than tradeable: a force-limit violation is not compensated by high offline accuracy, so no weighting can make the deployment safe.

Worked-example ladder.

WE-L.E.a (fully worked). The fine-tune yields seen-task lower bound 0.91, unseen-object lower bound 0.52, unseen-phrasing lower bound 0.78. The requirements are 0.90, 0.50, and 0.75. Does the candidate pass the offline gate?

Solution. All three lower bounds exceed their requirements, so the offline gate passes. The margins are 0.01, 0.02, and 0.03; the seen-task margin is thin and the next cycle should add episodes to that axis if the requirement is firm.

WE-L.E.b (completion). The shadow intervention rate is 4% and the incumbent’s is 3%. The tolerance is two points. Does the candidate promote? Completion. The difference is one point, within tolerance, so promotion proceeds subject to the monitor’s first window.

WE-L.E.c (faded). The monitor’s first window shows a 6% intervention rate against the shadow’s 4%. Apply the rollback rule. Faded step. Compare against the shadow rate plus the margin. Completion. 6% exceeds 4% plus the one-point window uncertainty in the relevant direction; the rule pauses the deployment and triggers a review rather than an immediate rollback, because the window is short. The review decides whether to roll back or extend the window.

WE-L.E.d (contrastive near-miss). A team commits after the offline gate and skips the shadow and the monitor’s first window. The cell runs for three days with a 12% intervention rate before anyone notices. Why it fails: the offline gate cannot see the deployment distribution, and the monitor’s first window is what catches the shift early. The fix: shadow and monitor are not optional; they are the phases that measure what offline cannot.

Comparative table: the plan’s gates.

Design principle P38.5. The fine-tuning output is a decision under a conjunction of necessary conditions; treat any single failure as sufficient to stop deployment.

Consolidation checkpoint 5

(a) Three capabilities. You can state the running case plan’s numbers. You can map each pitfall to its mitigation. You can apply the decision rule to a result set.

(b) Transfer probe. The unseen-object lower bound is 0.49 against a 0.50 requirement. Which action, and why not accept? Answer: run another cycle with a higher replay ratio; accepting would deploy a model whose semantic breadth is below the requirement, and that axis is the one the pretrained model uniquely supplies.

(c) Remediation rule. If you cannot apply the conjunction rule to a result set, re-read §L.E.3 before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: head warm-up omission, interpolation omission, and shield miscalibration.

The lab

Task. Fine-tune a small instruction-conditioned policy on a synthetic reaching dataset under three settings — full fine-tuning, low-rank adapters, and frozen backbone with a new head — and measure per-task forgetting, new-task learning, and the effect of a replay ratio. The policy is a stand-in for a VLA: it consumes a synthetic observation vector and a one-hot instruction and outputs a 2-D action.

Conditions. Python 3.10+, PyTorch 2.x, CPU sufficient. Two tasks: task A is the pretraining task, task B is the new task. The dataset has 200 episodes per task, 50 steps each. Seed all RNGs.

Controls. (i) Full fine-tune without replay; (ii) full fine-tune at $\rho = 0.3$; (iii) low-rank adapter at rank 4; (iv) frozen backbone with a new head.

Metrics.

Pre-registered hypotheses.

H1. Full fine-tuning without replay loses at least 20 points on task A, because the task-B gradient dominates at this data scale.

H2. Replay at $\rho = 0.3$ reduces the task-A loss by at least half relative to H1, because the old gradient is present in every batch.

H3. The low-rank adapter retains task A better than full fine-tuning at equal task-B learning, because the update is confined to a low-rank subspace.

H4. The frozen-backbone control cannot learn task B beyond a calibration-level improvement, because the representation it needs is not in the frozen features.

Break-it sequence.

1. Change the action scale silently. Multiply the task-B actions by 0.5 at training time. The model learns a half-speed policy. Silent: the loss falls normally.
2. Mis-weight the replay silently. Set the replay loss weight to zero while keeping the replay data in the batch. The retention collapses. Silent: the data loader looks correct.
3. Freeze the fusion silently. Train only the head. Task B learning stalls at calibration level. Silent: the training curve looks like slow convergence.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The three controls share the same data, schedule, and evaluation, so the comparison is clean. (2) Forgetting is reported per task, not in aggregate. (3) The adapter fraction is computed, not estimated. (4) The break-it results are a table. (5) The action-scale assertion is a line of code that fails loudly.

A runnable core, seeded and annotated:

# lab_3_1_4.py — fine-tuning controls with forgetting measurement
import numpy as np, torch, torch.nn as nn
torch.manual_seed(0); np.random.seed(0)

$D_OBS$, $D_ACT$, $N_EP$, EP_LEN = 16, 2, 200, 50

def make_task(offset, $n_ep=N_EP$):                      # synthetic episodes
    obs = torch.$randn(n_ep$, EP_LEN, $D_OBS$)
    act = torch.tanh(obs[..., :$D_ACT$] + offset)        # task-specific map
    return obs, act

obs_A, act_A = make_task(torch.tensor([0.5, -0.5]))    # pretraining task
obs_B, act_B = make_task(torch.tensor([-0.5, 0.5]))    # new task

class Policy(nn.Module):
    def __init__(self, d=64):
        super().__init__()
        self.backbone = nn.Sequential(nn.$Linear(D_OBS$, d), nn.ReLU(),
                                      nn.Linear(d, d), nn.ReLU())
        self.head = nn.Linear(d, $D_ACT$)                # output space
    def forward(self, x):
        return self.head(self.backbone(x))

def train(model, obs, act, replay=None, rho=0.0, steps=1500, lr=1e-3):
    opt = torch.optim.Adam([p for p in model.parameters() if p.requires_grad], lr=lr)
    for _ in range(steps):
        idx = torch.randint(0, obs.shape[0], (32,))
        t  = torch.randint(0, EP_LEN, (1,)).item()
        xb, yb = obs[idx, t], act[idx, t]
        loss = ((model(xb) - yb) ** 2).mean()          # robot loss
        if replay is not None and rho > 0:             # Eq. (B.2) replay
            ridx = torch.randint(0, replay[0].shape[0], (32,))
            rt = torch.randint(0, EP_LEN, (1,)).item()
            rx, ry = replay[0][ridx, rt], replay[1][ridx, rt]
            loss = (1 - rho) * loss + rho * ((model(rx) - ry) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()
    return model

def success(model, obs, act, tol=0.15):                 # fraction within tol
    with torch.no_grad():
        pred = model(obs.reshape(-1, $D_OBS$))
        err = (pred - act.reshape(-1, $D_ACT$)).norm(dim=1)
        return (err < tol).float().mean().item()

base = Policy(); train(base, obs_A, act_A, steps=3000)
print("baseline A:", success(base, obs_A, act_A))

# Control 1: full fine-tune without replay
full = Policy(); full.load_state_dict(base.state_dict())
train(full, obs_B, act_B, steps=1500, lr=5e-4)
print("full  A:", success(full, obs_A, act_A), "B:", success(full, obs_B, act_B))

# Control 2: full fine-tune with replay at rho=0.3
rep = Policy(); rep.load_state_dict(base.state_dict())
train(rep, obs_B, act_B, replay=(obs_A, act_A), rho=0.3, steps=1500, lr=5e-4)
print("replay A:", success(rep, obs_A, act_A), "B:", success(rep, obs_B, act_B))

# Control 3: frozen backbone, new head
frz = Policy(); frz.load_state_dict(base.state_dict())
for p in frz.backbone.parameters(): p.requires_grad = False
frz.head = nn.Linear(64, $D_ACT$)                         # reinitialized head
train(frz, obs_B, act_B, steps=1500, lr=1e-3)
print("frozen A:", success(frz, obs_A, act_A), "B:", success(frz, obs_B, act_B))

def action_scale_check(freq_train, freq_deploy):        # assertion from §L.C.2
    assert freq_train == freq_deploy, \
        f"action-scale shift: {freq_train} Hz vs {freq_deploy} Hz"

Line 7 builds the synthetic task map; line 14 defines the policy with a separate head, so the head can be reinitialized in control 3. Line 22 trains with optional replay; line 31 is the composite loss of Eq. (C.1) specialized to two terms, with the replay ratio of Eq. (B.2). Lines 40–58 run the three controls and print the per-task success, which is the forgetting measurement of Eq. (B.1). Line 60 is the action-scale assertion; it is one line and it prevents the silent failure of break-it 1.

Cheat sheet

Exercises with solutions

E 3.1.4.1. Write the seven phases and state each phase’s exit condition in one sentence.

S 3.1.4.1. Audit: three-axis baseline measured. Strategy: adaptation choice justified by the audit. Safety: force, shield, and recovery pass. Shadow: candidate runs and interventions are logged. Fine-tune: composite loss converges under the schedule. Monitor: four quantities tracked. Commit or rollback: the decision rule fires. The lesson: each phase exists to produce the next phase’s input, and skipping one removes a necessary condition from the decision.

E 3.1.4.2 (transfer). The audit shows 0.91 seen, 0.48 unseen object, 0.82 phrasing, with 400 episodes per axis. Compute the lower bounds at 95% and state which axes fail a 0.90, 0.50, 0.75 requirement.

S 3.1.4.2. Seen: $0.91 - 1.645\sqrt{0.91\cdot0.09/400} = 0.91 - 0.024 = 0.886$, fails 0.90. Object: $0.48 - 1.645\sqrt{0.48\cdot0.52/400} = 0.48 - 0.041 = 0.439$, fails 0.50. Phrasing: $0.82 - 0.036 = 0.784$, passes 0.75. The lesson: point estimates above a requirement can fail when the confidence interval is considered.

E 3.1.4.3. A full fine-tune takes unseen-object success from 0.55 to 0.44 and seen-task from 0.92 to 0.93. Compute both forgetting values and state the blocker.

S 3.1.4.3. $F_{obj} = 0.11$, $F_{seen} = -0.01$. The blocker is the unseen-object loss, because it is the semantic breadth the pretrained model supplies. The lesson: forgetting is per-axis and the blocking axis is the one to protect.

E 3.1.4.4 (diagnostic). A policy trained at 10 Hz is deployed at 20 Hz and overshoots. Name the pitfall, the formula, and the fix.

S 3.1.4.4. Action-scale shift. The per-step displacement corresponds to $f/f' = 0.5$ of the intended time, so the speed doubles. The fix is to freeze the frequency convention in the checkpoint metadata and assert it at load time. The lesson: the convention is part of the model definition.

E 3.1.4.5. The composite loss has $|\nabla L_{robot}| = 2.0$ and $|\nabla L_{web}| = 0.5$ on a pilot batch, with $\lambda_w = 1$. Compute $\lambda_r$ under Eq. (C.2).

S 3.1.4.5. $\lambda_r = \lambda_w |\nabla L_{web}|/|\nabla L_{robot}| = 0.5/2.0 = 0.25$. The lesson: the robot loss is weighted down because its gradient is already four times larger; sample counts would have gotten this backwards.

E 3.1.4.6 (transfer). A candidate scores 0.94 on 300 test episodes against a 0.90 requirement. Compute the lower bound and decide.

S 3.1.4.6. $\text{SE} = \sqrt{0.94\cdot0.06/300} = 0.0137$, lower bound $0.94 - 0.0225 = 0.9175$, which passes. The lesson: with enough episodes, a four-point margin is comfortably significant; with few episodes it would not be.

E 3.1.4.7 (diagnostic). After fine-tuning, the Q-shield rejects 40% of actions. Name two causes and the discriminating test.

S 3.1.4.7. (1) The threshold was not recalibrated, so the new policy’s value distribution falls below the old threshold; test by recomputing the value distribution on validation rollouts. (2) The policy is genuinely out of distribution and the shield is working; test by inspecting the rejected states for novelty. The lesson: a high rejection rate is either a calibration bug or a real distribution shift, and the tests separate them.

E 3.1.4.8 (transfer). The shadow intervention rate is 3%, the incumbent’s is 4%, and the tolerance is two points. The monitor’s first window shows 7%. Apply the rollback rule and justify the choice of pause rather than immediate rollback.

S 3.1.4.8. 7% is three points above the shadow and one point above the tolerance; the rule pauses and reviews, because the first window is short and a single window at small sample size has wide uncertainty. Immediate rollback is reserved for force violations or sustained divergence. The lesson: rollback thresholds must account for window size.

Exit test

You are ready for Module 3.2 when you can, without notes:

1. Write the seven phases and each phase’s exit condition.
2. Run the three-axis baseline audit and compute lower bounds.
3. Read the audit to choose an adaptation strategy.
4. Apply the adaptation ladder to a stated change type.
5. Compute per-axis forgetting and name the blocking axis.
6. Set a replay ratio from a forgetting budget.
7. Write the composite loss and balance it by gradient norm.
8. Compute an action-scale shift and state the frequency convention.
9. Schedule a head warm-up and explain why adapters stay frozen during it.
10. Name the five pitfalls and their signatures.
11. Compute the offline lower bound and apply the acceptance rule.
12. State the safety gate’s three checks.
13. Set the Q-shield threshold by recalibration.
14. Write the rollback triggers and the monitor’s window rule.
15. Apply the decision conjunction to a result set.

Cross-references. 3.1.1 supplies the interface and the breadth–precision distinction restated in §L.A.1. 3.1.2 supplies the four-block anatomy, the adaptation ladder, and the gradient-norm rule restated in §L.B.1 and §L.C.1. 3.1.3 supplies the schema, the quality gate, and the episode-level split restated in §L.C.2 and §L.D.1. 2.2.4 supplies the seven-phase loop and the Q-shield restated in §L.A.1 and §L.D.3. 2.3.3 supplies the force limits used in the safety gate. Module 3.2 uses the fine-tuned policy as the actor in world-model planning. Module 4.2 uses this protocol as the policy-layer integration step.

Ready. The next lesson to write is 3.2.1 — Latent World Models: the RSSM and Dreamer. Say “next” to generate it.