X2.2.4

Lesson 2.2.4 — Offline-to-Online Fine-Tuning: The Deployment Pipeline and the Trough

Week 12 · Prerequisites: 1.2.4, 1.2.5, 2.2.1, 2.2.2, 2.2.3, 1.1.4.

Prior results used. This lesson uses 1.2.4’s PPO and trust-region motivation, 1.2.5’s SAC and replay buffer, 2.2.1’s TD3 and twin critics, 2.2.2’s MBPO and model-based trust region, 2.2.3’s offline RL support constraints (BCQ, CQL, IQL, TD3+BC), and 1.1.4’s health checks and error-recovery pipeline. Each is restated at the point of use.

Hands forward. This lesson hands forward one capability: the learner can design and implement a complete offline-to-online fine-tuning pipeline — pretrain on the offline dataset, verify the pretrained policy’s safety, deploy with a runtime safety layer, collect online data, fine-tune with the calibrated Q-function, monitor the return and the safety constraint, and either commit or roll back — and diagnose the four fine-tuning bugs (the trough’s collapse, the Q’s miscalibration, the safety layer’s disabled, and the evaluation’s contamination) from their deployment’s symptoms — the capability that the VLA’s fine-tuning pipeline in 3.1.4, the sim-to-real’s transfer in 3.4, and the capstone’s real-robot training all assume.

Running case. The same 6-DOF MuJoCo arm from 1.1.1–2.2.3, in the ArmEnv, with:

• an offline dataset of 100,000 transitions collected by a mixed SAC + PID behavior policy (from 2.2.3),
• a pretrained offline policy from 2.2.3 (CQL or TD3+BC, whichever gave the best eval return),
• a real environment to fine-tune on with a 10,000-interaction budget (simulated as the same ArmEnv, but with the budget enforced as if it were hardware),
• a runtime safety layer implemented as a residual Q-shield (from 1.1.4’s safety concept, applied to Q-values instead of action bounds),
• a 20-target held-out evaluation with a safety constraint (the arm must not collide with the table, encoded as an absorbing state with negative reward),
• and the deployment’s decision: commit the fine-tuned policy or roll back to the offline checkpoint.

The lesson’s central result is that the pretrained policy is not a safe starting point — the moment the online SAC updates the actor with the standard max_a Q objective, the Q-function’s miscalibration on the pretrained policy’s state-action distribution pushes the actor away from the offline optimum, through a performance trough — and the fine-tuning algorithms (SAC+BC, CQL+SAC, TD3+BC→SAC) are designed to avoid or shorten the trough. The fine-tune succeeds only if the trough’s depth and length are smaller than the deployment’s safety margin.

§L.0 — Lesson map, objectives, and misconceptions

§L.0.1 — Lesson map and word budget

§L.0.2 — Objectives

By the end of this lesson you will be able to do eight things:

1. State the offline-to-online fine-tuning problem (the offline pretraining on D, the online budget on the real environment, and the deployment’s success criteria), and explain why a naive “continue training SAC from the offline checkpoint” fails through the performance trough.
2. State the trough’s mechanism: the Q-function’s estimates on the pretrained policy’s in-support but slightly off-distribution states are miscalibrated upward (the actor’s max_a Q finds the OOD overestimation), and the first online updates move the actor away from the offline optimum before the new online data has enough density to correct the Q.
3. Implement SAC+BC (SAC augmented with an $α_BC$ behavior-cloning penalty) and state the penalty’s role as a soft support constraint during fine-tuning, extending 2.2.3’s TD3+BC to the stochastic setting.
4. Implement CQL+SAC (pretrain the CQL critic and actor, then switch to online SAC with the CQL-trained critic frozen for the first $N_warmup$ steps, then unfreeze) and state the calibration’s role in shortening the trough.
5. Implement TD3+BC→SAC (pretrain with TD3+BC, then transition the deterministic actor to a stochastic SAC actor via the policy’s standard deviation’s initialization from the Q’s gradient’s scale) and state the transition’s mechanics.
6. Design the deployment pipeline (the pretrain phase, the safety check, the deployment phase, the collection phase, the fine-tune phase, the monitor phase, the commit-or-rollback decision) with the success criteria at each phase, and state the rollback’s trigger.
7. Implement the runtime safety layer (residual Q-shield: $a_safe$ = a_π + α · $a_shield$ where $a_shield$ = argmax_{a ∈ B(a_π, ε)} $Q_safe(s$, a), or the simpler action projection to a Q-threshold’s sublevel set, or the guard condition that falls back to the offline policy when the Q’s value is below a threshold).
8. Diagnose the four fine-tuning bugs — trough collapse (the fine-tuned return is worse than the offline return), Q’s miscalibration (the trough persists, the return does not recover), safety layer disabled (the fine-tuned policy violates the safety constraint), and evaluation contamination (the held-out targets leaked into the online data) — from their training and deployment symptoms.

§L.0.3 — Misconception inventory

Each misconception is displaced in the section named, and each is tagged in the concept check that follows.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

This is the twenty-ninth lesson. The warm-up draws from 2.2.2 (two steps back), 2.1.4 (three), 2.1.3 (four), and 2.1.2 (five).

Warm-up 1 (from 2.2.2, MBPO’s trust region). State MBPO’s real-data trust region and its role. Answer: the policy is constrained to stay close to the real-data distribution; the model’s imagined data provides efficiency, but the real data anchors the policy so the imagined-real gap does not compound. How this is used: §L.2’s SAC+BC fine-tuning’s BC penalty is the same trust-region idea in the offline-to-online transition: the BC penalty anchors the actor to the offline data, and the online data provides the improvement signal. The two are the trust-region’s two sides.

Warm-up 2 (from 2.1.4, ICP’s convergence basin). State the ICP’s local-convergence basin’s scale. Answer: roughly 30° rotation and 10% of the object’s size in translation from the initial pose. How this is used: §L.1’s trough is the fine-tuning’s basin problem: the pretrained actor is the initial pose, the online updates are the ICP’s iterations, and the basin is the region where the Q’s calibration is accurate. If the first update moves the actor outside the basin, the trough’s divergence is analogous to the ICP’s local-minimum failure — the actor lands on the wrong side of the loss surface.

Warm-up 3 (from 2.1.3, IBVS’s gain). State the visual servo’s control law and the gain’s effect. Answer: $v_c$ = -λ L⁺ e; the gain λ sets the convergence rate, large is fast but unstable, small is slow. How this is used: §L.2’s SAC+BC penalty coefficient $α_BC$ is the analogous knob: large $α_BC$ slows the fine-tuning (stays close to offline), small $α_BC$ speeds it (risk of trough). The analogy is quantitative; the trough’s depth depends on $α_BC$ the way the visual servo’s overshoot depends on λ.

Warm-up 4 (from 2.1.2, PnP’s calibration). State the PnP’s dependence on the 2D-3D correspondences’ accuracy. Answer: the PnP’s pose error is O(δu) in the correspondences, and the pose’s position error is δt ≈ δx · Z / f at depth Z. How this is used: §L.1’s Q’s miscalibration is the same amplification structure: a small error in the Q’s training target (the Bellman residual) amplifies through the actor’s max, and the fine-tuned policy’s error is O(trough_depth). The trough is the Q’s calibration’s amplification.

Part A — The Distributional Shift in Fine-Tuning and the Trough

§L.1 — Objects: the pretrained policy, the online data, and the trough

The fine-tuning pipeline takes:

• A pretrained policy $π_off$ from offline RL (the 2.2.3 pipeline: CQL, IQL, TD3+BC, or BCQ) trained on the offline dataset $D_off$.
• A real environment E with a limited interaction budget (10,000 steps in the running case).
• A Q-function Q_φ from the offline training, miscalibrated (its values on the pretrained policy’s distribution are correct up to O(1/√$|D_off|$), but its values on other in-support but slightly off-distribution states are systematically biased upward by the offline regularizer’s tendency to under-penalize the training distribution’s neighbors).
• A target of: fine-tune to exceed $π_off$’s return by at least δ on the held-out evaluation, while preserving the safety constraint (the collision’s absorbing state).

The distributional shift is the difference between $ρ_off$ (the offline state-action distribution) and $ρ_online$ (the online state-action distribution). At the start of the fine-tuning, $ρ_online$ ≈ $ρ_off$ (the pretrained policy generates the online data). After the first actor update, $ρ_online$ shifts toward the actor’s new policy’s distribution, and the Q’s calibration is only accurate within a ball of $ρ_off$ (the calibration’s basin is O(1/√$|D_off|$) in the Q’s error metric, translating to a ball of O(δQ) in the actor’s policy space).

The trough is the phenomenon: after the first actor update, the fine-tuned return drops below the pretrained return, and it takes $N_warmup$ further updates for the online data’s density to increase enough for the Q’s calibration to catch up. The depth of the trough is the maximum performance loss during the fine-tuning; the length is the number of online steps until the return recovers above the pretrained baseline.

The trough’s mechanism, step by step:

1. Pretrained actor’s Q: for states in $ρ_off$, Q_φ(s, a_π(s)) is the offline estimate. It is lower than Q*(s, a_π(s)) because the offline algorithm’s pessimism under the support constraint (the CQL penalty or the IQL expectile) pushed it down.
2. First online update’s Q: the actor’s max_a Q_φ(s, a) uses Q_φ on the whole action space. On actions slightly outside the offline support but still in the Q-network’s training region (a ball around $ρ_off$ where the network’s output was fit by the network’s smoothness), the Q’s estimate is higher than the offline action’s Q. The actor moves toward these actions.
3. The actor’s movement: the actor’s policy shifts to prefer the higher-Q actions, which are outside the offline support. The actor’s $π_new$ has Q_φ(s, $π_new(s$)) > Q_φ(s, $π_off(s$)) — the actor believes it has improved.
4. The real return: the real environment’s return under $π_new$ is worse than under $π_off$, because the Q’s estimate of the new actions was wrong. The real evaluation’s return drops.
5. The recovery: the online data collected by $π_new$ fills in the Q’s calibration in the new region. The Q’s estimate on $π_new$’s actions converges to the true Q*(s, $π_new(s$)), and the actor’s next update uses the calibrated Q. The trough bottoms out and recovers.
6. The trough’s depth and length: depend on the Q’s miscalibration magnitude (the depth) and the online data’s collection rate (the length). The deeper and longer the trough, the more likely it exceeds the deployment’s safety margin.

§L.1.1 — Result: the trough is unavoidable under naive fine-tuning, and the fine-tuning algorithms are the trough’s mitigations

Claim. Fine-tuning with a standard SAC actor from an offline pretrained checkpoint produces a performance trough because the offline Q’s calibration is not valid on the actor’s new policy’s distribution. The trough’s depth is O(trough_depth) in the return; the trough’s length is $O(N_warmup$) in the online steps. The three fine-tuning algorithms (SAC+BC, CQL+SAC, TD3+BC→SAC) shorten the trough by:

• SAC+BC constraining the actor to the offline distribution during the trough.
• CQL+SAC calibrating the Q on the offline data before switching to online SAC, so the trough’s depth is smaller.
• TD3+BC→SAC bridging the deterministic and stochastic actors, so the transition’s trough is smoother.

Proof sketch. The offline Q’s error on the actor’s new policy’s actions is O(1/√$|D_off|$) for actions in support and unbounded for actions outside support. The first update’s actor moves toward the argmax_a Q_φ, which is the most overestimated action; the trough’s depth is O(Q_φ(s, argmax) − Q*(s, argmax)), which is the Q’s overestimation on the actor’s new actions. The trough’s length is the number of steps for the online data to cover those actions, O(1/ρ_online(s, argmax)). The three algorithms target either the depth (CQL+SAC’s calibration) or the length (SAC+BC’s BC anchor).

Corollary. The trough cannot be eliminated entirely, because the offline Q’s calibration is fundamentally limited by $|D_off|$ and the online fine-tuning must update the actor. The best the algorithm can do is bound the trough’s depth and shorten its length.

Corollary. The deployment’s safety margin must exceed the trough’s depth. If the safety margin is small, the fine-tuning must use the short-trough algorithm (CQL+SAC), not the deep-trough one (naive SAC).

§L.1.2 — Worked-example ladder

WE-L.1.a — Fully worked: the trough’s depth in a 1D problem

Input. A 1D state, two actions $a_1$ = −0.5, $a_2$ = 0.5 from the offline dataset. The true Q*(s, $a_1$) = 1.0, Q*(s, $a_2$) = 0.8. The offline Q’s estimate: Q_φ(s, $a_1$) = 0.9 (under-pessimistic by 0.1 from the CQL penalty) and Q_φ(s, $a_2$) = 0.85 (over-optimistic by 0.05 from the network’s smoothness on the unsampled neighbor). The pretrained actor $π_off(s$) = $a_1$ (Q’s argmax in the offline estimate: 0.9 > 0.85).

Trace (before fine-tuning).

1. The offline actor’s real return: Q*(s, $a_1$) = 1.0.
2. The offline actor’s Q-estimate: Q_φ(s, $a_1$) = 0.9.

Trace (first online update).

3. The actor’s max_a Q_φ(s, a) picks $a_2$ because Q_φ(s, $a_2$) = 0.85 > Q_φ(s, $a_1$) = 0.9? No — 0.85 < 0.9, so the argmax is still $a_1$.

Hmm, this simple example doesn’t produce a trough because the offline estimate’s ordering matches the true ordering. The trough requires the offline estimate’s ordering to differ from the true ordering at the boundary. Let me redo with a case where they differ.

Redone input. True Q*(s, $a_1$) = 1.0 (offline action), Q*(s, $a_2$) = 0.8 (offline action). But an action $a_3$ = 0.1 (between the two, outside the offline support’s {a_1, a_2} but inside the Q-network’s training region) has Q*(s, $a_3$) = 0.85 (true), but the offline Q’s estimate Q_φ(s, $a_3$) = 1.2 (over-optimistic by 0.35 from the network’s smoothness, since the network interpolates between $a_1$ and $a_2$ with an upward bump).

Trace (before fine-tuning).

1. Pretrained actor $π_off(s$) = $a_1$ (the offline Q’s argmax on {a_1, a_2}: max(0.9, 0.85) = 0.9 → $a_1$).
2. Real return: Q*(s, $a_1$) = 1.0.

Trace (first online update, no support constraint).

3. The actor’s max_a Q_φ(s, a) is at $a_3$ (Q_φ(s, $a_3$) = 1.2 > 0.9).
4. The actor moves to $a_3$.
5. The real return: Q*(s, $a_3$) = 0.85 < 1.0. The trough has begun; the return dropped by 0.15.
6. After collecting $N_warmup$ online transitions, Q_φ(s, $a_3$) is corrected toward 0.85, and the actor’s argmax shifts back toward $a_1$ (or toward a smaller $a_3$’s neighbor with a true value between 0.85 and 1.0).
7. The trough’s depth is 1.0 − 0.85 = 0.15; the length is $N_warmup$.

Output and interpretation. The trough’s depth is the Q’s overestimation on the “gap” action $a_3$ (which is not in the dataset but is within the Q-network’s interpolation range). The naive actor’s first update moves to $a_3$, the real return drops, and the recovery requires the Q’s calibration on $a_3$ to correct.

Self-explanation 1. Why is $a_3$ = 0.1 “outside the dataset but inside the Q-network’s training region”? Isn’t “the Q’s training region” exactly the dataset’s support?

Answer. Not exactly. The Q-network is a continuous function; its fit on {a_1, a_2} extends smoothly to nearby values like $a_3$. The dataset’s support is the discrete set of actions the data actually contains; the Q-network’s training region is a ball around that set where the network’s fit is well-conditioned by the loss. The gap between the two is the overestimation region: actions that were not sampled (so Q* is unknown there) but that the network fills in by interpolation (so Q_φ is defined and often biased). The trough arises in this gap.

Self-explanation 2. If the offline algorithm is CQL (which pushes the Q down outside the support), would the trough be smaller?

Answer. Yes, in principle: CQL penalizes Q_φ in the gap, so Q_φ(s, $a_3$) is pushed down. The trough’s depth shrinks accordingly. If CQL is well-tuned, Q_φ(s, $a_3$) < Q_φ(s, $a_1$), and the actor stays at $a_1$ on the first update, so the trough’s depth is zero. In practice, CQL’s penalty is not perfect on every gap, so the trough is smaller but nonzero. The CQL+SAC fine-tuning’s benefit is exactly this: it inherits the CQL-calibrated Q at the start of fine-tuning, so the trough is shortened.

WE-L.1.b — Completion problem

Input. Same setup, but the offline algorithm is IQL instead of CQL. IQL’s Q-estimate: Q_φ(s, $a_3$) = 0.95 (less over-optimistic than the naive Q’s 1.2, because IQL’s Q is fit to V_ψ(s') which is a conservative in-sample target). Q_φ(s, $a_1$) = 0.95.

Trace (final 25% as prompts).

1. The pretrained actor: argmax_a Q_φ(s, a) → $a_1$ (tie with $a_3$, tie broken by initialization).
2. The first online update: Q_φ(s, $a_3$) = 0.95 = Q_φ(s, $a_1$) → the argmax is tied.
3. The actor’s update: it doesn’t move, because the Q’s gradient at $a_1$ is small (Q_φ(s, $a_1$) − Q_φ(s, $a_3$) = 0).
4. The real return: Q*(s, $a_1$) = 1.0, unchanged.

Answers.

1–4. Correct.

The lesson. IQL’s conservative Q (the in-sample expectile) reduces the over-optimism on the gap actions, which shrinks the trough. IQL→SAC fine-tuning often has a shallower trough than naive-Q→SAC fine-tuning.

WE-L.1.c — Faded: the trough’s depth vs. the online budget

Setup. Trough’s depth 0.15, $N_warmup$ = 500 online steps to recover, budget 10,000 steps.

Question. What is the fraction of the budget spent in the trough? What if the budget were 1,000 steps?

You must produce the intermediate steps. The setup and the answer are given.

Answer. 500/10,000 = 5% — the trough is short relative to the budget. With a 1,000-step budget, the trough is 500/1,000 = 50% — the trough dominates the budget, and the fine-tuned return may be worse than the offline baseline because the recovery only just begins. The trough’s length matters most when the budget is small.

The lesson. The trough’s length is a relative problem: it matters proportionally to the online budget. A 500-step trough that is 5% of a 10,000-step budget is fine; a 500-step trough that is 50% of a 1,000-step budget is a failure.

WE-L.1.d — Contrastive near-miss: the “high reward, no trough” claim

Correct solution. Report the trough’s depth and length, not just the final return.

Near-miss. A paper’s abstract claims “our fine-tuning achieves state-of-the-art return” but the training curve shows a 30% drop for the first 500 steps before recovering. The abstract is technically true (the final return is the state of the art) but practically misleading (a deployment that uses the policy during the trough would have a 30% performance drop).

Why the near-miss produces a plausible but incorrect number. The final return is high, the state of the art is achieved, but the deployment’s reality is the trough: the first $N_warmup$ steps are the ones where the policy is worse than the pretrained baseline, and those are the steps where the deployment is at risk. A policy that is 30% worse for 5% of its life is not the same as a policy that is 30% better throughout.

The discrepancy. The asymptotic return and the trajectory return are different quantities. A fine-tuning pipeline that uses the policy during the trough is exposed to the trough’s depth. The right metric is the integrated return: ∫_0^T r(t) dt / T, not r(T).

Self-explanation 3. If the deployment uses the policy only after the fine-tuning is complete (not during the fine-tuning), is the trough’s depth still a problem?

Answer. Yes, in two ways. First, the deployment’s policy is the one after fine-tuning, and it may have been trained on the trough’s poor data (the online data from the trough’s steps is in the buffer, and the buffer’s contribution to the fine-tuned Q is negative). Second, the safety check must pass throughout the fine-tuning, not just at the end; the trough’s steps are where the safety constraint is most likely to be violated. The trough is a deployment-time concern even if the post-fine-tuning policy is the deployed one.

Part B — The Three Fine-Tuning Algorithms

§L.2 — Objects: SAC+BC, CQL+SAC, and TD3+BC→SAC

SAC+BC (Fujimoto and Gu 2021, tier B, S) extends the offline TD3+BC of 2.2.3 to the online fine-tuning setting:

• The actor’s loss adds a BC penalty $α_BC$ · $E_{(s, a) ~ D}$[ (π_θ(s) − a)² ] to the SAC loss, so the actor is soft-constrained to the offline data’s actions.
• The BC penalty is annealed over the online steps: at step t, $α_BC(t$) = $α_BC^0$ · (1 − t/T). The start is heavily BC-constrained (the trough’s depth is bounded by the BC anchor); the end is pure SAC (the trough’s length is short because the BC is gone).
• The Q’s calibration is handled by the Q’s online updates: as the online data grows, the Q’s estimate on the actor’s actions improves.

CQL+SAC (Kumar et al. 2020, tier A, S; the fine-tuning extension is in Fujimoto et al. 2023, tier B, S):

• Phase 1 (pretrain): offline CQL on $D_off$. The Q is calibrated conservatively by the CQL regularizer, so Q_φ is a lower bound on Q* outside the support.
• Phase 2 (warm-up): the first $N_warmup$ online steps keep the Q’s CQL regularizer active during the online updates. The Q’s conservatism persists in the first phase, so the actor’s first updates are bounded by the calibrated Q.
• Phase 3 (fine-tune): after $N_warmup$, the CQL regularizer is removed and the Q is trained with standard SAC. The Q’s online data is dense enough to calibrate on its own.

TD3+BC→SAC (a practical hybrid, tier B, S):

• Phase 1 (pretrain): offline TD3+BC on $D_off$. The actor is deterministic.
• Phase 2 (transition): convert the deterministic actor μ_θ(s) to a stochastic SAC actor π_θ(a | s) = N(μ_θ(s), σ_θ(s)) by initializing σ_θ(s) to a small value (the actor’s Q-gradient’s scale) and letting SAC’s entropy term grow the std over training.
• Phase 3 (fine-tune): SAC on the online data, with the initial σ_θ small enough that the early policy’s variance is less than the trough’s safety margin.

The three algorithms’ trade-offs:

§L.2.1 — Result: the three fine-tuning algorithms shorten the trough by different mechanisms; the choice is the trough’s expected depth and the deployment’s safety margin

Claim. SAC+BC’s annealing bounds the trough’s depth and shortens its length; CQL+SAC’s warm-up calibrates the Q before unfreezing the actor; TD3+BC→SAC’s σ initialization bounds the early variance. The choice is task-dependent: broad applicability → SAC+BC; deep trough → CQL+SAC; simplicity → TD3+BC→SAC.

Proof sketch. The SAC+BC actor’s gradient is −∇Q + $α_BC$ · 2(π_θ − a). The BC term is the anchor; the Q term is the improvement. Annealing $α_BC$ from $α_BC^0$ > 0 to 0 transitions from the anchor to the improvement; the trough’s depth is bounded by the anchor’s strength at the start. CQL+SAC’s warm-up keeps the Q conservative for the first $N_warmup$ steps; the actor’s first updates cannot exceed the conservative Q’s estimate, so the trough is shallow. TD3+BC→SAC’s small σ at start limits the policy’s variance, so the sampled actions are close to the mean, which is the pretrained deterministic actor; the trough is shallow.

Corollary. The three algorithms do not eliminate the trough; they shorten it. A well-designed fine-tuning pipeline still reports the trough’s depth and length alongside the final return.

Corollary. The deployment’s safety margin must exceed the trough’s depth. If the safety margin is 5% and the trough’s depth is 15%, none of the fine-tuning algorithms help; the deployment must use a safety layer (Part D) to bound the runtime behavior.

§L.2.2 — Worked-example ladder

WE-L.2.a — Fully worked: SAC+BC’s annealing

Input. $α_BC^0$ = 1.0, T = 5,000 online steps, π_θ(s) = 0.3 (the actor’s output), a = 0.5 (the offline action), Q term’s gradient −0.8, BC term’s gradient 2(π − a) = 2(−0.2) = −0.4.

Trace (at step 0).

1. $α_BC(0$) = 1.0.
2. The BC gradient’s contribution: 1.0 · (−0.4) = −0.4.
3. The total gradient: −0.8 + (−0.4) = −1.2 (strongly toward the offline action’s direction).
4. The actor’s update: moves π toward a, the BC-anchored policy.

Trace (at step 2,500).

5. $α_BC(2$,500) = 1.0 · (1 − 2500/5000) = 0.5.
6. The BC gradient’s contribution: 0.5 · (−0.4) = −0.2.
7. The total gradient: −0.8 + (−0.2) = −1.0 (still BC-anchored, but less).

Trace (at step 5,000).

8. $α_BC(5$,000) = 0.
9. The BC gradient’s contribution: 0.
10. The total gradient: −0.8 (pure SAC).

Output and interpretation. The BC anchor is strong at the start (the trough’s depth is bounded) and fades over training (the improvement takes over). The actor’s trajectory is smooth from BC-anchored to SAC-tuned.

Self-explanation 4. Why annealing rather than a fixed $α_BC$?

Answer. A fixed large $α_BC$ keeps the actor at BC forever, so no improvement is possible (BC’s ceiling is the offline policy). A fixed small $α_BC$ allows the trough at the start (the anchor is too weak). The annealing gives the start the anchor and the end the improvement — the trough is bounded and the ceiling is passed.

Self-explanation 5. If the annealing is too fast (e.g., T = 100 steps), what happens?

Answer. The BC anchor disappears before the online data has calibrated the Q, so the trough re-emerges at step 100. The annealing rate must be slower than the Q’s calibration rate ($N_warmup$). The trough’s length is $O(N_warmup$); the annealing’s length must be ≥ $N_warmup$.

WE-L.2.b — Completion problem

Input. Same setup, but $α_BC^0$ = 10.0 (strong BC anchor).

Trace (final 25% as prompts).

1. At step 0, BC gradient’s contribution: 10 · (−0.4) = −4.0.
2. Total gradient: −0.8 + (−4.0) = −4.8.
3. The actor’s movement is heavily BC-dominated; the trough’s depth is small, but the improvement is also small (the actor is essentially BC for the first phase).
4. The fine-tuned final return is close to the offline baseline (small improvement).

Answers.

1. −4.0.
2. −4.8.
3. BC-dominated.
4. Small improvement.

The lesson. A strong $α_BC^0$ shrinks the trough’s depth at the cost of the ceiling. The fine-tuned return may be close to the offline baseline because the actor never leaves BC’s neighborhood. The $α_BC^0$’s choice is the trough-vs-ceiling trade-off.

WE-L.2.c — Faded: CQL+SAC’s warm-up

Setup. $N_warmup$ = 1,000 steps. The Q is trained with the CQL regularizer for the first 1,000 steps, then with standard SAC.

Question. What is the trough’s depth in the first 1,000 steps? What is it after?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The first 1,000 steps: the Q’s conservatism bounds the actor’s movement, so the trough’s depth is small (the actor is close to the offline policy). After 1,000 steps: the Q is calibrated on the online data, so the trough’s depth is zero (the actor’s updates are on well-calibrated Q-values). The trough is shallow but long — 1,000 steps of small trough rather than a short deep one.

The lesson. CQL+SAC trades the trough’s depth for its length. If the deployment’s safety margin is ≥ 5% but the trough’s depth must be < 5% throughout, CQL+SAC is the choice. If the deployment can tolerate a short deep trough, SAC+BC is simpler.

WE-L.2.d — Contrastive near-miss: the TD3+BC→SAC transition

Correct solution. Convert the TD3+BC deterministic actor μ_θ(s) to a SAC stochastic actor π_θ(a | s) with σ_θ initialized small (e.g., 0.01) and the entropy term growing it.

Near-miss. Convert by initializing σ_θ to 1.0 (a high-variance policy).

Why the near-miss produces a plausible but incorrect number. The high σ_θ samples actions far from the deterministic actor, so the online buffer fills with high-variance data that is not representative of the pretrained policy. The Q’s calibration is measured on this high-variance data, so the actor’s greedy action is poorly calibrated. The trough’s depth is larger than with a small σ_θ.

The discrepancy. The transition’s σ_θ is the trust region’s radius. Small σ means the online data is close to the pretrained policy (the trough is shallow); large σ means the online data is far (the trough is deep). The small σ is the correct choice; SAC’s entropy term grows it over training.

Self-explanation 6. Why does SAC’s entropy term grow the std over training, not shrink it?

Answer. SAC’s objective includes +α H[π(· | s)], which rewards the policy’s entropy. The entropy’s gradient increases the std. The Q’s gradient decreases it when the Q’s values are higher at lower variance. The balance is the learned σ. In the fine-tuning, the start is σ = 0.01 (the Q’s gradient dominates), and the equilibrium’s σ is higher (the entropy’s gradient eventually wins). The transition is gradient-driven, not imposed.

Part C — The Deployment Pipeline

§L.3 — Objects: the seven phases and their gates

The deployment pipeline has seven phases:

1. Pretrain — offline RL on $D_off$ (the 2.2.3 pipeline). Output: $π_off$, $Q_off$, and the FQE estimate $FQE(π_off$).
2. Safety check — evaluate $π_off$ on the held-out evaluation with the safety constraint’s measurement (collision rate, constraint violation rate). Gate: collision_rate < $c_max$ (e.g., 1%) and $FQE(π_off$) > $r_min$ (e.g., 50% of expert). If the gate fails, the offline training is redone (more data, different algorithm); the deployment does not proceed.
3. Deploy and collect — deploy $π_off$ in the real environment and collect the first $N_warmup$ online transitions. The runtime safety layer is active (Part D). Gate (continuous): the collision rate is < $c_max$.
4. Fine-tune — the 2.2.4 Part B algorithm: SAC+BC, CQL+SAC, or TD3+BC→SAC, with the trough bounded by the algorithm’s choice. The online data is collected through the safety layer, so the buffer stores the shield-modified actions, not the raw policy’s.
5. Monitor — the deployment’s return and the safety constraint are tracked every K steps (e.g., every 100). The gates: return(t) > $r_min$ (the return’s floor) and collision_rate(t) < $c_max$ (the safety’s ceiling). The floor prevents the trough from being silently below the pretrained baseline.
6. Commit or roll back — if the monitor’s gates hold at the end of the budget, the fine-tuned policy replaces the pretrained one. If any gate fails, the deployment rolls back to $π_off$ and the fine-tuning’s data is discarded (the buffer’s contribution to the future training is contaminated by the trough’s data).
7. Post-deployment — the fine-tuned policy’s behavior is tracked in production; any new safety incident triggers an investigation and possibly a new pretraining cycle.

§L.3.1 — Result: the pipeline’s gates are the deployment’s safety, and the rollback is the failure’s recovery; the phases’ order is non-negotiable

Claim. The deployment pipeline’s seven phases and their gates are the safety mechanism: the offline policy is checked before deployment, the runtime safety layer is active during collection, the fine-tuning is monitored against the trough, and the rollback is the fail-safe.

Proof sketch. The safety check’s gate is the offline policy’s quality; the collection’s gate is the runtime’s safety; the monitor’s gates are the trough’s bounds. Each phase’s gate is a necessary condition for the next phase. Skipping the safety check allows a poor offline policy to deploy; skipping the monitor allows the trough to go undetected; skipping the rollback allows a failed fine-tuning to remain in production.

Corollary. The pipeline’s cost is the phases’ overhead, not just the training. The gates’ evaluation, the runtime safety layer, the monitor’s data logging, and the rollback’s infrastructure are all part of the deployment’s cost. The cost is paid even if the fine-tuning succeeds, and the absence of the cost is the risk.

Corollary. The rollback is the pipeline’s last resort; a well-designed pipeline’s gates catch the failure before the rollback is needed. The rollback’s frequency is the pipeline’s failure rate; the target is the failure rate’s minimization, not the rollback’s elimination.

§L.3.2 — Worked-example ladder

WE-L.3.a — Fully worked: the pipeline’s phases for the running case

Input. The running case: $π_off$ from CQL (the 2.2.3 pipeline), the held-out 20 targets, the collision rate’s ceiling $c_max$ = 1%, the return’s floor $r_min$ = −0.5, the online budget 10,000 steps, the fine-tuning algorithm SAC+BC with $α_BC^0$ = 1.0, T = 5,000.

Trace (phase 1).

1. Pretrain CQL on $D_off$. Result: $π_off$, $Q_off$, $FQE(π_off$) = −0.35 (the FQE’s normalized return).

Trace (phase 2).

2. Evaluate $π_off$ on the held-out targets. $FQE(π_off$) = −0.35 > −0.5 (the gate passes). The collision rate 0% (the gate passes). Proceed to deployment.

Trace (phase 3).

3. Deploy $π_off$ for $N_warmup$ = 1,000 steps. The safety layer’s collision rate 0.1% (the gate passes). The buffer has 1,000 online transitions.

Trace (phase 4).

4. SAC+BC fine-tune for the remaining 9,000 steps. The trough’s depth (measured by the training return) 12% at step 1,500, and the recovery at step 2,500. The trough is 15% deep and 1,500 steps long, which is 15% of the budget.

Trace (phase 5).

5. Monitor at every 500 steps: the return’s trajectory shows the trough and the recovery; the collision rate stays < 1% (the safety gate holds).

Trace (phase 6).

6. The final return $FQE(π_fine$) = −0.20, which is +0.15 over the offline baseline. The safety gate holds. Commit the fine-tuned policy.

Output and interpretation. The pipeline’s phases execute in order, and each gate passes. The trough is bounded by SAC+BC’s annealing. The monitor catches the trough (the return drops, then recovers), and the rollback’s trigger (return < −0.5 and collision > 1%) does not fire.

Self-explanation 7. In phase 6, the return is −0.20. Is that the deployment’s return, or the FQE’s estimate?

Answer. It is the FQE’s estimate on the held-out targets. The deployment’s actual return is measured on the real environment’s held-out evaluation. The FQE’s estimate is a proxy; the deployment’s actual return is the metric. If the FQE’s estimate is biased (the 2.2.3 evaluation’s coverage issue), the commit’s decision uses the FQE but the deployment’s monitoring uses the actual return, and any discrepancy triggers an investigation.

Self-explanation 8. What if the trough’s depth exceeds the safety margin’s $c_max$ (the collision rate’s ceiling)?

Answer. The pipeline rolls back: the fine-tuned policy is discarded, $π_off$ remains in production, and the fine-tuning’s buffer is discarded. The offline training is redone with a more conservative algorithm (CQL+SAC instead of SAC+BC), and the deployment is retried. The rollback’s trigger is the gate’s failure; the recovery is the offline training’s redo.

WE-L.3.b — Completion problem

Input. Same as WE-L.3.a, but the fine-tuning algorithm is naive SAC (no support constraint).

Trace (final 25% as prompts).

1. The trough’s depth: 30% (larger than the SAC+BC’s 15%).
2. The trough’s length: 2,500 steps (longer than SAC+BC’s 1,500).
3. The collision rate during the trough: 2.5% (exceeds the 1% gate).
4. The monitor’s gate fails; the rollback is triggered.

Answers.

1–4. Correct.

The lesson. The naive SAC’s trough is deep and long, and the monitor’s safety gate fails. The naive algorithm fails the deployment. The gate catches it, and the rollback protects the deployment.

WE-L.3.c — Faded: the rollback’s cost

Setup. The rollback’s frequency 10% (1 in 10 fine-tunings fails), the offline pretraining’s cost $1,000, the online fine-tuning’s cost $500, the deployment’s cost of a failed fine-tuning (in production) $10,000 (the safety incident’s cost), and the rollback’s infrastructure’s cost $100 per rollback.

Question. What is the total expected cost per fine-tuning with the pipeline vs. without?

You must produce the intermediate steps. The setup and the answer are given.

Answer. With the pipeline: 10% fail rate → $1,000 + $500 + 0.1 · ($10,000 + $100 + $1,000 redo + $500 redo) ≈ $1,500 + 0.1 · $11,600 ≈ $1,500 + $1,160 = $2,660. Without the pipeline (the failed fine-tunings are deployed): $1,000 + $500 + 0.1 · $10,000 = $1,500 + $1,000 = $2,500, but the safety incident’s $10,000 is paid **in production**, which is often the more damaging quantity (regulatory, brand, human safety). The pipeline's cost is `+$160in expectation; the pipeline's **production** cost is$0` for the failed cases; the pipeline's **non-production** cost is the `$1,160` retry.

The lesson. The pipeline’s expected cost is slightly higher than the naive cost, but the production’s risk is eliminated. The pipeline’s value is the production risk’s elimination, not the expected cost’s reduction. The pipeline is the right choice when the production’s risk is large (safety-critical deployment); the naive approach is the choice when the production’s risk is small (simulated deployment, no human safety).

WE-L.3.d — Contrastive near-miss: the monitor’s absence

Correct solution. The monitor tracks the return and the safety at every K steps. The gates trigger rollback.

Near-miss. The fine-tuning runs to completion, and the final return is compared to the baseline only at the end.

Why the near-miss produces a plausible but incorrect number. The fine-tuning’s return may be better at the end (the trough is over) but the safety violation during the trough may be silent — the collision rate’s count may not be aggregated without the monitor. The final evaluation sees the post-trough policy, not the trough’s actual behavior in production. The safety violation is not caught.

The discrepancy. The monitor’s continuous check catches the violation at the time it happens; the final check catches the violation’s consequence. The two are different; the monitor’s continuous check is the deployment’s safety, and the final check is the training’s quality.

Self-explanation 9. Is the monitor’s continuous check feasible in a real deployment?

Answer. Yes, for the return (the FQE’s continuous estimate is fast) and the safety (the collision’s detection is a sensor reading). The monitor’s infrastructure is the deployment’s cost, and the monitor’s absence is the pipeline’s blind spot.

Part D — The Runtime Safety Layer

§L.4 — Objects: the residual Q-shield, the action projection, and the guard condition

The runtime safety layer has three variants:

1. Residual Q-shield: the policy’s action a_π is modified by a small residual a_shield chosen to increase the Q’s value (or, equivalently, to stay on the Q’s high-value manifold):

a_safe = a_π + α · ( argmax_{‖a‖ ≤ ε} Q_safe(s, a_π + a) − a_π )

The Q_safe is a separately trained Q-function that estimates the return plus the safety constraint’s value (the negative collision’s penalty). The shield’s residual is small (ε = 0.1, α = 0.5), so the policy’s intent is preserved but the shield’s estimate of the safe action’s Q is preferred.

2. Action projection: the policy’s action is projected onto the set of actions whose Q-value is above a threshold:

a_safe = argmin_{a ∈ B(a_π, ε)} ‖a − a_π‖ s.t. Q_safe(s, a) ≥ Q_min

The Q_min is the safety constraint’s threshold. The projection finds the closest action to a_π that satisfies the constraint.

3. Guard condition: the policy’s action is swapped for the offline policy’s π_off(s) if the Q’s value is below a threshold:

a_safe = a_π if Q_safe(s, a_π) ≥ Q_min else π_off(s)

The guard is a binary switch — the offline policy is the fallback, and the shield’s choice is a hard floor.

The three variants’ trade-offs:

§L.4.1 — Result: the runtime safety layer is the deployment’s only guarantee; the pretrained policy’s safety does not transfer to the fine-tuned one

Claim. The pretrained policy π_off’s safety is a property of π_off’s distribution; the fine-tuned policy π_θ’s safety is a property of π_θ’s distribution. The two are different; the pretrained safety does not transfer. The runtime safety layer is the only guarantee for π_θ’s safety during the trough.

Proof sketch. The safety constraint’s violation rate under π_off is O(ρ_off's safety region). Under π_θ during the trough, the policy’s distribution shifts to the actor’s new region, and the violation rate is O(ρ_trough's safety region). The two distributions differ by the trough’s shift; if the shift is outside the safe region, the violation rate is higher. The safety layer modifies the policy’s actions to stay inside the safe region, regardless of the policy’s distribution.

Corollary. The safety layer’s cost is the modification’s magnitude; the small residual (0.1) or the small projection (0.1) preserves the policy’s intent; the binary guard’s swap is a larger cost but a stronger guarantee.

Corollary. The safety layer’s data is what the fine-tuning learns from. If the layer is active during the fine-tuning, the buffer stores a_safe, and the policy learns from the shielded behavior, not its own. The layer changes the fine-tuning’s data distribution, and the algorithm’s trough is measured on the shielded policy, not the raw one.

§L.4.2 — Worked-example ladder

WE-L.4.a — Fully worked: the residual Q-shield for a specific action

Input. A state s, the policy’s action a_π = 0.3, the Q_safe’s gradient ∇_a Q_safe = [−2.0, 0.5] (the Q decreases in a_1 direction, increases in a_2), ε = 0.1, α = 0.5.

Trace.

1. The best residual within ‖a‖ ≤ ε = 0.1: a_shield = argmax Q_safe(s, a_π + a) ≈ (0.1 · 0.5/√(4.25), 0.1 · (−2.0)/√(4.25)) — the direction of the Q’s gradient, scaled by ε. Actually, the argmax’s direction is the gradient’s direction (positive for a_2, negative for a_1); let me normalize: a_shield ≈ (0.1 · (−2.0/√(4.25)), 0.1 · 0.5/√(4.25)) = (−0.097, 0.024). Wait, the Q decreases in a_1’s direction, so the max is in the −a_1 direction.
2. a_safe = a_π + α · a_shield = 0.3 + 0.5 · (−0.097, 0.024) = (0.3 − 0.0485, 0 + 0.012) = (0.2515, 0.012).

Output and interpretation. The safe action is the policy’s action adjusted by a small residual toward the Q_safe’s higher-value direction. The adjustment’s magnitude is α · ε = 0.05, which is small but nonzero.

Self-explanation 10. Why does the shield not fully move to argmax Q_safe?

Answer. The full argmax’s movement could be large (the Q’s maximum is far from a_π), and the full movement would override the policy’s intent. The α (0.5) and ε (0.1) bound the movement, so the policy’s intent is preserved and the Q’s safety is improved, but the shield does not replace the policy. The trade-off is the intent’s preservation vs. the safety’s improvement.

Self-explanation 11. What if α = 1.0 and ε = 1.0 (the shield dominates)?

Answer. The policy becomes the Q’s argmax, which is the safe action’s argmax but not the policy’s action. The shield is no longer a shield; it is a second policy. The fine-tuning’s buffer’s data is the shield’s data, and the actor learns the shield’s behavior, not its own. This is a safe but non-improving policy — the fine-tuning has failed to improve (the trough is small, but so is the ceiling).

WE-L.4.b — Completion problem

Input. Same setup, but Q_safe(s, a_π) = 0.5 (below the Q_min = 1.0 threshold), and the guard condition is the shield’s variant.

Trace (final 25% as prompts).

1. The Q_safe’s value at a_π is 0.5, below the Q_min = 1.0 threshold.
2. The guard triggers: a_safe = π_off(s), not a_π.
3. The safe action is the offline policy’s action.
4. The buffer stores the offline policy’s transition, not the policy’s.

Answers.

1–4. Correct.

The lesson. The guard condition’s swap is the binary safety: when the policy’s Q is below the threshold, the policy is overridden by the offline policy. The trade-off is the policy’s intent’s complete loss (the swap is binary), but the safety guarantee is strong.

WE-L.4.c — Faded: the safety layer’s data distribution

Setup. The shield is active for the entire fine-tuning, and the buffer stores the shield-modified actions.

Question. What does the policy learn from? What is the trough’s depth?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The policy learns from the shield’s behavior, not its own. The shield’s behavior is a_safe = shield(a_π), which is the policy’s action adjusted by the shield. The trough’s depth is measured on the shielded policy’s real return, which is bounded by the shield’s safety. The policy’s learned behavior is the shield’s behavior, not its own, so the shield is the deployed policy in some sense.

The lesson. The shield’s data distribution is the shield’s behavior’s distribution, not the policy’s. The policy’s learning is anchored by the shield, and the policy’s improvement is bounded by the shield. The two are coupled through the buffer.

WE-L.4.d — Contrastive near-miss: the shield’s disable for training speed

Correct solution. The shield is active during the fine-tuning.

Near-miss. The shield is disabled during the fine-tuning (to “let the policy explore”) and re-enabled at deployment.

Why the near-miss produces a plausible but incorrect number. The fine-tuning’s data comes from the raw policy, which explores the unsafe region during the trough. The buffer’s transitions include the collisions, which bias the Q’s calibration. At deployment, the shield is enabled, but the policy is trained on the unsafe data’s distribution, and its “safe” behavior at deployment is off-distribution from its training data.

The discrepancy. The shield’s disable lets the policy explore the unsafe region; the safety violation’s data is in the buffer, and the shield’s re-enablement at deployment places the policy on a distribution it did not train on. The coupled data distribution is what matters; the shield’s disable breaks the coupling.

Self-explanation 12. If the shield is disabled, is the deployment’s safety guaranteed by the shield?

Answer. Not in the trained policy. The shield’s runtime guarantee holds: any action is shielded, so no collision. But the policy trained on unsafe data has learned the unsafe data’s distribution, and the shield’s projection of its raw action may be far from what the policy was trained on. The policy’s behavior under the shield may be worse than the policy’s behavior without the shield, because the shield’s modification of an off-distribution action is arbitrary.

Part E — Failure taxonomy, lab, cheat sheet, exercises, exit test

§L.5 — Failure taxonomy

§L.6 — The lab

§L.6.1 — Task

Implement the offline-to-online fine-tuning pipeline for the ArmEnv: pretrain with CQL or TD3+BC, deploy with a residual Q-shield, fine-tune with SAC+BC, monitor, and commit or roll back.

§L.6.2 — Conditions

Python 3.10+, PyTorch 2.2+, CPU. Seed 42, dtype = float64. The 2.2.3 pipeline’s CQL output, the held-out 20 targets, the collision rate’s ceiling 1%, the return floor −0.5, the online budget 10,000 steps, the fine-tuning algorithm SAC+BC with α_BC^0 = 1.0, T = 5,000, the shield’s α = 0.5, ε = 0.1, Q_min = 0.0.

§L.6.3 — Controls held fixed

The seed, dtype, ArmEnv, offline data, pretrained policy, held-out targets, hyperparameters. Variants: SAC+BC vs. naive SAC, shield vs. no-shield, monitor vs. no-monitor.

§L.6.4 — Metrics table

§L.6.5 — Pre-registered hypotheses

• H1. Naive SAC’s trough_depth > 20%; SAC+BC’s trough_depth < 15%. Prediction: the two depths.
• H2. Naive SAC’s trough_length > 2,000; SAC+BC’s < 1,500. Prediction: the two lengths.
• H3. Naive SAC’s final_return exceeds the offline baseline by < 10%; SAC+BC’s by > 15%. Prediction: the two ratios.
• H4. The shield’s collision_rate < 1%; without the shield, > 2%. Prediction: the two rates.
• H5. With the monitor, rollback_triggered when trough_depth > safety_margin. Without the monitor, the failure is silent. Prediction: the two behaviors.

§L.6.6 — Break-it sequence

1. Naive SAC without a support constraint (silent). Predicted: trough’s depth > 20%, no monitor’s trigger. Actual: silent.
2. Shield disabled during fine-tuning (silent). Predicted: collision rate > 2% during the trough; the monitor’s gate fails but no rollback. Actual: silent.
3. Monitor disabled (silent). Predicted: trough’s depth unobserved; the deployment is worse than offline for the first 1,500 steps. Actual: silent.
4. Evaluation contamination (silent). Predicted: the fine-tuned policy’s held-out return is inflated by 5% because the held-out targets leaked into the online data. Actual: silent.

All four are silent.

§L.6.7 — Acceptance criteria

§L.6.8 — What good looks like

1. The trough’s depth and length are reported for both naive and SAC+BC.
2. The shield’s effect is measured (collision rate with and without).
3. The monitor’s trigger is demonstrated.
4. The four break-its are predicted with symptoms before running.
5. The rollback’s infrastructure is demonstrated (rollback’s effect on the deployment).

§L.6.9 — Runnable lab code

# lab_2_2_4.py — The offline-to-online fine-tuning pipeline
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import copy

torch.manual_seed(42)
dtype = torch.float64
device = torch.device("cpu")

# --- The environment --------------------------------------------------
class SimpleArmEnv:
    """A 2-D reach task with a no-go disc: the collision the shield prevents."""
    def __init__(self, max_steps=10):
        self.p_target = torch.tensor([0.5, 0.2], dtype=dtype)
        # Off the straight line from the origin to the target: an obstacle placed
        # on that line is unavoidable, and the "expert" would drive through it.
        self.obstacle = torch.tensor([0.22, 0.30], dtype=dtype)
        self.obstacle_r = 0.12
        self.max_steps = max_steps
        self.state_dim = 2; self.action_dim = 2
    def reset(self):
        self.pos = torch.randn(2, dtype=dtype) * 0.1
        self.step_count = 0
        return self.pos.clone()
    def step(self, a):
        self.pos = self.pos + a * 0.1
        dist = torch.norm(self.pos - self.p_target).item()
        in_obstacle = torch.norm(self.pos - self.obstacle).item() < self.obstacle_r
        self.step_count += 1
        r = -dist - (5.0 if in_obstacle else 0.0)
        return self.pos.clone(), r, dist < 0.05, self.step_count >= self.max_steps, in_obstacle

# --- The networks -----------------------------------------------------
class Actor(nn.Module):
    def __init__(self, s_dim, a_dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(s_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, a_dim), nn.Tanh(),
        )
        self.log_std = nn.Parameter(torch.zeros(a_dim, dtype=dtype) - 2.0)
    def forward(self, s):
        mu = self.net(s)
        return torch.distributions.Normal(mu, self.log_std.exp())
    def sample(self, s):
        dist = self.forward(s)
        a = dist.rsample()
        return torch.tanh(a), dist.log_prob(a).sum(-1)

class TwinCritic(nn.Module):
    def __init__(self, s_dim, a_dim, hidden=256):
        super().__init__()
        def mlp():
            return nn.Sequential(nn.Linear(s_dim + a_dim, hidden), nn.ReLU(),
                                 nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, 1))
        self.q1 = mlp(); self.q2 = mlp()
    def forward(self, s, a):
        sa = torch.cat([s, a], dim=-1)
        return self.q1(sa).squeeze(-1), self.q2(sa).squeeze(-1)

# --- The offline pretraining -----------------------------------------
def collect_offline(n_episodes=1500, seed=0, expert_frac=0.4, noise=0.35):
    torch.manual_seed(seed)
    env = SimpleArmEnv()
    S, A, R, S2, TERM = [], [], [], [], []
    for _ in range(n_episodes):
        s = env.reset(); expert = torch.rand(1).item() < expert_frac
        done = trunc = False
        while not (done or trunc):
            if expert:
                a = torch.clamp((env.p_target - s) / 0.1, -1, 1) + torch.randn(2, dtype=dtype) * noise
            else:
                a = (torch.rand(2, dtype=dtype) * 2 - 1) * 0.5
            a = a.clamp(-1, 1)
            s2, r, done, trunc, _ = env.step(a)
            S.append(s); A.append(a); R.append(torch.tensor(r, dtype=dtype))
            S2.append(s2); TERM.append(float(done)); s = s2
    return (torch.stack(S), torch.stack(A), torch.stack(R), torch.stack(S2),
            torch.tensor(TERM, dtype=dtype))

def offline_bc(data, n_steps=4000):
    """The offline baseline: behaviour cloning on the mixed dataset."""
    s, a, r, s2, term = data
    N = len(s)
    actor = Actor(2, 2).to(dtype)
    opt = torch.optim.Adam(actor.parameters(), lr=3e-4)
    for _ in range(n_steps):
        idx = torch.randint(0, N, (256,))
        loss = F.mse_loss(actor(s[idx]).mean, a[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    return actor

# --- The shield and the monitor --------------------------------------
def make_shield(env, step_size=0.1, margin=0.02):
    """Project an action so the next position stays outside the obstacle."""
    def shield(a, s):
        nxt = s + a * step_size
        d = torch.norm(nxt - env.obstacle)
        if d >= env.obstacle_r + margin:
            return a
        # Push the action radially away from the obstacle centre until it clears.
        away = (nxt - env.obstacle) / (d + 1e-9)
        for _ in range(20):
            a = a + 0.05 * away
            a = a.clamp(-1, 1)
            nxt = s + a * step_size
            if torch.norm(nxt - env.obstacle) >= env.obstacle_r + margin:
                return a
        return torch.zeros_like(a)
    return shield

class Monitor:
    """Tracks the online return against the offline baseline and rolls back."""
    def __init__(self, offline_return, safety_margin=0.20):
        self.offline_return = offline_return
        self.safety_margin = safety_margin
        self.best = None; self.best_actor = None
        self.rollback_triggered = False
        self.trough_depth = 0.0
    def update(self, ret, actor, step):
        if self.best is None or ret > self.best:
            self.best = ret
            self.best_actor = copy.deepcopy(actor)
        drop = (self.best - ret) / max(abs(self.best), 1e-9)
        self.trough_depth = max(self.trough_depth, drop)
        if drop > self.safety_margin:
            self.rollback_triggered = True
        return drop

# --- The fine-tuning --------------------------------------------------
def sac_bc_finetune(pi_off, data_off, n_online=6000, alpha_bc0=1.0, anneal_steps=3000,
                    use_shield=True, use_monitor=True, offline_return=-1.0):
    s_off, a_off, r_off, s2_off, term_off = data_off
    N_off = len(s_off)
    env = SimpleArmEnv()
    actor = copy.deepcopy(pi_off)
    critic = TwinCritic(2, 2).to(dtype)
    critic_t = copy.deepcopy(critic)
    actor_opt = torch.optim.Adam(actor.parameters(), lr=3e-4)
    critic_opt = torch.optim.Adam(critic.parameters(), lr=3e-4)
    log_alpha = torch.tensor(np.log(0.2), dtype=dtype, requires_grad=True)
    alpha_opt = torch.optim.Adam([log_alpha], lr=3e-4)
    shield = make_shield(env) if use_shield else None
    monitor = Monitor(offline_return) if use_monitor else None
    returns = []; collisions = 0; steps = 0
    s = env.reset(); ep_ret = 0.0
    for step in range(n_online):
        with torch.no_grad():
            a, _ = actor.sample(s)
            if shield is not None:
                a = shield(a, s)
        s2, r, done, trunc, in_obs = env.step(a)
        collisions += int(in_obs); steps += 1
        ep_ret += r
        # Critic update on a mixed batch: offline data plus the fresh transition
        idx = torch.randint(0, N_off, (128,))
        sb = torch.cat([s_off[idx], s.unsqueeze(0)])
        ab = torch.cat([a_off[idx], a.unsqueeze(0)])
        rb = torch.cat([r_off[idx], torch.tensor([r], dtype=dtype)])
        s2b = torch.cat([s2_off[idx], s2.unsqueeze(0)])
        tb = torch.cat([term_off[idx], torch.tensor([float(done)], dtype=dtype)])
        with torch.no_grad():
            a2, lp2 = actor.sample(s2b)
            q1t, q2t = critic_t(s2b, a2)
            alpha = log_alpha.exp()
            y = rb + 0.99 * (1 - tb) * (torch.min(q1t, q2t) - alpha * lp2)
        q1, q2 = critic(sb, ab)
        c_loss = F.mse_loss(q1, y) + F.mse_loss(q2, y)
        critic_opt.zero_grad(); c_loss.backward(); critic_opt.step()
        # Actor update: SAC objective plus an annealed BC penalty
        a_new, lp = actor.sample(sb)
        q1p, q2p = critic(sb, a_new)
        alpha_bc = alpha_bc0 * max(0.0, 1.0 - step / anneal_steps)
        actor_loss = (alpha.detach() * lp - torch.min(q1p, q2p)).mean() \
                     + alpha_bc * F.mse_loss(a_new, ab)
        actor_opt.zero_grad(); actor_loss.backward(); actor_opt.step()
        alpha_loss = -(log_alpha * (lp + (-2.0)).detach()).mean()
        alpha_opt.zero_grad(); alpha_loss.backward(); alpha_opt.step()
        with torch.no_grad():
            for p_, pt in zip(critic.parameters(), critic_t.parameters()):
                pt.data.mul_(0.995); pt.data.add_(0.005 * p_.data)
        if done or trunc:
            returns.append(ep_ret); ep_ret = 0.0; s = env.reset()
            if monitor is not None and len(returns) % 20 == 0:
                monitor.update(np.mean(returns[-20:]), actor, step)
        else:
            s = s2
    if monitor is not None and monitor.best_actor is not None and monitor.rollback_triggered:
        actor = monitor.best_actor
    return actor, returns, collisions / max(steps, 1), monitor

def evaluate(actor, n_episodes=20, use_shield=True):
    env = SimpleArmEnv(); shield = make_shield(env) if use_shield else None
    rets, coll = [], 0
    for _ in range(n_episodes):
        s = env.reset(); ep = 0.0; done = trunc = False
        while not (done or trunc):
            with torch.no_grad():
                a = actor(s).mean
                if shield is not None:
                    a = shield(a, s)
            s, r, done, trunc, in_obs = env.step(a)
            ep += r; coll += int(in_obs)
        rets.append(ep)
    return np.mean(rets), coll

def trough_stats(returns, offline_return):
    """Depth and length of the worst dip below the offline baseline."""
    r = np.array(returns)
    depth = max(0.0, (offline_return - r.min()) / max(abs(offline_return), 1e-9))
    below = r < offline_return
    length = 0; best = 0
    for b in below:
        length = length + 1 if b else 0
        best = max(best, length)
    return depth, best

# --- Run --------------------------------------------------------------
print("Collecting offline data and training the BC baseline...")
data = collect_offline()
pi_off = offline_bc(data)
offline_ret, _ = evaluate(pi_off)
print(f"offline BC return: {offline_ret:.4f}")

print("\n=== Naive SAC fine-tune (no BC, no shield, no monitor) ===")
pi_naive, ret_naive, coll_naive, _ = sac_bc_finetune(
    pi_off, data, n_online=6000, alpha_bc0=0.0, use_shield=False, use_monitor=False,
    offline_return=offline_ret)
naive_ret, _ = evaluate(pi_naive, use_shield=False)
d_naive, l_naive = trough_stats(ret_naive, offline_ret)
print(f"final return: {naive_ret:.4f}, collision rate: {coll_naive:.4f}")

print("\n=== SAC+BC fine-tune (BC penalty, shield, monitor) ===")
pi_bc, ret_bc, coll_bc, mon = sac_bc_finetune(
    pi_off, data, n_online=6000, alpha_bc0=1.0, use_shield=True, use_monitor=True,
    offline_return=offline_ret)
bc_ret, _ = evaluate(pi_bc)
d_bc, l_bc = trough_stats(ret_bc, offline_ret)
print(f"final return: {bc_ret:.4f}, collision rate: {coll_bc:.4f}, rollback: {mon.rollback_triggered}")

print("\n=== No shield (with BC) ===")
pi_nosh, ret_nosh, coll_nosh, _ = sac_bc_finetune(
    pi_off, data, n_online=6000, alpha_bc0=1.0, use_shield=False, use_monitor=True,
    offline_return=offline_ret)
nosh_ret, _ = evaluate(pi_nosh, use_shield=False)
print(f"final return: {nosh_ret:.4f}, collision rate: {coll_nosh:.4f}")

print("\n--- Summary ---")
print(f"offline baseline: {offline_ret:.4f}")
print(f"H1: naive trough depth {d_naive:.3f} (pred > 0.20); SAC+BC {d_bc:.3f} (pred < 0.15)")
print(f"H2: naive trough length {l_naive} eps (pred > 2000 steps); SAC+BC {l_bc} eps (pred < 1500)")
print(f"H3: naive {100*(naive_ret/offline_ret - 1):+.1f}% vs offline (pred < +10%); "
      f"SAC+BC {100*(bc_ret/offline_ret - 1):+.1f}% (pred > +15%)")
print(f"H4: shield collision {coll_bc:.4f} (pred < 0.01); no shield {coll_nosh:.4f} (pred > 0.02)")
print(f"H5: monitor rollback = {mon.rollback_triggered} (pred: triggers when depth > margin)")

§L.7 — The cheat sheet

§L.8 — Exercises with solutions

E 2.2.4.1 — The trough’s mechanism (conceptual). Why does the first online update worsen the return?

S 2.2.4.1. The offline Q’s max_a Q_φ picks the overestimated action (in the gap between offline data and the Q-network’s training region); the actor moves there; the true Q is lower; the return drops. The lesson: the trough’s depth is the overestimation’s magnitude.

E 2.2.4.2 — SAC+BC’s annealing (numerical). α_BC^0 = 1.0, T = 5,000. What is α_BC at step 2,500?

S 2.2.4.2. 1.0 · (1 − 2500/5000) = 0.5. The lesson: annealing halves the anchor at the midpoint.

E 2.2.4.3 — CQL+SAC’s warm-up (conceptual). Why does the warm-up keep the Q conservative?

S 2.2.4.3. The Q’s CQL regularizer is active during the warm-up, so the Q’s OOD estimates are penalized. The actor’s first updates are on the conservative Q, so the trough is shallow. The lesson: warm-up’s length bounds the trough’s depth.

E 2.2.4.4 — TD3+BC→SAC’s σ (numerical). Initial σ = 0.01. What is the variance?

S 2.2.4.4. 0.0001. The small variance keeps the sampled actions close to the deterministic actor. The lesson: the small σ preserves the pretrained policy.

E 2.2.4.5 — The monitor’s trigger (conceptual). The monitor’s return drops to −0.6, below the floor −0.5. What happens?

S 2.2.4.5. The rollback triggers. The fine-tuned policy is discarded, the offline checkpoint is restored, the buffer’s trough data is discarded, and the offline training is redone. The lesson: the monitor’s floor is the trough’s bound.

E 2.2.4.6 — The shield’s variant (conceptual). Which shield variant best preserves the policy’s intent?

S 2.2.4.6. The residual Q-shield (small residual, small ε). The action projection is next; the guard condition is the least (binary swap). The lesson: the trade-off is the intent vs. the safety.

E 2.2.4.7 — The evaluation’s contamination (diagnostic). The fine-tuned held-out return is 10% higher than the deployed return. Cause?

S 2.2.4.7. The held-out targets leaked into the online data. Fix: a truly held-out set (not the online collection’s targets) and the deployment’s monitor as the check. The lesson: the held-out set’s separation is the evaluation’s validity.

E 2.2.4.8 — The algorithm’s choice (constructive). A safety-critical deployment with a 5% safety margin and a deep trough (> 20% under naive SAC). Which algorithm?

S 2.2.4.8. CQL+SAC (the shallow, long trough fits the 5% margin). SAC+BC’s trough (15%) exceeds the 5% margin. The lesson: the algorithm’s choice is the trough vs. the safety margin.

§L.9 — Exit test

You are ready for Module 2.3 (Grasping and Manipulation) when you can, without notes:

1. State the offline-to-online fine-tuning problem.
2. Explain the trough’s mechanism.
3. State SAC+BC and its annealing.
4. State CQL+SAC and its warm-up.
5. State TD3+BC→SAC and its σ initialization.
6. State the deployment pipeline’s seven phases and gates.
7. State the monitor’s role and the rollback’s trigger.
8. State the residual Q-shield and its three variants.
9. State why the pretrained policy’s safety does not transfer.
10. Diagnose the trough’s depth and length; the shield’s disabled; the monitor’s absence; the evaluation’s contamination.
11. State the algorithm’s choice by trough vs. safety margin.
12. State the shield’s data-distribution coupling.

If any is not answerable, return: 1 to §L.1, 2 to §L.1.1, 3 to §L.2, 4 to §L.2, 5 to §L.2, 6 to §L.3, 7 to §L.3, 8 to §L.4, 9 to §L.4.1, 10 to §L.5, 11 to §L.2.1, 12 to §L.4.1.

End of Lesson 2.2.4. The learner has now completed the offline-to-online fine-tuning pipeline: the trough’s mechanism, the three fine-tuning algorithms (SAC+BC, CQL+SAC, TD3+BC→SAC), the seven-phase deployment pipeline with gates, the runtime safety layer, and the four fine-tuning bugs. The module ends here; Module 2.3 begins with grasping and manipulation, where the fine-tuned policies meet the physical gripper, and the perception stack from Module 2.1 supplies the target’s 6D pose.