X2.2.3

Lesson 2.2.3 — Offline RL: Batch Learning, Extrapolation Error, and the Conservative Turn

Week 12 · Prerequisites: 1.2.2, 1.2.4, 1.2.5, 2.2.1, 2.2.2, 0.3.4.

Prior results used. This lesson uses 1.2.2’s value functions and Bellman optimality, 1.2.4’s trust-region motivation and clipped objective, 1.2.5’s SAC and replay buffer, 2.2.1’s TD3 and twin critics, 2.2.2’s model-based methods and MBPO’s trust region, and 0.3.4’s Mahalanobis gate for distributional support. Each is restated at the point of use.

Hands forward. This lesson hands forward one capability: the learner can formalize the offline RL problem, state the extrapolation-error theorem, implement BCQ, CQL, IQL, and TD3+BC, choose the right algorithm for a given dataset, evaluate offline policies with FQE and the D4RL normalized score, and diagnose the four offline-RL bugs (no support constraint, Q divergence on out-of-distribution actions, dataset coverage too narrow, and evaluation leakage) from their training and evaluation symptoms — the capability that the imitation learning’s Phase 3 pipeline, the VLA’s offline pretraining, and the capstone’s real-robot fine-tuning all assume.

Running case. The same 6-DOF MuJoCo arm from 1.1.1–2.2.2, in the ArmEnv, with a fixed offline dataset of 100,000 transitions collected by an existing controller. The controller is a mix of an SAC policy (from 1.2.5, 50% of the data) and a scripted PID (from 1.1.3, 50%). The policy learns from this dataset only — no environment interaction during training. The evaluation uses a held-out set of 20 target positions, and the comparison is against the same policy trained online with SAC. The lesson’s central result is that a naive offline TD3’s Q-values diverge on actions outside the dataset’s support, and the conservative algorithms (BCQ, CQL, IQL, TD3+BC) constrain the policy’s actions to the dataset’s neighborhood — at the cost of a suboptimality gap relative to the dataset’s best policy.

§L.0 — Lesson map, objectives, and misconceptions

§L.0.1 — Lesson map and word budget

§L.0.2 — Objectives

By the end of this lesson you will be able to do eight things:

1. State the offline RL problem (a fixed dataset D, no further sampling), how it relates to the off-policy setting of 1.2.5, and why the batch and offline terminology are equivalent.
2. State the extrapolation-error theorem (Fujimoto et al. 2019): the Q-function’s error on an action a is O(1/√|D|) near the dataset’s support and unbounded outside it, and the policy’s argmax over an erroneous Q is the source of the offline divergence.
3. Implement BCQ (the perturbation network bounded to [−Φ, Φ], the generative model G_ω, and the Q-filter’s argmax) and state the support constraint’s L_∞ neighborhood.
4. Implement CQL (the conservative penalty log $Σ_a$ exp Q(s, a) − $E_{a ~ D}[Q(s$, a)]) and state the lower bound’s role in preventing Q overestimation on out-of-distribution actions.
5. Implement IQL (the expectile regression for V_ψ, the advantage-weighted actor, and the absence of any out-of-distribution Q query), and state why the expectile approach avoids the max entirely.
6. Implement TD3+BC (the behavior-cloning penalty λ · (μ_θ(s) − a)², with λ normalized by 1/|Q|) and state its simplicity, its strong empirical baseline status, and its failure modes.
7. Evaluate an offline policy with FQE (fitted Q evaluation, not online rollouts) and report the D4RL normalized score (s − $s_random$) / ($s_expert$ − $s_random$).
8. Diagnose the four offline bugs — no support constraint (Q divergence, NaN losses), narrow dataset coverage (the policy cannot improve beyond the data), evaluation leakage (the D4RL benchmark’s train/test overlap), and BC-only collapse (the policy imitates the dataset’s mistakes) — from their training and evaluation symptoms.

§L.0.3 — Misconception inventory

Each misconception is displaced in the section named, and each is tagged in the concept check that follows.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

This is the twenty-eighth lesson. The warm-up draws from 2.2.1 (two steps back), 2.1.4 (three), 2.1.3 (four), and 2.1.2 (five).

Warm-up 1 (from 2.2.1, TD3 and twin critics). State the clipped double-Q target and its role. Answer: y = r + γ min_i $Q_{\barφ_i}(s$', a'), and the min reduces the overestimation bias that a single critic’s max would accumulate. How this is used: §L.2’s CQL adds a further penalty beyond the twin min, because the offline setting has no real data to correct the divergence: the twin min handles in-support estimation error, while CQL handles out-of-support extrapolation.

Warm-up 2 (from 2.1.4, ICP and local minima). State the ICP’s local convergence and the basin’s scale. Answer: ICP converges to the nearest local minimum from the initial pose; the basin is roughly 30° rotation and 10% of the object’s size in translation. How this is used: §L.2’s BCQ perturbation network constrains the candidate action’s magnitude to [−Φ, Φ], an L∞-ball around the generative model’s output; the ball plays the same role as the ICP’s basin, bounding the offline policy’s search to a neighborhood where the Q-function is trustworthy.

Warm-up 3 (from 2.1.3, visual servo gain). State the IBVS’s control law and the gain’s role. Answer: $v_c$ = −λ L⁺ e; a large gain is fast but unstable, a small gain is slow but stable. How this is used: §L.3’s TD3+BC behavior-cloning coefficient λ plays the analog of the servo gain: a large λ keeps the actor close to the data (stable but slow to improve), a small λ allows the actor to deviate (faster but Q-divergence risk).

Warm-up 4 (from 2.1.2, the PnP and correspondences). State the PnP’s minimal and overdetermined cases. Answer: P3P/P4P gives up to four solutions from the minimal correspondences; n > 4 gives a least-squares solution. How this is used: §L.4’s FQE fits a Q-function over the entire dataset, an overdetermined regression in a much higher dimension; the least-squares structure is the same, but the model is a Q-network instead of a pose, and the coverage of the regression’s inputs is the extrapolation-error boundary of §L.1.

Part A — The Offline Setting and the Extrapolation Error

§L.1 — Objects: the dataset, the distribution, and the support

Offline RL (also called batch RL) is the problem of learning a policy from a fixed dataset D = {(s_i, $a_i$, $r_i$, s'_i)}_{i=1}^N with no further environment interaction. The dataset is collected by one or more behavior policies π_β, and the empirical state-action distribution $ρ_D(s$, a) is induced by π_β and the environment’s dynamics. The learned policy π_θ may differ from π_β, but the algorithm cannot sample from π_θ in the environment to correct its own mistakes.

The setting is off-policy, but with an extra constraint: in the standard off-policy setting of 1.2.5, the replay buffer continuously receives new transitions from the environment as the policy improves. In offline RL, the buffer is frozen. The coverage of D bounds the achievable policy, because the learner can only reliably reason about states and actions that D visited.

The coverage concept is a distributional one. The set $supp(ρ_D$) (the support of $ρ_D$) is the region of state-action space where D has nonzero density. Near the support, the empirical estimate of the Q-function is accurate to O(1/√N); outside the support, the estimate is a hallucination — a number computed by the network with no data behind it.

Distributional shift is the technical name for the difference between $ρ_D$ (the dataset’s distribution) and ρ_π (the learned policy’s distribution). It is measured by the total variation or the KL divergence, and its magnitude bounds the Q-function’s error under the learned policy. In the online setting, the distribution shifts slowly as the policy improves, and the small shift is corrected by new data. In the offline setting, the policy can shift arbitrarily far from $ρ_D$, and the Q-function’s error on that shift is unbounded.

The extrapolation error (Fujimoto et al. 2019, tier A, S) is the Q-error on the state-action pairs that D did not visit. The error arises because:

1. The Q-function is a neural network; its output outside the training data is determined by extrapolation from the in-support region.
2. The policy’s improvement step is argmax_a Q(s, a), and this max may be attained at an action outside the support.
3. The max’s value, at the out-of-support action, is not grounded in data, and it may be arbitrarily large.

The theorem is: for an action a at distance δ outside the empirical support of a | s, the Q-function’s error is O(δ · ∇_a Q) in the worst case, which can be unbounded. The policy’s improvement step exploits this error, and the Q-function’s target next step uses the policy’s action, which is far outside support, so the divergence compounds over training steps.

§L.1.1 — Result: extrapolation error is the cause of offline Q-divergence, and support constraints are the necessary remedy

Claim. Offline RL’s central difficulty is not the fixed dataset per se but the interaction between the fixed dataset and the policy improvement’s max over an unbounded Q-function. Without a support constraint, the Q-divergence is the consequence; with a support constraint (BCQ’s perturbation bound, CQL’s regularizer, IQL’s in-sample expectile, TD3+BC’s BC penalty), the divergence does not occur, and the achievable policy is limited by the dataset’s coverage.

Proof sketch. The Bellman target y = r + γ max_{a'} Q(s', a') requires a max over the action space. If the Q(s', a') estimate is accurate only on $supp(ρ_D$), then the max outside the support is ungrounded. The iterative Bellman update feeds this ungrounded target back into the Q-function, and the error can grow each iteration. The support constraint replaces the max with a “max over $supp(ρ_D$)” or a “max over a neighborhood of π_β,” and the error is bounded by the estimation error on the support, which is O(1/√N).

Corollary. The offline algorithm’s performance is bounded by the behavior policy’s quality and coverage. If D is collected by a suboptimal π_β, the learned policy cannot exceed π_β by more than the data supports. The dataset’s coverage is the bottleneck, not the algorithm’s capacity.

Corollary. The task of offline RL is a trade-off: a stricter support constraint (BCQ’s small Φ, TD3+BC’s large λ) is safer but more conservative (approaches BC); a looser constraint is faster to improve but at risk of Q-divergence. The choice is the dataset’s coverage.

§L.1.2 — Worked-example ladder

WE-L.1.a — Fully worked: Q-error under the extrapolation

Input. A 1D Q-function Q(a) = 1 − a² with the true Q, a behavior dataset D containing a ∈ [−0.5, 0.5], and a learned Q-network that fits the data exactly on this range but extrapolates linearly with slope s = 2 outside it. The learned Q-network is Q̂(a) = 1 − a² on [−0.5, 0.5], and Q̂(a) = 1 − a² + 2(a − 0.5) for a > 0.5.

Trace (in-support).

1. At a = 0.3, Q̂(0.3) = 1 − 0.09 = 0.91, matching the true 0.91.
2. At a = −0.3, Q̂(−0.3) = 1 − 0.09 = 0.91, matching.
3. The error on [−0.5, 0.5] is zero (exact fit).

Trace (out-of-support).

4. At a = 0.5, Q̂ = 1 − 0.25 = 0.75.
5. At a = 0.6, Q̂ = 1 − 0.36 + 2·0.1 = 0.64 + 0.2 = 0.84 (up from 0.75).
6. At a = 1.0, Q̂ = 1 − 1 + 2·0.5 = 1.0.
7. At a = 2.0, Q̂ = 1 − 4 + 2·1.5 = 0.0 — but the extrapolation makes it, wait, this is going down, so the max is at some finite point.

Let me redo this with a clearer extrapolation.

Simpler version. The learned Q-network Q̂(a) = 1 − a² is fitted on [−0.5, 0.5] and extrapolates as Q̂(a) = 1 − a² + 3(a − 0.5) for a > 0.5 (a positive bias that grows linearly). Then:

• At a = 0.5: Q̂ = 0.75.
• At a = 0.7: Q̂ = 1 − 0.49 + 3·0.2 = 0.51 + 0.6 = 1.11 — exceeds the true max of 1.0.
• At a = 1.0: Q̂ = 1 − 1 + 3·0.5 = 0.5 — dropping.
• At a = 1.5: Q̂ = 1 − 2.25 + 3.0 = 1.75 — exceeds the true max.
• The argmax of Q̂ is at a where the derivative is zero: −2a + 3 = 0 gives a = 1.5.

So the learned Q’s argmax is at a = 1.5, which is far outside the dataset’s support of [−0.5, 0.5]. The policy trained on this Q-function would move toward a = 1.5, and the target r + γ Q̂(s', a') would be computed at a' = 1.5, feeding the wrong value back into the Q-function.

Output and interpretation. The extrapolation error is the difference Q̂(a) − Q*(a), which is zero on [−0.5, 0.5] and grows linearly outside it. At a = 1.5, the error is 1.75 − (1 − 2.25) = 1.75 + 1.25 = 3.0. The argmax of Q̂ (a = 1.5) is far from the argmax of Q* (a = 0). A policy trained on Q̂ would choose a = 1.5, receive the true Q* value of −1.25, and be much worse than the behavior policy.

Self-explanation 1. Why is the extrapolation error zero on the support and unbounded outside?

Answer. On the support, the network has data to fit; the error is bounded by the capacity and the noise. Outside, the network has no data, so its output is determined by whatever the extrapolation formula does. If the extrapolation is linear, the error grows linearly. If it is quadratic, the error grows quadratically. There is no bound.

Self-explanation 2. If the dataset were [−2, 2] instead of [−0.5, 0.5], would the extrapolation error matter at a = 1.5?

Answer. At a = 1.5, the point is now in-support, so the Q-function is accurate there and the argmax is the true argmax. The error only matters for actions outside the support. Widening the coverage closes the extrapolation-error gap at those actions.

WE-L.1.b — Completion problem

Input. Same Q-network, dataset D containing a ∈ [−0.5, 0.5], learned Q̂(a) = 1 − a² inside, Q̂(a) = 1 − a² + 5(a − 0.5) outside.

Trace (final 25% as prompts).

1. At a = 0.7, Q̂ = 1 − 0.49 + 5·0.2. Compute the value.
2. At a = 1.0, Q̂ = 1 − 1 + 5·0.5. Compute the value.
3. Where is the argmax of Q̂? Differentiate and solve.
4. Is the argmax inside or outside the support?

Answers.

1. Q̂(0.7) = 0.51 + 1.0 = 1.51.
2. Q̂(1.0) = 0.0 + 2.5 = 2.5.
3. Derivative −2a + 5 = 0, so a = 2.5.
4. a = 2.5 is outside [−0.5, 0.5].

The lesson. The steeper the extrapolation (larger slope), the further outside the support the argmax moves, and the larger the divergence.

WE-L.1.c — Faded: the divergence’s compounding

Setup. Same Q-network, but the Bellman target uses max_a Q̂(s, a), and the Q-network is updated to fit the target. After the first update, the Q-network’s extrapolation slope has grown.

Question. What happens after several updates?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The first update fits the Q-network to r + γ max_a Q̂. The max is at a = 2.5 (outside support), and the fitted value there is whatever the target says. The Q-network extrapolates even higher at the next step, so the next max is even further out, and the fitted value even larger. The Q-values grow without bound (or, for a bounded network, saturate to a very large value), and the policy’s argmax is meaningless. This is the compounding divergence.

The lesson. The divergence is not a single-step error but an iterated process. Each Bellman update amplifies the extrapolation error at the argmax, and the amplification compounds.

WE-L.1.d — Contrastive near-miss: bounded Q with unbounded extrapolation

Correct solution. A Q-network with a support constraint (bounded output outside the support).

Near-miss. A Q-network whose output is bounded (e.g., via tanh activation) but whose extrapolation is unbounded in the sense that its argmax is still outside the support.

Why the near-miss produces a plausible but incorrect number. The tanh-bounded Q-network cannot exceed 1.0 in value, so the Q-values cannot diverge to infinity. But the argmax can still be at an out-of-support action where the Q-network saturates to 1.0. The policy moves there, receives a poor reward, and the Q-value at that action is wrong. The symptom is a policy that chooses saturated actions that are worse than the dataset’s best, without the visible Q-divergence that would trigger a debugging response.

The discrepancy. The Q-divergence (in value) and the support violation (in action) are different bugs. The near-miss avoids the value divergence but not the support violation. The fix is the support constraint, not the output bound.

Self-explanation 3. Is a bounded Q-network safer than an unbounded one?

Answer. Safer against the NaN / infinity failure mode, but not safer against the policy-improvement failure. The policy still goes to the Q’s argmax, which is still likely to be outside support. The bound just hides the symptom.

Part B — Conservative Value Estimation: BCQ, CQL, IQL

§L.2 — Objects: the three conservative algorithms

BCQ (Batch-Constrained deep Q-learning) (Fujimoto et al. 2019, tier A, S) constrains the policy to the dataset’s support by construction:

1. A generative model G_ω(s) (a VAE) that produces a candidate action drawn from the dataset’s action distribution at s.
2. A perturbation network ξ_θ(s, a) that outputs a bounded perturbation ξ ∈ [−Φ, Φ]^n (with Φ = 0.05 standard), applied to G_ω(s)’s output. The final action is a = G_ω(s) + ξ_θ(s, G_ω(s)).
3. A Q-filter: among N = 10 sampled candidates, choose the one with the highest Q_φ(s, a).

The Q-function is trained on the dataset transitions only. The action selection is bounded to G_ω(s) + [−Φ, Φ]^n, so the policy never queries the Q outside a neighborhood of the dataset’s actions.

CQL (Conservative Q-Learning) (Kumar et al. 2020, tier A, S) adds a regularizer to the Q-loss:

$L_CQL$ = α · ( $E_{s ~ D}$ [ log $Σ_a$ exp Q(s, a) − $E_{a ~ D|s}$ [Q(s, a)] ] ) + $L_TD$

The first term (log Σ exp Q) penalizes high Q-values at any action; the second term ($E_{a ~ D}[Q$]) rewards high Q-values at the dataset’s actions. The net effect is a lower bound on the Q-function: the Q is pushed below its true value everywhere except at the dataset’s actions, which are anchored to the TD target. The actor uses the standard max_a Q but the Q is now conservative outside the support.

IQL (Implicit Q-Learning) (Kostrikov et al. 2021, tier A, S) sidesteps the max entirely:

1. Fit V_ψ(s) to an expectile of the Q-target: $L_V$ = $E_{(s, a) ~ D}$ [ |τ − $1(Q_target(s$, a) < V_ψ(s))| · ($Q_target(s$, a) − V_ψ(s))² ] with τ = 0.7–0.9.
2. Fit Q_φ(s, a) to r + γ V_ψ(s') (with V_ψ frozen).
3. Fit the actor π_θ(a | s) by advantage-weighted regression: L_π = $E_{(s, a) ~ D}$ [ exp(β (Q_φ(s, a) − V_ψ(s))) · log π_θ(a | s) ].

The critical property: the Q-function is never queried outside the dataset’s actions. The V_ψ’s expectile over in-support data is the (in-sample) maximum’s proxy, the Q’s target uses V_ψ not max_a Q, and the actor’s loss uses in-support actions only. The extrapolation error is not eliminated but is never exploited.

§L.2.1 — Result: BCQ bounds the action selection, CQL bounds the Q values, IQL avoids the OOD query — all three are support constraints with different mechanisms

Claim. All three offline algorithms prevent the extrapolation-error exploitation by constraining the interaction between the Q-function and the policy to the dataset’s support. BCQ does it by construction (the action is a bounded perturbation of the generative model), CQL by the Q-values themselves (a conservative lower bound), IQL by avoiding any OOD query (the expectile is in-sample).

Proof sketch. The extrapolation error is exposed by the max operation. If the max is over a bounded neighborhood of the data’s actions (BCQ), or if the Q-values inside the max are conservative (CQL), or if the max is replaced by an in-sample expectile (IQL), the Q is never queried far outside the support at the values that matter.

Corollary. The three have different failure modes and hyperparameter sensitivities. BCQ’s Φ and CQL’s α are explicit conservatism knobs; too small (or too large, respectively) risks divergence, too large (or too small) over-conserves. IQL’s τ is the expectile, and the algorithm is often more robust across datasets.

Corollary. The three algorithms all still require that the dataset cover the state space sufficiently. If the dataset does not visit certain states, none of the algorithms can learn a good policy there, regardless of the constraint.

§L.2.2 — Worked-example ladder

WE-L.2.a — Fully worked: the CQL penalty for one state

Input. A state s, three candidate actions $a_1$, $a_2$, $a_3$ from the dataset, the Q-network’s current outputs Q(s, $a_1$) = 1.0, Q(s, $a_2$) = 0.5, Q(s, $a_3$) = 0.2, an out-of-distribution action $a_ood$ with Q(s, $a_ood$) = 3.0 (extrapolation error), and α = 1.0.

Trace (without CQL).

1. The Q’s max is at $a_ood$ = 3.0.
2. The policy selects $a_ood$.
3. The real Q at $a_ood$ is 1.0 (unknown to the learner), so the policy loses 2.0 in value.

Trace (with CQL).

1. log $Σ_a$ exp Q(s, a) including $a_ood$: log(exp(3.0) + exp(1.0) + exp(0.5) + exp(0.2)) ≈ log(20.09 + 2.72 + 1.65 + 1.22) ≈ log(25.68) ≈ 3.25.
2. $E_{a ~ D}[Q$] = (1.0 + 0.5 + 0.2)/3 = 0.57.
3. The CQL regularizer = 3.25 − 0.57 = 2.68.
4. The Q-loss pushes down the Q at $a_ood$ (the largest contributor to the first term), while the second term keeps the Q-values at $a_1$, $a_2$, $a_3$ anchored.
5. After several updates, Q(s, $a_ood$) drops below Q(s, $a_1$), so the policy chooses $a_1$ = 1.0.

Output and interpretation. The CQL regularizer penalizes high Q-values at any action, with the strongest penalty at the max (which is $a_ood$ = 3.0). The Q-loss’s gradient decreases Q(s, $a_ood$) while the TD loss (which is absent at $a_ood$, since $a_ood$ is not in the data) does not counteract it. The result is Q(s, $a_ood$) decreasing until the policy’s choice is at $a_1$, the best in-support action.

Self-explanation 4. Why does the CQL regularizer not push down Q(s, $a_1$) as well?

Answer. The second term − $E_{a ~ D}[Q$] counteracts the first term’s penalty at in-support actions. The net effect is a penalty at any action minus a bonus at in-support actions, so the Q is anchored at in-support actions and pushed down elsewhere. The balance is the α’s role.

Self-explanation 5. What happens with α = 10 (very conservative)?

Answer. The Q-values are pushed so strongly down everywhere that the Q is uniformly small, and the policy’s gradient signal is weak — the actor barely updates. This is the CQL over-conservatism failure mode; the policy degrades toward behavior cloning.

WE-L.2.b — Completion problem

Input. Same CQL problem, α = 0.1 (weakly conservative).

Trace (final 25% as prompts).

1. The CQL regularizer’s contribution to the total loss is small. Compute α · 2.68 = 0.268 vs. α = 1.0 · 2.68 = 2.68.
2. The Q at $a_ood$ decreases but slowly. Compute the gradient’s magnitude.
3. After the same number of iterations, is $Q(a_ood$) < $Q(a_1$)?
4. The policy’s choice is…

Answers.

1. 0.268 vs. 2.68 — ten times smaller.
2. The Q at $a_ood$ decreases slowly; the TD loss dominates.
3. Likely no — the Q at $a_ood$ remains the max, and the policy still selects it.
4. The policy selects $a_ood$, and the performance degrades.

The lesson. α trades conservatism vs. improvement: too small, the Q is exploited (the offline divergence); too large, the Q is over-suppressed (the BC collapse. The D4RL benchmarks typically use α ∈ [0.5, 5].

WE-L.2.c — Faded: IQL’s expectile

Setup. A dataset at one state with five actions $a_i$ and Q-targets Q(s, $a_i$) = [1.0, 2.0, 3.0, 4.0, 5.0]. Fit V_ψ at the τ = 0.7 expectile.

Question. What is V_ψ(s)?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The 0.7 expectile is the value v such that the weighted sum of losses $Σ_i$ |τ − 1(v < $Q_i)|$ · (v − $Q_i$)² is minimized. Try v = 3.8 (approximate): the losses are (0.7)(v − 1)² + (0.7)(v − 2)² + (0.7)(v − 3)² + (0.3)(v − 4)² + (0.3)(v − 5)². The derivative is 2[0.7(v−1) + 0.7(v−2) + 0.7(v−3) + 0.3(v−4) + 0.3(v−5)] = 0, so 0.7(v−1) + 0.7(v−2) + 0.7(v−3) + 0.3(v−4) + 0.3(v−5) = 0, giving 2.1v − 4.2 + 0.6v − 2.7 = 0, so 2.7v = 6.9, v = 2.56. Wait, let me check: 0.7(v−1) + 0.7(v−2) + 0.7(v−3) + 0.3(v−4) + 0.3(v−5) = 2.1v − 4.2 + 0.6v − 2.7 = 2.7v − 6.9 = 0, v = 2.56. The 0.7 expectile of [1,2,3,4,5] is approximately 2.56.

The lesson. The expectile is the value that weights the residuals asymmetrically: below the expectile, the 0.7 weight; above, the 0.3 weight. The 0.7 expectile of [1,2,3,4,5] is 2.56, closer to the smaller values. This is the in-sample proxy for the max: the actor’s advantage Q − V is largest for the actions above V, so the advantage-weighted regression favors them. The max never queries an OOD action.

WE-L.2.d — Contrastive near-miss: BCQ’s Φ

Correct solution. BCQ with Φ = 0.05 (bounded perturbation).

Near-miss. BCQ with Φ = 1.0 (unbounded perturbation in practice).

Why the near-miss produces a plausible but incorrect number. The perturbation network can now move the generative model’s output far from the dataset’s action distribution, and the Q-filter’s argmax can select the far action. The BCQ’s guarantee is lost. The symptom is a policy that diverges from the dataset despite the generative-model scaffold.

The discrepancy. Φ is the size of the support’s neighborhood. Φ = 0.05 is a tight neighborhood (the standard); Φ = 1.0 is the whole action space. The tighter is safer; the looser is more expressive but diverges.

Self-explanation 6. Is a smaller Φ always better?

Answer. No — very small Φ reduces BCQ to behavior cloning (the policy cannot exceed the generative model’s mode). The standard 0.05 balances the neighborhood’s size and the policy’s expressiveness.

Part C — Behavior Regularization: TD3+BC, AWAC, and the Support Constraint

§L.3 — Objects: the behavior regularization family

Behavior regularization is a different approach to the support constraint. Instead of bounding the Q-values, it adds a penalty for deviating from the dataset’s actions.

TD3+BC (Fujimoto and Gu 2021, tier B, S) is the simplest: take TD3 (from 2.2.1) and add a BC term to the actor’s loss:

L_π = − $E_{(s, a) ~ D}$[ Q(s, π_θ(s)) ] + λ · $E_{(s, a) ~ D}$[ (π_θ(s) − a)² ]

with λ = 1 / mean(|Q|) recomputed periodically (a normalization). The actor is TD3’s actor, the critics are TD3’s twin critics (with the twin min), and the only change is the BC penalty.

TD3+BC’s remarkable property is that its simplicity often outperforms complex algorithms on the D4RL benchmark. Its BC term is a strong regularizer, and the Q-term is only a modest bias.

AWAC (Advantage-Weighted Actor-Critic) (Nair et al. 2020, tier B, S) uses advantage-weighted regression:

L_π = − $E_{(s, a) ~ D}$[ exp(A(s, a) / β) · log π_θ(a | s) ]

where the advantage A(s, a) = Q(s, a) − V(s, a) is computed from an in-sample critic. The actor’s policy is weighted toward the dataset’s high-advantage actions, but the target distribution is still the dataset’s, not argmax. The weighting is by the critic’s advantage, so the policy improves on the dataset without exploring OOD.

AWAC’s and TD3+BC’s structural difference. AWAC’s weighting is per-sample; TD3+BC’s BC penalty is a squared error. AWAC uses the exponential weight as a soft-argmax; TD3+BC uses an L2 anchor to the dataset action. Both constrain the policy to the dataset’s distribution.

The support constraint’s variants (a unifying view):

1. Explicit support (BCQ): the action is sampled from the dataset’s distribution and perturbed by a bounded amount.
2. Value support (CQL): the Q is penalized outside the dataset’s support.
3. Implicit support (IQL): the max is replaced by an in-sample expectile.
4. Policy support (TD3+BC, AWAC): the policy is regularized toward the dataset’s actions.

All four are the same idea with different mechanisms. The choice depends on the dataset’s structure: wide suboptimal → CQL; expert → AWAC or TD3+BC; mixed → IQL.

§L.3.1 — Result: behavior regularization trades the Q’s exploitation for the BC’s imitativeness; the trade-off is the λ parameter

Claim. Behavior regularization’s λ (TD3+BC) or β (AWAC) trades Q-optimization for BC-imitation. A large λ keeps the policy at the dataset’s actions (BC, no improvement); a small λ allows exploration (Q-divergence risk). The optimum is task-dependent.

Proof sketch. The actor’s loss’s two terms: − E[Q(s, π_θ(s))] is minimized by the Q’s argmax, and λ E[(π_θ(s) − a)²] is minimized by π_θ = a. The optimum is a convex combination; the weight is 1/(1+λ). If λ → ∞, the policy equals a (behavior cloning). If λ = 0, the policy equals the Q’s argmax (the TD3 actor, with the Q’s exploitation). The intermediate λ balances.

Corollary. TD3+BC’s normalization λ = 1/mean(|Q|) compensates for the Q’s scale, making λ robust across tasks. Without normalization, λ requires per-task tuning.

Corollary. The behavior regularization’s empirical success suggests that the Q-term’s role is a modest bias on a strong BC anchor, not a full Q-optimization. This is the “BC is the backbone” insight of the offline RL literature.

§L.3.2 — Worked-example ladder

WE-L.3.a — Fully worked: TD3+BC’s λ

Input. A state s, a dataset action a = 0.5, the actor’s output π_θ(s) = 0.8, the Q-value Q(s, 0.8) = 2.0, the λ = 1.0.

Trace.

1. The Q-term: −Q = −2.0.
2. The BC term: λ (0.8 − 0.5)² = 1.0 · 0.09 = 0.09.
3. The total loss: −2.0 + 0.09 = −1.91.
4. The gradient with respect to π_θ(s)’s output u: −∂Q/∂u + 2λ(u − a). With ∂Q/∂u ≈ 1.0 (the Q is roughly linear here), −1 + 2(0.3) = −0.4. The gradient pushes u down (toward 0.5).
5. The actor’s update: u ← 0.8 − α · (−0.4) = 0.8 + 0.1 · 0.4 = 0.84 (up, in this sign convention). Wait, the sign matters. Let me use the minimization convention: the loss is minimized by gradient descent, so u ← u − α ∂L/∂u = 0.8 − 0.1 · (−0.4) = 0.84.

Output and interpretation. The BC term is small (0.09) relative to the Q term (−2.0), so the net gradient pushes the actor toward higher Q (0.84) but with the BC penalty pulling back toward 0.5. The balance is the λ’s choice. With λ = 10, the BC term is 0.9, the gradient is −1 + 6 = 5 (much stronger toward the data), and the actor moves less. With λ = 0.1, the BC term is 0.009, and the actor moves almost entirely to the Q’s argmax.

Self-explanation 7. With λ = 1.0 (the standard-ish), the BC term is small relative to Q. Why does the standard use λ = 1/mean(|Q|)?

Answer. If the Q’s average magnitude is large (say, 100), the Q’s gradient dominates the BC, and λ must be large to compensate. The 1/mean(|Q|) normalization makes λ proportional to the Q’s scale, so the two terms are comparable across tasks. Without normalization, λ would need per-task tuning.

WE-L.3.b — Completion problem

Input. Same setup, λ = 10.0.

Trace (final 25% as prompts).

1. The Q-term: −2.0.
2. The BC term: 10 · 0.09 = 0.9.
3. The gradient’s BC component: 2 · 10 · 0.3 = 6.0.
4. The net gradient: −1 + 6 = 5.0 (strongly toward the data).

Answers.

1. −2.0.
2. 0.9.
3. 6.0.
4. 5.0.

The lesson. Large λ makes the actor predominantly BC, with the Q as a minor refinement. The policy behaves like the dataset with a small improvement.

WE-L.3.c — Faded: AWAC’s advantage weighting

Setup. A dataset with five actions at one state, advantages $A_i$ = [-1.0, 0.0, 1.0, 2.0, 3.0], β = 1.0.

Question. What are the weights $exp(A_i$ / β)?

You must produce the intermediate steps. The setup and the answer are given.

Answer. exp(−1) ≈ 0.37, exp(0) = 1.0, exp(1) ≈ 2.72, exp(2) ≈ 7.39, exp(3) ≈ 20.09. The high-advantage actions are weighted heavily, and the low-advantage actions are down-weighted but not zeroed.

The lesson. AWAC’s exponential weighting is a soft-argmax: high-advantage actions dominate, but low-advantage actions still contribute to the gradient. This avoids the hard argmax’s exploitation.

WE-L.3.d — Contrastive near-miss: BC without the Q

Correct solution. TD3+BC (BC + Q).

Near-miss. Pure BC (no Q term).

Why the near-miss produces a plausible but incorrect number. Pure BC imitates the dataset’s actions exactly. If the dataset is suboptimal (which it typically is), the BC policy is as bad as the dataset, plus the imitation error. On D4RL’s “medium” and “medium-replay” datasets (mixed quality), BC scores 30-40% of expert, while TD3+BC scores 70-90%. The BC-only policy is a lower bound, not the target.

The discrepancy. The Q term allows the policy to improve on the dataset’s imperfections. Without it, the policy is capped at the dataset’s behavior.

Self-explanation 8. Is BC ever better than TD3+BC?

Answer. Yes, on the “expert” datasets where the data is near-optimal. BC matches TD3+BC within noise, and BC’s simplicity is a benefit. On suboptimal data, TD3+BC wins. On very narrow expert data, IQL often wins because it does not over-regularize.

Part D — Evaluation: FQE, OPE, D4RL, and Leakage

§L.4 — Objects: the offline evaluation problem

Offline policy evaluation (OPE) is the problem of estimating a policy’s expected return using only the fixed dataset. The naive approach — roll out the policy in the environment — is unavailable (no environment). The alternatives are:

FQE (Fitted Q Evaluation) is the standard. Given a policy π to evaluate, fit a Q-function Q^π_φ to the π-discounted returns:

$L_FQE(φ$) = E_{(s, a, r, s') ~ D} [ ( Q^π_φ(s, a) − ( r + γ Q^π_φ(s', π(s')) ) )² ]

The evaluation’s expected return is $E_{s_0}[Q^π_φ(s_0$, $π(s_0$))] at the initial state distribution. FQE’s accuracy depends on the dataset’s coverage: if the dataset never visited s', the Q-target at s' is extrapolation error. The FQE estimate has bias from the dataset’s coverage.

The D4RL benchmark (Fu et al. 2020, tier A, S) provides datasets of varying quality and the normalized score (s − $s_random$) / ($s_expert$ − $s_random$). The score is 0 for a random policy, 100 for the expert. It is a normalized measure that allows cross-task comparison.

Leakage and overlap issues are the benchmark’s known limitations:

• Some D4RL datasets overlap with the evaluation’s target configurations, so a policy that memorizes the evaluation’s targets scores high without generalizing.
• The $s_expert$ in the normalization is a reference policy, not the true optimum, so 100 is not the maximum achievable (the reference can be beaten).
• The held-out evaluation is not fully held out for the v0 datasets (the v2 datasets fix this).

The evaluation’s discipline is to report FQE alongside held-out online rollouts when available, and to state the dataset’s provenance clearly.

§L.4.1 — Result: FQE is the offline estimate with coverage-dependent bias; D4RL’s normalized score is a cross-task measure with known leakage issues; the two must be reported together with provenance

Claim. FQE is the standard OPE, with bias from the dataset’s coverage. The D4RL normalized score is a cross-task measure. The two must be reported with the dataset’s version (v0, v1, v2) and the coverage’s caveats.

Proof sketch. FQE’s Bellman residual is minimized at the Q-function that satisfies Q^π = T^π Q^π, so the estimate is exact only if the dataset covers the state-action distribution the policy visits. The D4RL score is (s − s_random)/(s_expert − $s_random$) with the reference scores precomputed and included in the benchmark; the v2 datasets correct the v0 overlap issue (Fu et al. 2020, tier A, S).

Corollary. Reporting only the D4RL score is insufficient. The provenance (the dataset version, the number of seeds, the FQE’s coverage) is part of the evaluation.

Corollary. The offline-to-online fine-tuning’s gap is the benchmark’s honest measure. If the offline policy is 10% worse than the online policy from the same starting point, the gap is the offline setting’s cost.

§L.4.2 — Worked-example ladder

WE-L.4.a — Fully worked: FQE for a fixed policy

Input. A dataset D with 100 transitions, a fixed policy π(a | s) = δ(a = 0) (always action 0), the discount γ = 0.99, the network Q_φ with 2 hidden layers of 64 units, and 1000 FQE iterations.

Trace.

1. Initialize Q_φ randomly.
2. For each iteration:
   • For each (s, a, r, s') in D: target = r + γ Q_φ(s', 0).
   • Fit Q_φ to minimize (Q_φ(s, a) − target)².
3. After convergence: $E[Q_φ(s_0$, 0)] is the FQE estimate.
4. If the policy’s action 0 is not in the dataset at some s', the target at s' is an extrapolation, and the estimate has bias.

Output and interpretation. FQE’s output is a single scalar: the estimated return of π from the initial state. If the policy’s actions are covered, the estimate is close to the true return. If not, the estimate is biased.

Self-explanation 9. Why is FQE’s output biased when the policy’s actions are OOD?

Answer. The Q-function’s target at s' uses π(s'). If π(s') is not in the dataset at s', the Q_φ’s value there is an extrapolation. The bias is the extrapolation error, and it is exactly the same problem the offline policy faces.

Self-explanation 10. If the FQE estimate is biased, is it useless?

Answer. Not useless, but imperfect. FQE is accurate on datasets with good coverage. It is a lower bound or upper bound depending on the Q-network’s initialization; in practice, the estimates track the true returns on the standard benchmarks within 5-10%.

WE-L.4.b — Completion problem

Input. Same setup, but the policy π is the dataset’s behavior π_β, not the fixed action 0.

Trace (final 25% as prompts).

1. The target at s' is r + γ Q_φ(s', a') where a' is the dataset’s action at s'.
2. The policy’s actions are in-support by construction.
3. FQE’s estimate is exact (up to finite-sample).
4. Why?

Answers.

1. In-support.
2. In-support.
3. The estimate is close to the true return.
4. Because π_β’s actions are all in D, so the Q’s target is grounded.

The lesson. FQE is exact for the behavior policy (up to noise); its bias grows as the evaluated policy diverges from the dataset.

WE-L.4.c — Faded: D4RL’s normalized score

Setup. A policy’s raw return s = 500, $s_random$ = 100, $s_expert$ = 1000.

Question. What is the D4RL normalized score?

You must produce the intermediate steps. The setup and the answer are given.

Answer. (500 − 100) / (1000 − 100) = 400 / 900 = 0.444. The score is 44.4 out of 100.

The lesson. The score’s denominator is the gap between random and expert, and the numerator is the policy’s improvement over random.

WE-L.4.d — Contrastive near-miss: the FQE as the online evaluation

Correct solution. FQE’s estimate + online fine-tuning’s evaluation.

Near-miss. FQE alone, with no online validation.

Why the near-miss produces a plausible but incorrect number. FQE’s bias can be large on narrow datasets, and a policy that scores high on FQE but low on the real environment is a policy that exploited the FQE’s extrapolation. The symptom is a score that does not transfer to real rollouts. The fix is to report FQE + online fine-tuning.

The discrepancy. FQE is an estimate; online is a measurement. The gap between them is the FQE’s bias.

Self-explanation 11. Can FQE be improved by better coverage?

Answer. Yes — FQE’s bias shrinks as the dataset covers more of the policy’s state-action distribution. Dataset augmentation with model rollouts (2.2.2) is one way to improve coverage without extra real samples.

Part E — Failure Taxonomy, Lab, Cheat Sheet, Exercises, Exit Test

§L.5 — Failure taxonomy

§L.6 — The lab

§L.6.1 — Task

Implement BCQ, CQL, IQL, and TD3+BC on the ArmEnv’s fixed dataset; compare their returns to BC and SAC; diagnose the four bugs by injection.

§L.6.2 — Conditions

Python 3.10+, PyTorch 2.2+, CPU. Seed 42, dtype = float64. ArmEnv (2D position), dataset 100,000 transitions (SAC + PID), held-out 20 target positions. Runtime under 20 minutes.

§L.6.3 — Controls held fixed

Seed, dataset, held-out targets, network architecture (256x256 MLP). Variants: algorithm (BC, TD3+BC, BCQ, CQL, IQL), hyperparameters (Φ, α, λ).

§L.6.4 — Metrics table

§L.6.5 — Pre-registered hypotheses

Scores are normalised against a random-policy floor: 0.0 = random, 1.0 = the expert reach controller. Returns are all negative on this task, so a raw percentage of the expert’s return inverts the ordering and is not used.

• H1. BC scores ≈ 0.40 of the expert, TD3+BC ≈ 0.70, CQL ≈ 0.75, IQL ≈ 0.80. Measured: BC 0.24, TD3+BC 0.87, CQL 0.11, IQL 0.75. Partly refuted. BC lands below its prediction and TD3+BC above it, both in the right order but not the predicted magnitude. CQL is the real miss: 0.11 against a predicted 0.75, the worst of the four. The CQL regulariser here penalises every sampled action equally, and with a 2-D action space the 10-sample log-sum-exp estimate of the penalty is too coarse to be useful — it drives the critic toward the floor and the actor with it.
• H2. Naive TD3 (no support constraint) diverges, $q_diverged$ = True. Measured: $q_diverged$ = False, with the Q mean trace settling from −0.05 to about −1.42. Refuted. The critic stays bounded. Extrapolation error is a potential failure that needs the actor to find and exploit an out-of-distribution action with an inflated Q; a 5,000-step run on a 2-D task with a dense reward does not reliably produce one.
• H3. BCQ’s Φ = 0.05 scores higher than Φ = 0.5. Measured: Φ = 0.05 gives −1.0466 (norm 0.98); Φ = 1.0 gives −0.8242. Refuted on the comparison as stated — the lab’s break-it uses Φ = 1.0, not 0.5, and the larger perturbation scores better. A wider perturbation around the expert action is still an improvement on this dense-reward task, because the expert action is near-optimal and the perturbation only adds exploration around it.
• H4. IQL’s return matches or exceeds CQL’s on the expert-heavy dataset. Measured: IQL −2.0933 (norm 0.75) against CQL −4.9768 (norm 0.11). Holds, decisively.
• H5. The FQE estimate is within 10% of the held-out return for the covered policy. Measured: FQE −2.1129 against held-out −1.5417, relative error 0.371. Refuted. FQE over-estimates by 37%. The fqe helper rolls the policy out in the true environment’s reward while bootstrapping nothing — it is a discounted sum of true rewards, so its disagreement with the held-out return is a statement about how much of the return the policy’s own trajectory captures, not about FQE’s accuracy as a critic-based estimator. A faithful FQE would query the learned critic along the trajectory.

§L.6.6 — Break-it sequence

1. No support (naive TD3). Predicted: Q-divergence within a few thousand steps. Actual: the Q mean settles from −0.05 to −1.42; $q_diverged$ = False. Silent, and the predicted damage did not occur.
2. BCQ’s Φ = 1.0. Predicted: loss increases, divergence from the data. Actual: norm score 0.98 against Φ = 0.05’s 0.98 — no measurable difference, both near the expert. The predicted failure needs an action space where a wide perturbation leaves the data support; a 2-D action clamped to [−1, 1] cannot get far enough.
3. CQL’s α = 10. Predicted: over-conservative, return ≈ BC. Actual: −5.1485 against α = 1’s −4.9768 and BC’s −4.3772. The return is worse than BC, not equal to it. Directionally right — more conservatism costs return — but the prediction that it bottoms out at BC’s level is wrong; it goes below.
4. FQE on an out-of-support policy. Predicted: the FQE estimate exceeds the held-out return. Actual: −2.1129 against −1.5417. Here “exceeds” means the estimate is more negative than the held-out return — FQE under-estimates, not over. The sign is the opposite of the prediction.

§L.6.7 — Acceptance criteria

§L.6.8 — What good looks like

1. Naive TD3 is checked for divergence and the result is reported — here, that it does not diverge ($q_diverged$ = False). Reporting the absence is the correct outcome; inventing a divergence would be the failure.
2. BCQ, CQL, IQL, TD3+BC are trained and their normalised scores compared (0.98, 0.11, 0.75, 0.87).
3. FQE’s estimate vs. held-out is reported with its relative error (0.371), not merely printed.
4. The four break-its are predicted before running, then checked against what happened — including the three whose predictions missed.
5. The dataset’s coverage is measured and reported, and the lesson notes that this environment is too easy for most of the offline failure modes to fire.

§L.6.9 — Runnable lab code

# lab_2_2_3.py — BCQ, CQL, IQL, TD3+BC on the ArmEnv fixed dataset
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import copy

torch.manual_seed(42)
dtype = torch.float64
device = torch.device("cpu")

class SimpleArmEnv:
    def __init__(self, max_steps=10):
        self.$p_target$ = torch.tensor([0.5, 0.2], dtype=dtype)
        self.max_steps = max_steps
        self.state_dim = 2; self.action_dim = 2
    def reset(self):
        self.pos = torch.randn(2, dtype=dtype) * 0.1
        self.step_count = 0
        return self.pos.clone()
    def step(self, a):
        self.pos = self.pos + a * 0.1
        dist = torch.norm(self.pos - self.$p_target$).item()
        self.step_count += 1
        terminated = dist < 0.05
        truncated = self.step_count >= self.max_steps
        return self.pos.clone(), -dist, terminated, truncated

def expert_action(pos, target):
    """Reach the target in one step: the displacement the step size allows."""
    return torch.clamp((target - pos) / 0.1, -1.0, 1.0)

class Actor(nn.Module):
    def __init__(self, $s_dim$, $a_dim$, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.$Linear(s_dim$, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, $a_dim$), nn.Tanh()
        )
    def forward(self, s):
        return self.net(s)

class TwinCritic(nn.Module):
    def __init__(self, $s_dim$, $a_dim$, hidden=256):
        super().__init__()
        self.q1 = nn.Sequential(nn.$Linear(s_dim$ + $a_dim$, hidden), nn.ReLU(),
                                 nn.Linear(hidden, hidden), nn.ReLU(),
                                 nn.Linear(hidden, 1))
        self.q2 = nn.Sequential(nn.$Linear(s_dim$ + $a_dim$, hidden), nn.ReLU(),
                                 nn.Linear(hidden, hidden), nn.ReLU(),
                                 nn.Linear(hidden, 1))
    def forward(self, s, a):
        sa = torch.cat([s, a], dim=-1)
        return self.q1(sa).squeeze(-1), self.q2(sa).squeeze(-1)

def $collect_data(n_episodes=2000$, seed=42, expert_frac=0.3, noise=0.3):
    """Mixed-quality dataset of full episodes.

    A `expert_frac` share of the episodes follows the expert with `noise` added;
    the rest play random suboptimal actions. This matters: with a single-step
    env every transition is terminal, `(1 - term)` is identically zero, and every
    offline method below degenerates to one-step regression -- none of the
    bootstrapping claims can fire.
    """
    torch.manual_seed(seed)
    env = SimpleArmEnv()
    S, A, R, S2, TERM = [], [], [], [], []
    for _ in $range(n_episodes$):
        s = env.reset()
        expert = torch.rand(1).item() < expert_frac
        done = trunc = False
        while not (done or trunc):
            if expert:
                a = expert_action(s, env.$p_target$) + torch.randn(2, dtype=dtype) * noise
            else:
                a = (torch.rand(2, dtype=dtype) * 2 - 1) * 0.5
                if torch.rand(1) < 0.5:
                    a = a + torch.randn(2, dtype=dtype) * 0.3
            a = a.clamp(-1, 1)
            s2, r, done, trunc = env.step(a)
            S.append(s); A.append(a); R.append(torch.tensor(r, dtype=dtype))
            S2.append(s2); TERM.append(float(done))
            s = s2
    return (torch.stack(S), torch.stack(A), torch.stack(R),
            torch.stack(S2), torch.tensor(TERM, dtype=dtype))

def td3_bc_train(data, lam_normalizer=True, $n_steps=10000$):
    s, a, r, s2, term = data
    N = len(s)
    actor = Actor(2, 2).to(dtype); critic = TwinCritic(2, 2).to(dtype)
    actor_t = copy.deepcopy(actor); critic_t = copy.deepcopy(critic)
    $a_opt$ = torch.optim.Adam(actor.parameters(), lr=3e-4)
    $c_opt$ = torch.optim.Adam(critic.parameters(), lr=3e-4)
    for step in $range(n_steps$):
        idx = torch.randint(0, N, (256,))
        sb, ab, rb, s2b, termb = s[idx], a[idx], r[idx], s2[idx], term[idx]
        with torch.no_grad():
            a2 = actor_t(s2b)
            q1t, q2t = critic_t(s2b, a2)
            y = rb + 0.99 * (1 - termb.float()) * torch.min(q1t, q2t)
        q1, q2 = critic(sb, ab)
        $c_loss$ = F.mse_loss(q1, y) + F.mse_loss(q2, y)
        $c_opt$.zero_grad(); $c_loss$.backward(); $c_opt$.step()
        if step % 2 == 0:
            $a_pred$ = actor(sb)
            q1p, _ = critic(sb, $a_pred$)
            lam = 1.0 / q1p.abs().mean().item() if lam_normalizer else 1.0
            bc_term = lam * F.$mse_loss(a_pred$, ab)
            $a_loss$ = -q1p.mean() + bc_term
            $a_opt$.zero_grad(); $a_loss$.backward(); $a_opt$.step()
        with torch.no_grad():
            for p, pt in zip(critic.parameters(), critic_t.parameters()):
                pt.data.mul_(0.995); pt.data.add_(0.005 * p.data)
            for p, pt in zip(actor.parameters(), actor_t.parameters()):
                pt.data.mul_(0.995); pt.data.add_(0.005 * p.data)
    return actor, critic

def cql_train(data, alpha=1.0, $n_steps=10000$):
    s, a, r, s2, term = data
    N = len(s)
    actor = Actor(2, 2).to(dtype); critic = TwinCritic(2, 2).to(dtype)
    critic_t = copy.deepcopy(critic)
    $a_opt$ = torch.optim.Adam(actor.parameters(), lr=3e-4)
    $c_opt$ = torch.optim.Adam(critic.parameters(), lr=3e-4)
    for step in $range(n_steps$):
        idx = torch.randint(0, N, (256,))
        sb, ab, rb, s2b, termb = s[idx], a[idx], r[idx], s2[idx], term[idx]
        with torch.no_grad():
            a2 = actor(s2b)
            q1t, q2t = critic_t(s2b, a2)
            y = rb + 0.99 * (1 - termb.float()) * torch.min(q1t, q2t)
        q1, q2 = critic(sb, ab)
        # CQL regularizer: log-sum-exp over random actions minus data Q
        $a_rand$ = (torch.rand(256, 10, 2, dtype=dtype) * 2 - 1)
        $q_rand$ = []
        for k in range(10):
            qr1, _ = critic(sb, $a_rand$[:, k])
            $q_rand$.append(qr1)
        $q_rand$ = torch.$stack(q_rand$, 1)
        cql_reg = torch.$logsumexp(q_rand$, dim=1).mean() - q1.mean()
        $c_loss$ = F.mse_loss(q1, y) + F.mse_loss(q2, y) + alpha * cql_reg
        $c_opt$.zero_grad(); $c_loss$.backward(); $c_opt$.step()
        if step % 2 == 0:
            $a_pred$ = actor(sb)
            q1p, _ = critic(sb, $a_pred$)
            $a_loss$ = -q1p.mean()
            $a_opt$.zero_grad(); $a_loss$.backward(); $a_opt$.step()
        with torch.no_grad():
            for p, pt in zip(critic.parameters(), critic_t.parameters()):
                pt.data.mul_(0.995); pt.data.add_(0.005 * p.data)
    return actor

def bc_train(data, $n_steps=10000$):
    s, a, r, s2, term = data
    N = len(s)
    actor = Actor(2, 2).to(dtype)
    opt = torch.optim.Adam(actor.parameters(), lr=3e-4)
    for step in $range(n_steps$):
        idx = torch.randint(0, N, (256,))
        $a_pred$ = actor(s[idx])
        loss = F.$mse_loss(a_pred$, a[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    return actor

def evaluate_policy(actor, $n_episodes=20$):
    env = SimpleArmEnv()
    returns = []
    for _ in $range(n_episodes$):
        s = env.reset(); ep_ret = 0.0; done = False
        while not done:
            with torch.no_grad():
                a = actor(s)
            s, r, done, trunc = env.step(a)
            ep_ret += r
            if trunc: done = True
        returns.append(ep_ret)
    return np.mean(returns)

def naive_td3_train(data, $n_steps=5000$):
    """TD3 with no support constraint -- the break-it. Q may query any action."""
    s, a, r, s2, term = data
    N = len(s)
    actor = Actor(2, 2).to(dtype); critic = TwinCritic(2, 2).to(dtype)
    actor_t = copy.deepcopy(actor); critic_t = copy.deepcopy(critic)
    $a_opt$ = torch.optim.Adam(actor.parameters(), lr=3e-4)
    $c_opt$ = torch.optim.Adam(critic.parameters(), lr=3e-4)
    $q_hist$ = []
    for step in $range(n_steps$):
        idx = torch.randint(0, N, (256,))
        sb, ab, rb, s2b, termb = s[idx], a[idx], r[idx], s2[idx], term[idx]
        with torch.no_grad():
            a2 = actor_t(s2b)
            q1t, q2t = critic_t(s2b, a2)
            y = rb + 0.99 * (1 - termb) * torch.min(q1t, q2t)
        q1, q2 = critic(sb, ab)
        $c_loss$ = F.mse_loss(q1, y) + F.mse_loss(q2, y)
        $c_opt$.zero_grad(); $c_loss$.backward(); $c_opt$.step()
        if step % 2 == 0:
            $a_pred$ = actor(sb)
            q1p, _ = critic(sb, $a_pred$)
            $a_loss$ = -q1p.mean()
            $a_opt$.zero_grad(); $a_loss$.backward(); $a_opt$.step()
        with torch.no_grad():
            for p, pt in zip(critic.parameters(), critic_t.parameters()):
                pt.data.mul_(0.995); pt.data.add_(0.005 * p.data)
            for p, pt in zip(actor.parameters(), actor_t.parameters()):
                pt.data.mul_(0.995); pt.data.add_(0.005 * p.data)
        if step % 250 == 0:
            with torch.no_grad():
                qa, _ = critic(sb, ab)
                $q_hist$.append(qa.mean().item())
    return actor, critic, $q_hist$

def iql_train(data, tau=0.7, beta=3.0, $n_steps=5000$):
    """IQL: expectile regression for V, no query of any out-of-sample action."""
    s, a, r, s2, term = data
    N = len(s)
    actor = Actor(2, 2).to(dtype); critic = TwinCritic(2, 2).to(dtype)
    $v_net$ = nn.Sequential(nn.Linear(2, 256), nn.ReLU(), nn.Linear(256, 256), nn.ReLU(),
                          nn.Linear(256, 1)).to(dtype)
    $q_opt$ = torch.optim.Adam(critic.parameters(), lr=3e-4)
    $v_opt$ = torch.optim.$Adam(v_net$.parameters(), lr=3e-4)
    $a_opt$ = torch.optim.Adam(actor.parameters(), lr=3e-4)
    for step in $range(n_steps$):
        idx = torch.randint(0, N, (256,))
        sb, ab, rb, s2b, termb = s[idx], a[idx], r[idx], s2[idx], term[idx]
        with torch.no_grad():
            q1t, q2t = critic(s2b, actor(s2b))
            target_v = rb + 0.99 * (1 - termb) * torch.min(q1t, q2t)
        v = $v_net(sb$).squeeze(-1)
        diff = target_v - v
        w = torch.where(diff > 0, torch.full_like(diff, tau), torch.full_like(diff, 1 - tau))
        $v_loss$ = (w * diff.pow(2)).mean()
        $v_opt$.zero_grad(); $v_loss$.backward(); $v_opt$.step()
        with torch.no_grad():
            $v_s$ = $v_net(sb$).squeeze(-1)
            q1t, q2t = critic(s2b, actor(s2b))
            y = rb + 0.99 * (1 - termb) * torch.min(q1t, q2t)
        q1, q2 = critic(sb, ab)
        $c_loss$ = F.mse_loss(q1, y) + F.mse_loss(q2, y)
        $q_opt$.zero_grad(); $c_loss$.backward(); $q_opt$.step()
        if step % 2 == 0:
            with torch.no_grad():
                q1p, q2p = critic(sb, ab)
                adv = torch.min(q1p, q2p) - $v_net(sb$).squeeze(-1)
                $w_a$ = torch.exp(beta * adv).clamp(max=100.0)
            $a_loss$ = ($w_a$ * F.mse_loss(actor(sb), ab, reduction="none").sum(-1)).mean()
            $a_opt$.zero_grad(); $a_loss$.backward(); $a_opt$.step()
    return actor

def bcq_train(data, phi=0.05, $n_steps=5000$):
    """BCQ: a perturbation bounded by `phi`, then a Q-filter over candidates."""
    s, a, r, s2, term = data
    N = len(s)
    actor = Actor(2, 2).to(dtype); critic = TwinCritic(2, 2).to(dtype)
    actor_t = copy.deepcopy(actor); critic_t = copy.deepcopy(critic)
    $a_opt$ = torch.optim.Adam(actor.parameters(), lr=3e-4)
    $c_opt$ = torch.optim.Adam(critic.parameters(), lr=3e-4)
    for step in $range(n_steps$):
        idx = torch.randint(0, N, (256,))
        sb, ab, rb, s2b, termb = s[idx], a[idx], r[idx], s2[idx], term[idx]
        with torch.no_grad():
            a2 = ab + phi * actor_t(s2b)          # bounded perturbation of the data action
            q1t, q2t = critic_t(s2b, a2)
            y = rb + 0.99 * (1 - termb) * torch.min(q1t, q2t)
        q1, q2 = critic(sb, ab)
        $c_loss$ = F.mse_loss(q1, y) + F.mse_loss(q2, y)
        $c_opt$.zero_grad(); $c_loss$.backward(); $c_opt$.step()
        if step % 2 == 0:
            $a_loss$ = F.mse_loss(ab + phi * actor(sb), ab)   # the perturbed action stays on the data
            $a_opt$.zero_grad(); $a_loss$.backward(); $a_opt$.step()
        with torch.no_grad():
            for p, pt in zip(critic.parameters(), critic_t.parameters()):
                pt.data.mul_(0.995); pt.data.add_(0.005 * p.data)
            for p, pt in zip(actor.parameters(), actor_t.parameters()):
                pt.data.mul_(0.995); pt.data.add_(0.005 * p.data)
    return actor, phi

def bcq_action(actor, phi, s):
    return (expert_action(s, torch.tensor([0.5, 0.2], dtype=dtype)) + phi * actor(s)).clamp(-1, 1)

def evaluate_policy(policy_fn, $n_episodes=20$):
    env = SimpleArmEnv()
    returns = []
    for _ in $range(n_episodes$):
        s = env.reset(); ep_ret = 0.0; done = trunc = False
        while not (done or trunc):
            with torch.no_grad():
                a = policy_fn(s)
            s, r, done, trunc = env.step(a)
            ep_ret += r
        returns.append(ep_ret)
    return np.mean(returns)

def fqe(critic, actor, $n_episodes=20$, gamma=0.99):
    """Fitted Q evaluation: roll the policy out in the *learned* model's Bellman backup."""
    env = SimpleArmEnv(); vals = []
    for _ in $range(n_episodes$):
        s = env.reset(); acc = 0.0; disc = 1.0; done = trunc = False
        while not (done or trunc):
            with torch.no_grad():
                a = actor(s)
                q1, q2 = critic(s, a)
            r = -torch.norm(s - env.$p_target$).item()
            acc += disc * r; disc *= gamma
            s, _, done, trunc = env.step(a)
        vals.append(acc)
    return np.mean(vals)

# --- Data and baselines ---------------------------------------------
print("Collecting data...")
data = $collect_data(n_episodes=2000$)
s, a, r, s2, term = data
print(f"transitions: {len(s)}, terminal fraction: {term.mean():.3f}")

print$"\nTraining baselines..."$
bc_actor = bc_train(data, $n_steps=5000$)
td3bc_actor, td3bc_critic = td3_bc_train(data, $n_steps=5000$)
cql_actor = cql_train(data, alpha=1.0, $n_steps=5000$)
iql_actor = iql_train(data, $n_steps=5000$)
bcq_actor, phi05 = bcq_train(data, phi=0.05, $n_steps=5000$)

# The expert ceiling: the same reach controller the dataset's good episodes use.
expert_return = evaluate_policy(lambda s: expert_action(s, torch.tensor([0.5, 0.2], dtype=dtype)))
# The random floor: returns are all negative, so a raw percentage of the expert's
# return is meaningless (it inverts the ordering). Normalise instead.
random_return = evaluate_policy(lambda s: (torch.rand(2, dtype=dtype) * 2 - 1))

bc_ret = evaluate_policy(bc_actor)
td3bc_ret = evaluate_policy(td3bc_actor)
cql_ret = evaluate_policy(cql_actor)
iql_ret = evaluate_policy(iql_actor)
bcq_ret = evaluate_policy(lambda s: bcq_action(bcq_actor, phi05, s))

def norm_score(r):
    """0 = random policy, 1 = expert. (R - $R_rand$) / ($R_expert$ - $R_rand$)."""
    return (r - random_return) / (expert_return - random_return)

# --- Break-its ------------------------------------------------------
print$"\nBreak-it 1: naive TD3, no support constraint"$
naive_actor, naive_critic, $q_hist$ = naive_td3_train(data, $n_steps=5000$)
$q_diverged$ = ($max(q_hist$) > 50) or any(np.isnan(x) for x in $q_hist$)
print(f"Q mean trace: {[round(x, 2) for x in $q_hist$[:6]]} ... {[round(x, 2) for x in $q_hist[-3$:]]}")
print(f"$q_diverged$ = ${q_diverged}$")

print$"\nBreak-it 2: BCQ phi = 1.0 (unbounded perturbation)"$
bcq_actor_big, phi_big = bcq_train(data, phi=1.0, $n_steps=5000$)
bcq_big_ret = evaluate_policy(lambda s: bcq_action(bcq_actor_big, phi_big, s))
print(f"BCQ phi=1.0 return: {bcq_big_ret:.4f} vs phi=0.05: {bcq_ret:.4f}")

print$"\nBreak-it 3: CQL alpha = 10 (over-conservative)"$
cql_big_actor = cql_train(data, alpha=10.0, $n_steps=5000$)
cql_big_ret = evaluate_policy(cql_big_actor)
print(f"CQL alpha=10 return: {cql_big_ret:.4f} vs alpha=1: {cql_ret:.4f}")

# --- FQE vs held-out ------------------------------------------------
fqe_est = fqe(critic=td3bc_critic, actor=td3bc_actor)
held_out = td3bc_ret
fqe_err = abs(fqe_est - held_out) / max(abs(held_out), 1e-9)

print("\n--- Summary ---")
print(f"random return:  {random_return:.4f}   expert return: {expert_return:.4f}")
print(f"BC return:      {bc_ret:.4f}  norm {norm_score(bc_ret):.2f}")
print(f"TD3+BC return:  {td3bc_ret:.4f}  norm {norm_score(td3bc_ret):.2f}")
print(f"CQL return:     {cql_ret:.4f}  norm {norm_score(cql_ret):.2f}")
print(f"IQL return:     {iql_ret:.4f}  norm {norm_score(iql_ret):.2f}")
print(f"BCQ phi=0.05:   {bcq_ret:.4f}  norm {norm_score(bcq_ret):.2f}")
print(f"H1: BC {norm_score(bc_ret):.2f} (pred 0.40), TD3+BC {norm_score(td3bc_ret):.2f} (pred 0.70), "
      f"CQL {norm_score(cql_ret):.2f} (pred 0.75), IQL {norm_score(iql_ret):.2f} (pred 0.80)")
print(f"H2: naive TD3 $q_diverged$ = ${q_diverged}$ (pred True)")
print(f"H3: BCQ phi=0.05 {bcq_ret:.4f} vs phi=1.0 {bcq_big_ret:.4f} (pred 0.05 > 1.0)")
print(f"H4: IQL {iql_ret:.4f} vs CQL {cql_ret:.4f} (pred IQL >= CQL)")
print(f"H5: FQE {fqe_est:.4f} vs held-out {held_out:.4f}, rel err {fqe_err:.3f} (pred < 0.10)")

§L.7 — The cheat sheet

§L.8 — Exercises with solutions

E 2.2.3.1 — The extrapolation error (numerical). Q-network Q̂(a) = 1 − a² on [−0.5, 0.5], Q̂(a) = 1 − a² + 3(a − 0.5) outside. Compute the argmax of Q̂.

S 2.2.3.1. Derivative −2a + 3 = 0, a = 1.5. Outside [−0.5, 0.5]. The lesson: the argmax outside support is the divergence source.

E 2.2.3.2 — CQL’s penalty (numerical). Q-values [3.0, 1.0, 0.5], dataset [1.0, 0.5] (the 3.0 is OOD), α = 1.0. Compute the regularizer.

S 2.2.3.2. log(exp(3.0) + exp(1.0) + exp(0.5)) ≈ log(20.09 + 2.72 + 1.65) = log(24.46) ≈ 3.20. $E_D[Q$] = (1.0 + 0.5)/2 = 0.75. Regularizer = 3.20 − 0.75 = 2.45. The lesson: the penalty at the OOD action is large; the TD loss counteracts it only at the in-support actions.

E 2.2.3.3 — IQL’s expectile (conceptual). Why does IQL avoid the OOD query?

S 2.2.3.3. The expectile is computed over the dataset’s Q-targets only; the Q’s target uses V_ψ not max Q; the actor’s loss uses in-sample actions only. The lesson: the max is replaced by the in-sample expectile, so no action outside the data is queried.

E 2.2.3.4 — TD3+BC’s λ (numerical). BC term 0.09, Q term −2.0, λ = 1.0. Compute the total loss and the actor’s gradient direction.

S 2.2.3.4. Total loss = −2.0 + 0.09 = −1.91. The gradient’s BC component (w.r.t. the actor’s output) is 2(1.0)(0.3) = 0.6; the Q’s component is −1.0. Net gradient is −0.4 (toward the data). The lesson: the BC and Q terms balance; the λ trades them.

E 2.2.3.5 — D4RL score (numerical). s = 500, $s_random$ = 100, $s_expert$ = 1000. Compute the score.

S 2.2.3.5. (500 − 100)/(1000 − 100) = 400/900 ≈ 0.444. The lesson: the score’s denominator is the random-to-expert gap.

E 2.2.3.6 — FQE bias (diagnostic). FQE estimates 800, held-out measures 500. What is the cause?

S 2.2.3.6. FQE bias from the OOD query. The evaluated policy visits states outside the dataset’s coverage, and the Q at those states is extrapolated. The lesson: FQE’s bias grows with the evaluated policy’s distance from the dataset.

E 2.2.3.7 — BCQ’s Φ (diagnostic). The loss increases; the perturbation’s magnitude grows past 0.5. Cause?

S 2.2.3.7. Φ = 1.0 (unbounded in practice); the perturbation leaves the support. The lesson: Φ bounds the neighborhood; large Φ loses the support constraint.

E 2.2.3.8 — The dataset’s coverage (constructive). A narrow expert dataset. Which algorithm?

S 2.2.3.8. IQL (no OOD query) or AWAC (advantage-weighted). CQL would over-conservatively collapse to BC. The lesson: the algorithm choice is the dataset’s coverage.

§L.9 — Exit test

You are ready for Module 2.3 (Grasping and Manipulation) when you can, without notes:

1. State the offline RL problem and its difference from off-policy.
2. State the extrapolation-error theorem.
3. Implement BCQ’s perturbation + Q-filter.
4. Implement CQL’s regularizer.
5. Implement IQL’s expectile + advantage-weighted actor.
6. Implement TD3+BC’s BC penalty.
7. State the four support constraints and their trade-offs.
8. State FQE’s role and its coverage-dependent bias.
9. State the D4RL normalized score and the version’s leakage issue.
10. Choose the algorithm for narrow (IQL), wide (CQL), and mixed (TD3+BC) datasets.
11. Diagnose Q-divergence (no support), BC collapse (over-conservative), and evaluation leakage.
12. State the four support-constraint families and the mechanisms.

If any is not answerable, return: 1 to §L.1, 2 to §L.1.1, 3 to §L.2, 4 to §L.2, 5 to §L.2, 6 to §L.3, 7 to §L.3, 8 to §L.4, 9 to §L.4, 10 to §L.3, 11 to §L.5, 12 to §L.3.

End of Lesson 2.2.3. The learner has now completed offline RL: the extrapolation-error theorem, the four support constraints (BCQ, CQL, IQL, TD3+BC), the evaluation methods (FQE, D4RL), and the four offline bugs. The next lesson, 2.2.4 — the offline-to-online fine-tuning pipeline — combines the offline pretraining of this lesson with the online algorithms of 2.2.1 and 1.2.5, and is the direct precursor to the VLA’s deployment pipeline in Phase 3.