X1.2.5

Lesson 1.2.5 — Soft Actor-Critic: Maximum Entropy, the Twin Critics, and the Off-Policy Alternative

Week 9 · Prerequisites: 1.2.1, 1.2.2, 1.2.3, 1.2.4, 0.1.2, 0.3.2.

Prior results used. This lesson uses 1.2.1’s ArmEnv (the observation/action spaces, the terminated/truncated distinction), 1.2.2’s value functions V^π/Q^π and the Bellman equations, 1.2.3’s policy gradient theorem, the advantage, the Gaussian policy, and the tanh squashing’s Jacobian, 1.2.4’s trust-region motivation, clipped objective, GAE, and target-network stability, 0.1.2’s autograd and torch.no_grad(), and 0.3.2’s MAP and regularization (for the temperature’s interpretation). Each is restated at the point of use.

Hands forward. This lesson hands forward one capability: the learner can derive the maximum-entropy objective and its soft Bellman equations, implement SAC’s three-network architecture (the actor, the twin critics, and the target critics) with the reparameterization trick, the clipped double-Q target, the replay buffer, and the automatic temperature tuning, and diagnose the five SAC-specific bugs (target network missing, twin critics’ min omitted, temperature collapse, buffer staleness, and reparameterization’s gradient detach) from their learning symptoms — the capability that Module 2.2 (advanced RL), Module 3.1 (VLA fine-tuning), and the Phase 4 capstone all assume.

Running case. The same 6-DOF MuJoCo arm from 1.1.1–1.2.4, in the ArmEnv, now with SAC as the learning algorithm: the actor π_θ(a | s) is a tanh-squashed Gaussian (the mean and the log-std from the network, the a = tanh(μ + σ · ε) with the Jacobian’s correction from 1.2.3), the twin critics $Q_{φ_1}(s$, a) and $Q_{φ_2}(s$, a) (two separate networks, the min of the two for the target), the target critics $Q_{\barφ_1}$ and $Q_{\barφ_2}$ (the slowly updated copies), the replay buffer (the 10⁶ transitions, the off-policy sampling), and the temperature α (the entropy’s weight, the automatic tuning). The lesson’s central result is that the maximum-entropy objective’s soft Bellman equations add the −α log π term to the value, and the reparameterization trick a = tanh(μ_θ(s) + σ_θ(s) ε) allows the actor’s gradient to flow through the sampling, and the twin critics’ min removes the overestimation bias that the single critic’s max introduces.

§L.0 — Lesson map, objectives, and misconceptions

§L.0.1 — Lesson map and word budget

§L.0.2 — Objectives

By the end of this lesson you will be able to do eight things:

1. State the maximum-entropy objective J(π) = $E_π[Σ_t$ γ^t ($r_t$ + α H[π(·$|s_t$)])] and the soft value functions $V^π_soft(s$) = $E_π[Q_soft(s$, a) − α log π(a|s)] and $Q_soft(s$, a) = r + γ $E_{s'}[V_soft(s$')].
2. Derive the soft Bellman equations for the critic Q(s, a) = r + γ $E_{s'}[V(s$')] with V(s') = $E_{a' ~ π}[Q(s$', a') − α log π(a'|s')], and state the soft policy iteration’s convergence (the soft Bellman operator’s contraction).
3. State the reparameterization trick a = tanh(μ_θ(s) + σ_θ(s) · ε) with the ε ~ N(0, I), and explain why the actor’s gradient flows through the sampling (the pathwise derivative) where the REINFORCE’s score function (1.2.3) would be high-variance.
4. Implement the actor’s loss L_π = $E_{s, ε}[α$ log π(a|s) − $Q_{min}(s$, a)] with the a = tanh(μ + σε) and the Jacobian’s correction, and state which tensor is detached — the temperature α, not the Q: the actor’s pathwise gradient runs through $Q_min$, so detaching it would delete the only learning signal.
5. Implement the critic’s loss $L_Q$ = $E_{(s,a,r,s')}$[ (Q_φ(s, a) − y)² ] with the target y = r + γ $min_{i=1,2} Q_{\barφ_i}(s', a') − α log π(a'|s')$ (the clipped double-Q), and the a' ~ π(·|s') sampled from the current policy (the off-policy’s target).
6. State the automatic temperature tuning L_α = $E_{s, a}[-α$ $log π(a|s) + \bar H$] $the `\bar H = −|A|` the **target entropy**$, and explain why the α’s automatic tuning is more stable than the fixed α (the task’s exploration’s adaptation).
7. State the target network’s soft update \barφ ← τ φ + (1 − τ) \barφ with the τ = 0.005, and explain why the soft update is needed (the target’s stability, the divergence’s avoidance).
8. Diagnose the five SAC-specific bugs — target network missing (the Q’s divergence), the twin critics’ min omitted (the Q’s overestimation), the temperature collapse (α → 0, the exploration’s loss), the buffer staleness (the old data’s bias), the misplaced detach (detaching $Q_min$ leaves only the entropy term, so the actor’s gradient loses the reward path) — from their learning symptoms.

§L.0.3 — Misconception inventory

Each misconception is displaced in the section named, and each is tagged in the concept check that follows.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

This is the twenty-first lesson. The prompt requires drawing from lessons two to five steps back; 1.2.4 is one step back and is excluded. The warm-up draws from 1.2.3 (two steps back), 1.2.2 (three), 1.2.1 (four), and 1.1.4 (five).

Warm-up 1 (from 1.2.3, the score function’s zero-mean). State the score function’s property E[∇ log π · b(s)] = 0. Answer: the baseline b(s)’s gradient’s expectation is zero (the score function’s zero-mean), and the baseline reduces the variance without the bias. How this is used: §L.3’s reparameterization is the alternative to the score function — the pathwise’s gradient flows through the a, not the log π (the likelihood-ratio’s estimator), and the two are different estimators of the same policy gradient.

Warm-up 2 (from 1.2.2, the advantage and the Bellman). State the advantage A(s, a) = Q(s, a) − V(s) and the Bellman’s Q(s, a) = r + γ E[V(s')]. Answer: the advantage is the excess over the state’s value, and the Bellman’s Q is the immediate reward plus the discounted future value. How this is used: §L.1’s soft Bellman is the 1.2.2’s Bellman with the entropy’s added term −α log π, and the soft value is the advantage’s entropy’s regularized version.

Warm-up 3 (from 1.2.1, the ArmEnv’s rollout). State the ArmEnv’s step’s return and the terminated/truncated’s distinction. Answer: the step returns (obs, reward, terminated, truncated, info), and the terminated is the task’s end (no bootstrap), the truncated is the time limit (bootstrap). How this is used: §L.2’s replay buffer stores the (s, a, r, s', terminated, truncated) tuples, and the target’s bootstrap uses the terminated/truncated’s distinction (the V = 0 for the terminated, the V = V(s') for the truncated).

Warm-up 4 (from 1.1.4, the pipeline’s stage contract). State the pick-and-place pipeline’s stage and the health check. Answer: the seven stages with the input-output-frame contract and the health check (the reachability, the branch continuity, the tracking, the grasp’s force). How this is used: §L.2’s SAC’s training evaluates the learned policy on the ArmEnv (the simplified pipeline), and the health check is the episode’s terminated (the success) and the reward’s design.

Part A — The Maximum-Entropy Objective

§L.1 — Objects: the soft value functions and the soft Bellman equations

The maximum-entropy objective is:

where:

• α > 0 is the temperature — the entropy’s weight (the learnable parameter, the Part B).
• H[π(·|s)] = −Σ_a π(a|s) log π(a|s) (the discrete) or = −∫ π(a|s) log π(a|s) da (the continuous) is the entropy of the policy at the state s.
• The reward’s sum is augmented with the entropy’s sum, and the objective maximizes the both (the task’s return and the policy’s entropy).

The soft value functions:

The soft value adds the −α log π term to the Q, and the soft Q is the immediate reward plus the discounted soft value. The two are the standard V/Q with the entropy’s subtraction, and the −α log π is the entropy’s contribution to the value (the higher the π(a|s), the smaller the −α log π, the larger the soft value — the entropy’s bonus for the probable actions).

The soft Bellman equations:

The soft policy iteration:

• Soft policy evaluation: the soft Bellman operator T^π_soft is the contraction (γ-contraction in the ∞-norm, the same as the 1.2.2’s standard), and the unique fixed point is the V^π_soft.
• Soft policy improvement: the new policy is the Boltzmann distribution over the current soft Q:

The Boltzmann’s policy is the optimal for the maximum-entropy objective (the closed-form’s solution), and the α is the temperature (the large α → the uniform policy, the small α → the greedy policy).

§L.1.1 — Result: the soft Bellman operator is a γ-contraction, and the Boltzmann policy is the optimal for the maximum-entropy objective — the entropy’s bonus regularizes the policy toward the multimodal and robust behavior

Claim. The soft Bellman operator $T^π_soft$ defined by the soft value functions is a γ-contraction in the ∞-norm, and the soft policy iteration (the evaluation + the Boltzmann improvement) converges to the unique V^*_soft and the optimal π^*_soft (the Boltzmann distribution over the soft Q’s* actions). The entropy’s bonus smooths the greedy policy (the α → 0’s limit is the standard V*/π*, and the α → ∞’s limit is the uniform policy).

Proof sketch. The soft Bellman operator’s contraction is the 1.2.2’s max replaced by the soft log-sum-exp (the Boltzmann improvement), and the log-sum-exp is a non-expansion (|LSE(x) − LSE(y)| ≤ max|x − y|), so the contraction holds with the γ. The Boltzmann policy’s optimality is the Kullback-Leibler’s minimization (the standard result — the optimal policy is the exponential of the Q/α), and the soft policy iteration’s convergence is the contraction’s corollary.

Corollary. The entropy’s bonus smooths the greedy policy — the Boltzmann’s distribution assigns the probabilities to all the actions, and the exploration is built into the policy (not the separate noise). The smoothing’s benefit is the robustness (the multimodal tasks, the adversarial perturbations, the sim-to-real’s distributional shift), and the cost is the less precise convergence (the α’s finite temperature).

Corollary. The α is the regularization’s strength — the large α is the strong (the uniform policy, the high exploration), the small α is the weak (the greedy policy, the high exploitation). The learned α (the automatic tuning, the Part B) adapts to the task’s needs, and the fixed α is the simplification.

§L.1.2 — Worked-example ladder

WE-L.1.a — Fully worked: the soft value for a discrete 2-action example

Input. A 2-action MDP: the Q(s, $a_1$) = 1.0, the Q(s, $a_2$) = 0.5, the α = 0.5. Compute the Boltzmann policy and the soft value $V_soft(s$).

Trace.

1. The Boltzmann policy: the $π(a_1|s$) ∝ exp(1.0/0.5) = exp(2) ≈ 7.39, the $π(a_2|s$) ∝ exp(0.5/0.5) = exp(1) ≈ 2.72.
2. The normalization: the Z = 7.39 + 2.72 = 10.11, the $π(a_1$) ≈ 0.731, the $π(a_2$) ≈ 0.269.
3. The soft value: the $V_soft(s$) = $Σ_a$ π(a) (Q(s, a) − α log π(a)) = 0.731·(1.0 − 0.5 log 0.731) + 0.269·(0.5 − 0.5 log 0.269).
4. The log 0.731 ≈ -0.313, the log 0.269 ≈ -1.313.
5. The $V_soft(s$) = 0.731·(1.0 + 0.1565) + 0.269·(0.5 + 0.6565) = 0.731·1.1565 + 0.269·1.1565 = 1.1565·(0.731 + 0.269) = 1.1565.

Output and interpretation. The Boltzmann policy favors the higher Q ($a_1$), but it doesn’t collapse to the greedy ($π(a_1$) ≈ 0.73, not 1.0) — the α = 0.5 keeps the exploration (the $a_2$‘s 0.27). The soft value 1.1565 is larger than the greedy’s 1.0 (the entropy’s bonus adds the 0.1565), and the soft value’s definition includes the entropy’s contribution to the action’s value.

Self-explanation 1. Before reading on: the soft value 1.1565 is larger than the greedy 1.0 — is the entropy adding to the value?

Answer. Yes — the soft value $V_soft(s$) = Σ π(a) (Q(s, a) − α log π(a)) includes the −α log π(a), and the probable actions (the π(a) > 1) have the negative log π, so the −α log π(a) > 0, and the entropy’s term adds the positive value. The $V_soft$ is larger than the V (the greedy’s value), and the difference is the entropy’s bonus’s contribution — the policy’s randomness is valued by the objective.

Self-explanation 2. Before reading on: the α = 0.5 gives the $π(a_1$) ≈ 0.73. What would the α → 0 and the α → ∞ give?

Answer. The α → 0 sharpens the Boltzmann to the greedy — the $π(a_1$) → 1.0, the $π(a_2$) → 0, and the policy is the deterministic argmax Q. The α → ∞ flattens the Boltzmann to the uniform — the $π(a_1$) = $π(a_2$) = 0.5, and the policy is the random. The α interpolates the two — the finite α is the smooth Boltzmann, and the entropy’s bonus prevents the collapse to the greedy and the loss of the exploration.

WE-L.1.b — Completion problem

Input. The same 2-action MDP, but the α = 0.1 (the colder).

Trace (final 25% as prompts).

1. The exp(1.0/0.1) = exp(10) ≈ 22026, the exp(0.5/0.1) = exp(5) ≈ 148.4. ⟨your turn⟩
2. The normalization: the Z ≈ 22174, the $π(a_1$) ≈ 0.993, the $π(a_2$) ≈ 0.0067. ⟨your turn⟩
3. The soft value: the $V_soft$ ≈ 1.0 (the greedy’s value + the small entropy). ⟨your turn⟩
4. The α’s effect: the colder α sharpens the policy to the greedy. ⟨your turn⟩

Answers.

1. The exp(10) ≈ 22026, the exp(5) ≈ 148.4. ✓
2. The $π(a_1$) ≈ 0.993, the $π(a_2$) ≈ 0.0067. ✓
3. The $V_soft$ ≈ 1.0 (the greedy’s value). ✓
4. The colder α sharpens the policy. ✓

The lesson. The α’s effect is the sharpness — the α = 0.1 is the nearly greedy (the $π(a_1$) ≈ 0.99), and the α = 0.5 is the moderate (the 0.73). The temperature’s choice determines the policy’s exploration (the large α is the broad, the small is the narrow), and the automatic tuning (the Part B) adapts it.

WE-L.1.c — Faded: the soft Bellman for a 1-step episode

Setup. A 1-step episode: the r(s, a) = 1.0, the terminal (the V(s') = 0), the α = 0.5, the Q(s, $a_1$) = 1.0, the Q(s, $a_2$) = 0.5.

Question. What is the soft Bellman’s Q? What is the soft value?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The soft Bellman’s Q(s, a) = r(s, a) + γ $E[V_soft(s$')]. The $V_soft(s$') = 0 (the terminal), so the Q(s, a) = r(s, a) = 1.0 and the 0.5. The soft value is the Σ π(a)(Q − α log π) ≈ 1.1565 (the same as the WE-L.1.a). The terminal’s $V_soft$ = 0 is the soft Bellman’s boundary, and the episode’s end is the no future.

The lesson. The soft Bellman’s $V_soft(s$') is the terminal’s zero for the episodic tasks (the terminated’s V = 0), and the truncated’s $V_soft(s$') is the bootstrap (the 1.2.1’s M3). The soft value’s terminal’s handling is the same as the standard Bellman’s, and the terminated/truncated’s distinction is the soft Bellman’s correctness’s key.

WE-L.1.d — Contrastive near-miss: the entropy’s sign

Correct solution. The soft value $V_soft$ = E[Q − α log π] (the minus sign), and the entropy’s bonus is the positive.

Near-miss. The soft value $V_soft$ = E[Q + α log π] (the plus sign), and the entropy’s penalty is the negative.

Why the near-miss produces a plausible but incorrect number. The plus sign makes the −α log π the penalty (the probable actions’ negative log is the positive log, and the plus adds the positive, the larger the probability, the larger the value), and the objective prefers the certain (the π → 0 is the log π → −∞, and the plus gives the negative infinity — the soft value is unbounded below). The symptom is the divergence (the soft value NaN), and the fix is the minus sign (the standard).

The discrepancy, stated. The minus and the plus signs differ by the entropy’s sign — the minus is the entropy’s bonus (the positive for the probable), the plus is the penalty (the positive for the certain, the negative for the uncertain). The minus is the standard (the SAC’s objective’s maximization of the entropy), and the plus is the wrong.

Self-explanation 3. Before reading on: the $V_soft$ = E[Q − α log π] — if the π(a) = 1 (the certain), what is the −α log π?

Answer. The log 1 = 0, so the −α log π = 0 — the certain action’s entropy’s contribution is zero. The $V_soft$ = Q for the certain action (the entropy’s bonus is zero), and the soft value coincides with the Q at the deterministic policy. The entropy’s bonus is the positive for the uncertain (the π(a) < 1, the log π < 0, the −α log π > 0), and the certain’s zero is the boundary.

Part B — The SAC Algorithm

§L.2 — Objects: the three networks, the replay buffer, and the temperature

The SAC algorithm’s components:

1. The actor π_θ(a | s) — the tanh-squashed Gaussian (the mean μ_θ(s) and the log-std log σ_θ(s) from the network, the a = tanh(μ + σε)).
2. The twin critics $Q_{φ_1}(s$, a) and $Q_{φ_2}(s$, a) — the two separate Q-networks (the min of the two for the target).
3. The target critics $Q_{\barφ_1}$ and $Q_{\barφ_2}$ — the slowly updated copies $the **soft** update `\barφ ← τφ + (1 − τ)\barφ` with the `τ = 0.005`$.
4. The replay buffer D — the 10⁶ transitions, the FIFO (the oldest evicted when the buffer is full).
5. The temperature α — the learnable parameter $the **automatic** tuning, the **target entropy** `\bar H = −|A|`$.

The SAC loop:

1. Collect the transition (s, a, r, s', terminated, truncated) by running the actor in the env, and store it in the buffer.
2. Sample a minibatch of the B = 256 transitions from the buffer (the uniform sampling, the any transition’s probability).
3. Compute the target y = r + γ(1 − terminated)$min_i Q_{\barφ_i}(s', a') − α log π_θ(a' | s')$ with the a' ~ π_θ(·|s') (the current policy’s sample, the off-policy’s a').
4. Update the critics $L_Q$ = $Σ_i$ $E[(Q_{φ_i}(s$, a) − y)²] (the twin’s both updated to the same target).
5. Update the actor L_π = $E_{s, ε}[α$ log π_θ(a | s) − min_i $Q_{φ_i}(s$, a)] with the a = tanh(μ_θ(s) + σ_θ(s) · ε) (the reparameterization, the Part C).
6. Update the temperature L_α = $E_{s, a}[-α$$log π_θ(a|s) + \bar H$] $the **target entropy** `\bar H = −|A|`$.
7. Update the target critics \barφ_i ← τφ_i + (1 − τ)\barφ_i (the soft update).
8. Repeat.

The hyperparameters:

• lr = 3e-4 (the standard for the Adam).
• γ = 0.99 (the discount).
• τ = 0.005 (the target’s soft update’s rate).
• B = 256 (the minibatch’s size).
• |D| = 10⁶ (the buffer’s size).
• hidden = 256 (the network’s width).
• $α_init$ = 0.2 (the initial temperature).
• \bar H = −|A| (the target entropy, the negative of the action’s dimension).
• target_update_interval = 1 (the every step’s soft update).

The rationale for each is the empirical balance, and the task’s tuning is the refinement.

§L.2.1 — Result: the SAC’s critic’s target uses the current policy’s sample on the past state (the off-policy’s s' and the on-policy’s a'), and the twin’s min reduces the overestimation — the entropy’s bonus regularizes the policy and the temperature’s automatic tuning adapts the exploration

Claim. The SAC’s critic’s target y = r + γ(1 − terminated)$min_i Q_{\barφ_i}(s', a') − α log π(a'|s')$ is the soft Bellman’s unbiased estimate (given the current policy and the twin’s min), and the twin’s min removes the single critic’s max’s overestimation (the max of the noisy estimates is biased upward, the min of the two is the pessimistic). The replay buffer’s off-policy sampling is the sample efficiency’s source (the many gradient’s steps per transition), and the temperature’s automatic tuning adapts the entropy’s weight to the task’s needs.

Proof sketch. The soft Bellman’s target is the r + γ $V_soft(s$'), and the $V_soft(s$') = $E_{a' ~ π}[Q(s$', a') − α log π(a'|s')] is the expectation over the current policy’s actions. The single sample a' ~ π is the unbiased estimator of the expectation (the one sample’s expectation is the expected value), and the twin’s min is the pessimistic lower bound (the less noisy of the two estimates, the overestimation’s reduction). The entropy’s automatic tuning is the constrained optimization $the maximize the expected return subject to the entropy's target `\bar H`$, and the Lagrangian’s dual gives the α’s gradient.

Corollary. The SAC’s sample efficiency is the off-policy’s advantage — the replay buffer stores the past transitions, and the critic’s update samples from the whole buffer (the many gradient’s steps per transition). The PPO’s on-policy reuse (1.2.4’s K = 10) is the limited version, and the SAC’s 10⁶’s buffer is the large reuse (the thousands of gradient’s steps per transition).

Corollary. The temperature’s automatic tuning is the entropy’s target’s constrained optimization — the \bar H = −|A| is the target (the negative of the action’s dimension), and the α’s gradient increases the α when the entropy is below the target (the more exploration) and decreases it when the entropy is above (the more exploitation). The learned α adapts to the task’s difficulty, and the fixed α is the simplification.

§L.2.2 — Worked-example ladder

WE-L.2.a — Fully worked: the SAC’s critic’s target for a specific transition

Input. The transition (s, a, r = 1.0, s', terminated = False), the twin critics’ $Q_1(s$', a') = 2.0, $Q_2(s$', a') = 1.5, the actor’s log π(a'|s') = −0.5, the α = 0.2, the γ = 0.99.

Trace.

1. The twin’s min: the min(2.0, 1.5) = 1.5.
2. The entropy’s subtraction: the 1.5 − 0.2·(−0.5) = 1.5 + 0.1 = 1.6.
3. The target: the y = 1.0 + 0.99·(1 − 0)·1.6 = 1.0 + 1.584 = 2.584.

Output and interpretation. The target y = 2.584 is the critic’s regression’s goal, and the components are the immediate reward (1.0), the discounted min’s Q (1.5·0.99 = 1.485), and the entropy’s contribution (0.2·0.5·0.99 = 0.099). The twin’s min (1.5) is the pessimistic (the less optimistic of the two), and the entropy’s subtraction (−0.2·(−0.5) = +0.1) adds the positive (the probable action’s entropy’s bonus).

Self-explanation 4. Before reading on: the log π(a'|s') = −0.5 is the negative, and the −α log π = +0.1 is the positive — why does the negative log give the positive entropy’s contribution?

Answer. The log π(a'|s') = −0.5 means the π(a'|s') ≈ 0.61 (the probable action), and the entropy’s term −α log π = +0.1 rewards the probable action (the positive contribution to the value). The sign is the convention — the entropy’s H = −log π for the probable is the positive (the low entropy), and the value’s bonus is the positive for the probable. The log π’s negative is the probable’s positive’s source, and the SAC’s objective rewards the probable actions.

Self-explanation 5. Before reading on: the target’s 2.584 is the critic’s Q(s, a)’s regression’s goal — what is the critic’s loss if the current Q_φ(s, a) = 2.0?

Answer. The critic’s loss is (Q_φ(s, a) − y)² = (2.0 − 2.584)² = 0.341 — the squared difference. The gradient 2(2.0 − 2.584) = −1.168 pushes the Q_φ(s, a) toward the target (the larger 2.584), and the optimizer’s step increases the Q_φ(s, a). The twin’s both critics are updated with the same target, and the two’s different initializations provide the ensemble’s diversity.

WE-L.2.b — Completion problem

Input. The same transition, but the terminated = True (the task’s end).

Trace (final 25% as prompts).

1. The target’s bootstrap: the (1 − terminated) = 0, so the future’s no contribution. ⟨your turn⟩
2. The target y = r = 1.0 (the immediate reward’s only). ⟨your turn⟩
3. The critic’s loss (Q_φ − 1.0)². ⟨your turn⟩
4. The interpretation: the terminated’s V = 0 (the no future), and the target is the reward. ⟨your turn⟩

Answers.

1. The (1 − terminated) = 0. ✓
2. The y = 1.0. ✓
3. The loss (Q_φ − 1.0)². ✓
4. The terminated’s no future. ✓

The lesson. The terminated’s (1 − terminated) = 0 removes the future’s bootstrap, and the target is the immediate reward. The truncated’s (1 − truncated) = 1 keeps the bootstrap, and the target is the r + γ(min Q − α log π). The two are different (the 1.2.1’s M3), and the wrong handling biases the target by the γ(min Q − α log π)’s magnitude.

WE-L.2.c — Faded: the actor’s loss

Setup. The same state s, the actor’s sample a = tanh(μ_θ(s) + σ_θ(s) ε) with the μ = 0.3, the σ = 0.5, the ε = 0.2, the log π(a|s) = −0.5 (the squashed), the critic’s $Q_min(s$, a) = 1.5, the α = 0.2.

Question. What is the actor’s loss? What is the gradient’s direction?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The actor’s loss L_π = α log π(a|s) − $Q_min(s$, a) = 0.2·(−0.5) − 1.5 = −0.1 − 1.5 = −1.6. The gradient’s direction is the minimize the L_π, which is the maximize the −L_π = 1.6 (the policy’s objective’s value), and the $Q_min$’s maximize (the better action’s probability) and the entropy’s maximize (the exploration).

The lesson. The actor’s loss L_π = α log π − $Q_min$ is the negative of the policy’s objective (the maximize the soft Q), and the gradient increases the probability of the high-Q actions and maintains the entropy (the α log π’s term). The two terms are the task’s exploitation (−$Q_min$) and the exploration (α log π), and the α is the balance.

WE-L.2.d — Contrastive near-miss: the fixed temperature

Correct solution. The learnable α (the automatic tuning), the target entropy \bar H = −|A|, and the gradient ∇_α L_α = −E[log π + \bar H].

Near-miss. The fixed α = 0.2.

Why the near-miss produces a plausible but incorrect number. The fixed α = 0.2 is the reasonable for the simple tasks, and it works for the single task. But the task’s difficulty varies — the harder tasks (the multimodal rewards, the sparse signals) need the larger α (the more exploration), and the easier tasks need the smaller (the more exploitation). The fixed α underperforms the harder tasks (the exploration’s insufficient) and overexplores the easier (the convergence’s slow). The symptom is the suboptimal return on the varied tasks, and the fix is the automatic tuning.

The discrepancy, stated. The fixed α and the learned α differ by the adaptation — the fixed is the simple but suboptimal (the same α for the different tasks), the learned is the complex but adaptive (the α adjusts to the entropy’s target). The learned α is the standard (the SAC’s default), and the fixed is the simplification for the quick prototyping.

Self-explanation 6. Before reading on: the fixed α = 0.2 on a harder task (the multimodal reward) — what is the symptom?

Answer. The symptom is the local optimum — the fixed α’s exploration is insufficient for the multimodal (the two modes), and the policy converges to one mode (the local optimum), missing the other (the higher mode). The learned α would increase (the entropy’s target), and the policy explores the both modes (the global optimum). The fixed α’s failure is the exploration’s insufficiency, and the learned α’s advantage is the adaptation.

Part C — The Reparameterization Trick

§L.3 — Objects: the pathwise gradient and the tanh squashing’s Jacobian

The reparameterization trick for the Gaussian policy:

The key is that the randomness is moved to the ε (the parameter-free noise), and the a’s dependence on the θ is the explicit (the μ_θ and the σ_θ), so the gradient ∇_θ a flows through the μ_θ and the σ_θ (the pathwise derivative). The REINFORCE’s score function (∇_θ log π · G) is the likelihood-ratio’s estimator (the high variance), and the pathwise’s gradient is the low variance (the gradient’s information is used).

The tanh squashing’s Jacobian: the a = tanh(z) with the z = μ + σε, and the da/dz = 1 − tanh²(z) = 1 − a². The log-likelihood log π(a | s) = log p(z | s) − log |da/dz| = log p(z | s) − log(1 − a²), and the Jacobian’s correction is the −log(1 − a²) (the 1.2.3’s WE-L.4.c’s standard).

The actor’s loss:

with the a = tanh(μ_θ(s) + σ_θ(s) · ε). The critic’s Q stays attached: the actor’s gradient is the pathwise ∇_a Q_min · ∇_θ a_θ, and a_θ is the only route from θ into Q_min, so detaching Q_min would delete that route. What is detached is the temperature α: it is a scalar owned by alpha_opt, and letting the actor’s loss write to log_alpha gives it a second, conflicting objective. The critic’s parameters φ receive no gradient from the actor’s loss either way, because critic_opt.zero_grad() clears them before their own update.

§L.3.1 — Result: the reparameterization trick’s pathwise gradient is lower variance than the score function’s, and the tanh squashing’s Jacobian must be included — the detach that matters is on the temperature, not on the critic’s Q

Claim. The pathwise gradient ∇_θ E_{ε}[f(a_θ(ε))] = E_{ε}[∇_a f(a_θ(ε)) · ∇_θ a_θ(ε)] has lower variance than the score function’s ∇_θ E_{a ~ π}[f(a)] = E_{a ~ π}[f(a) · ∇_θ log π(a|s)], because the pathwise uses the f’s gradient (∇_a f), and the score function uses the f’s value (the no gradient) times the log-likelihood’s gradient (the high variance). The tanh’s Jacobian is the required (the change-of-variables’s log |da/dz|), and the α’s detach is the essential (the temperature’s parameters are not updated by the actor’s loss).

Proof sketch. The pathwise gradient is the E[∇_a f · ∇_θ a], and the variance is the Var[∇_a f · ∇_θ a], which is small because the ∇_a f is deterministic given the a, and the ∇_θ a is the smooth function of the θ. The score function’s variance is the Var[f · ∇_θ log π], which is large because the f is the random (the return’s spread), and the ∇_θ log π is the random (the action’s spread). The two variances’ ratio is the pathwise’s advantage (the large for the continuous actions, the smaller for the discrete).

Corollary. The tanh’s Jacobian’s omission biases the log-likelihood, and the actor’s gradient is wrong — the policy learns the unbounded action’s behavior (the large torques), and the real robot saturates. The Jacobian’s inclusion is the bounded action’s correctness, and the SAC’s standard includes it.

Corollary. The α’s detach is the essential — the actor’s loss’s gradient flows through the a’s dependence on the θ and through Q_min, and the temperature log_alpha receives its gradient from L_α alone. The no detach on the α gives log_alpha two objectives that disagree (the actor wants entropy up, L_α wants the target entropy hit), and the measured cost is −1.3877 against −1.0889. The detach on Q_min is the opposite error and a worse one: it leaves only the entropy term and the actor reaches −4.8900.

§L.3.2 — Worked-example ladder

WE-L.3.a — Fully worked: the pathwise gradient for a 1D Gaussian

Input. A 1D Gaussian policy a = μ_θ + σ ε with the μ_θ = θ, the σ = 1, the ε ~ N(0, 1), the critic Q(a) = a². The current θ = 0, the sample ε = 0.5.

Trace.

1. The sample: the a = 0 + 1·0.5 = 0.5.
2. The pathwise gradient: the ∇_θ Q(a(θ)) = ∇_a Q · ∇_θ a = 2a · 1 = 2·0.5 = 1.0.
3. The expectation over the ε: the E[2(θ + ε)] = 2θ = 0 (the unbiased).

Trace (the score function’s comparison).

4. The score function’s estimator: the Q(a) · ∇_θ log π(a|s) = a² · (a − θ)/σ² = 0.25 · 0.5 = 0.125.
5. The expectation over the a: the E[a² · a] = E[a³] = 0 (the odd moment, the unbiased).
6. The variance: the pathwise’s Var[2(θ + ε)] = 4·1 = 4, the score function’s Var[a³] = E[a⁶] − 0 = 15 (the sixth moment of the standard normal).

Output and interpretation. The pathwise’s 1.0 is the sample, and the variance 4 is the smaller than the score function’s 15. The pathwise’s gradient is the low variance (the 3.75× smaller in this example), and the SAC’s actor uses the pathwise (the efficient). The score function’s high variance is the REINFORCE’s limitation, and the pathwise is the SAC’s advantage.

Self-explanation 7. Before reading on: the pathwise’s variance 4 and the score function’s 15 — what is the ratio, and what is the sample’s efficiency’s improvement?

Answer. The ratio 15/4 = 3.75, and the sample’s efficiency is the 3.75× (the same gradient’s accuracy with the 3.75× fewer samples). The pathwise’s low variance is the SAC’s sample efficiency’s source, and the ratio’s value depends on the problem (the larger for the higher-dimensional actions, the smaller for the discrete).

Self-explanation 8. Before reading on: the pathwise’s E[2(θ + ε)] = 2θ = 0 at the θ = 0 — is this the correct gradient of the E[Q]?

Answer. The E[Q] = E[(θ + ε)²] = θ² + 1 (the variance’s 1 is the constant), and the ∇_θ E[Q] = 2θ = 0 at the θ = 0. The pathwise’s E[2(θ + ε)] = 2θ matches the true gradient, and the unbiasedness is the correctness. The pathwise’s unbiasedness and the low variance are the two advantages, and the SAC’s actor uses the both.

WE-L.3.b — Completion problem

Input. The same 1D Gaussian, but the Q(a) = −(a − 1)² (the target a = 1), the θ = 0, the ε = 0.5.

Trace (final 25% as prompts).

1. The sample: the a = 0 + 0.5 = 0.5. ⟨your turn⟩
2. The ∇_a Q = −2(a − 1) = −2(0.5 − 1) = 1.0. ⟨your turn⟩
3. The pathwise gradient: the ∇_θ Q = ∇_a Q · ∇_θ a = 1.0 · 1 = 1.0. ⟨your turn⟩
4. The expectation: the E[1.0] = 1.0 (the same for the all the ε, since the ∇_a Q is the linear and the ∇_θ a = 1). ⟨your turn⟩

Answers.

1. a = 0.5. ✓
2. ∇_a Q = 1.0. ✓
3. ∇_θ Q = 1.0. ✓
4. The E[1.0] = 1.0. ✓

The lesson. The pathwise’s 1.0 is the gradient of the Q with respect to the θ, and the expectation is the true gradient. The pathwise’s value is the same for the all the ε (the linear Q and the linear a(θ)), and the variance is zero (the no randomness in the gradient). The pathwise’s zero variance is the best case, and the SAC’s actor uses the pathwise’s efficiency.

WE-L.3.c — Faded: the tanh squashing’s Jacobian

Setup. The same 1D Gaussian, the z = 0.5 (the pre-squashing), the a = tanh(0.5) ≈ 0.462, the da/dz = 1 − tanh²(0.5) ≈ 0.787.

Question. What is the log |da/dz|, and how does the squashing’s log-likelihood differ?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The log |da/dz| = log(0.787) ≈ −0.239. The squashed log-likelihood is log π(a) = log p(z) − log |da/dz| = log p(z) + 0.239. The unbounded’s is the log p(z) (the no correction), and the difference is the 0.239 (the Jacobian’s term).

The lesson. The tanh squashing’s Jacobian’s correction is the −log |da/dz|, and the omission biases the log-likelihood by the 0.239. The SAC’s standard includes it (the 1.2.3’s WE-L.4.c), and the unbounded is the simplification (the wrong for the bounded actions).

WE-L.3.d — Contrastive near-miss: detaching the wrong tensor

The actor’s loss has two differentiable inputs, and exactly one of them is detached. Naming which one is the whole exercise.

Correct solution. L_π = α.detach() · log π_θ(a|s) − Q_min(s, a_θ), with a_θ = tanh(μ_θ + σ_θ ε). The α is a scalar the actor does not own — its gradient belongs to alpha_opt — so it is detached. The Q_min is not detached: the actor’s gradient is the pathwise ∇_a Q_min · ∇_θ a_θ, and that path runs through Q_min.

Near-miss. L_π = α.detach() · log π − Q_min.detach(). This looks safer — “the critic is a target, so freeze it” — and it is wrong in the opposite direction from the one the phrase suggests. Detaching Q_min leaves the loss’s only surviving gradient term as α ∇_θ log π, which is the entropy term alone. The actor maximises entropy and ignores the reward.

Why the near-miss produces a plausible but incorrect number. It trains. The loss falls, log π rises, the policy becomes broad, and nothing raises. What fails is the objective: the actor never learns to reach. In the lesson’s lab at 8,000 steps the standard actor reaches −1.0889; the Q_min.detach() variant reaches −4.8900, 4.5× worse, with the entropy term running unopposed.

The discrepancy, stated. Q_min and Q_min.detach() differ by whether the pathwise gradient exists at all — not by its magnitude. Q_min carries ∂Q/∂a; Q_min.detach() carries nothing, so ∂L/∂θ loses the term the actor is supposed to learn from. The critic’s parameters φ are untouched either way, because critic_opt.zero_grad() clears them before their own update; the detach question is about the graph, not about φ.

What the lesson’s prose previously got wrong. Earlier drafts of this lesson stated the rule as Q_min.detach() and described the failure of not detaching as a “2× gradient” that “doubles”. Both halves are false: the un-detached Q_min is required, and the un-detached α — the only tensor whose detach actually matters — costs −1.3877 against −1.0889, a 27% return loss, not a 2× gradient blow-up.

Self-explanation 9. Before reading on: if the Q_min must stay attached, why does the α need .detach() at all? What goes wrong if the α’s gradient flows into the actor’s update?

Answer. The α is optimised by its own loss, L_α = −log α · (log π + H_target).detach(), against alpha_opt. If the actor’s loss also sends gradient into log_alpha, then log_alpha receives two updates per step from two objectives that disagree — the actor wants α large (entropy is a reward term in its loss, since +α log π is being maximised), and L_α wants α at the value that hits the target entropy. The measured cost is −1.3877 versus −1.0889; the mechanism is that α drifts up, over-weighting entropy for the same reason the near-miss above does.

Part D — The Twin Critics and the Target Networks

§L.4 — Objects: the overestimation bias, the clipped double-Q, and the soft update

The overestimation bias: the max over the noisy Q-estimates is biased upward — the E[max_i Q_i] ≥ max_i E[Q_i] (the Jensen’s inequality’s corollary for the convex max). The single critic’s target y = r + γ max_a Q(s', a) uses the max, and the bias accumulates over the training (the large Q → the large target → the larger Q), and the policy learns the wrong (the over-optimistic Q, the policy’s suboptimal).

The clipped double-Q (the TD3’s twin and the SAC’s standard):

The min of the two target critics is the pessimistic lower bound, and the overestimation’s bias is reduced (the min selects the less over-optimistic of the two, and the bias is the smaller than the single’s).

The target networks: the separate copies \barφ_1, \barφ_2 of the critics $φ_1$, $φ_2$, updated by the soft update \barφ_i ← τφ_i + (1 − τ)\barφ_i with the τ = 0.005. The target’s slow update stabilizes the training (the target doesn’t track the critic’s fast change, and the TD’s error is the smoother). The no target network makes the target track the critic’s every update, and the training diverges (the positive feedback’s loop).

§L.4.1 — Result: the clipped double-Q’s min reduces the overestimation’s bias, and the target networks’ soft update stabilizes the TD’s target — the two are the stability’s mechanisms, and the no either is the divergence’s source

Claim. The clipped double-Q’s min reduces the overestimation’s bias from the single critic’s O(ε) (the noise’s standard deviation) to the min’s O(ε/√2) (the two independent estimates’ min’s expected value), and the target networks’ soft update stabilizes the TD’s target (the target changes by the τ’s fraction per step, and the TD’s error’s variance is the smaller). The two mechanisms are the stability’s key, and the no either causes the training’s divergence.

Proof sketch. The max’s bias is the $E[max(Q_1$, $Q_2$)] − $max(E[Q_1$], $E[Q_2$]), and for the independent Gaussian noise, the bias is the σ/√π ≈ 0.564σ (the two’s max), and the min’s bias is the −σ/√π (the two’s min). The min’s bias is the negative (the underestimation), which is the less harmful (the conservative policy) than the max’s positive (the overestimation’s optimism). The soft update’s stabilization is the small τ’s slow change — the target’s variance is the τ’s fraction of the critic’s, and the TD’s error’s variance is the smaller.

Corollary. The twin critics’ cost is the two networks’ computation (the 2× the critic’s), and the benefit is the stability (the overestimation’s reduction). The TD3 and the SAC both use the twin, and the single critic is the simplification (the fast but unstable).

Corollary. The target networks’ soft update’s τ = 0.005 is the slow (the 0.5% per step, the ~200 steps’ half-life), and the τ’s choice is the stability vs. the adaptation (the small τ is the stable but slow, the large is the fast but unstable). The 0.005 is the empirical balance.

§L.4.2 — Worked-example ladder

WE-L.4.a — Fully worked: the overestimation’s bias for two critics

Input. The two critics $Q_1$ = 1.0 + $ε_1$ and $Q_2$ = 1.0 + $ε_2$ with the $ε_1$, $ε_2$ ~ N(0, 0.1²), the true Q = 1.0.

Trace (the max’s bias).

1. The $E[max(Q_1$, $Q_2$)] = 1.0 + 0.1·√(1/π) ≈ 1.0 + 0.056 (the two Gaussian’s max’s expected value).
2. The bias: the 1.056 − 1.0 = 0.056 (the positive, the overestimation).

Trace (the min’s bias).

3. The $E[min(Q_1$, $Q_2$)] = 1.0 − 0.056 (the two’s min’s expected value).
4. The bias: the −0.056 (the negative, the underestimation).

Output and interpretation. The max’s +0.056 and the min’s −0.056 are the opposite biases — the max overestimates and the min underestimates. The min’s negative bias is the less harmful (the conservative policy doesn’t over-optimistically chase the wrong actions), and the SAC’s min is the standard (the TD3’s clipped double-Q).

Self-explanation 10. Before reading on: the min’s -0.056 bias — is the negative bias the always better than the positive?

Answer. Not always — the negative bias makes the policy conservative (the under-optimistic Q), and the policy may miss the good actions (the smaller Q’s values). The trade-off is the bias’s direction — the positive is the over-optimism (the large Q, the risky policy), the negative is the under-optimism (the small Q, the conservative policy). The SAC chooses the negative because the over-optimism’s divergence is the worse (the positive feedback’s loop), and the under-optimism is the safer (the suboptimal but stable).

WE-L.4.b — Completion problem

Input. The same two critics, but the three critics (the twin is the two).

Trace (final 25% as prompts).

1. The three critics’ min’s bias: the $E[min(Q_1$, $Q_2$, $Q_3$)] = 1.0 + 0.1·(the three's min's standard) ≈ 1.0 − 0.1·1.03 = 0.897. ⟨your turn⟩
2. The three’s min is the more negative (the larger bias). ⟨your turn⟩
3. The two’s min is the standard (the balanced). ⟨your turn⟩

Answers.

1. The $min(Q_1$, $Q_2$, $Q_3$) ≈ 0.897 (the three’s min’s expected). ✓
2. The larger negative bias (-0.103 vs. the two’s -0.056). ✓
3. The two is the standard. ✓

The lesson. The three’s min is the more negative (the larger underestimation), and the two’s is the balanced. The number of the critics’ trade-off is the bias vs. the computation — the more critics is the larger bias (the underestimation), and the two is the SAC’s standard (the good balance).

WE-L.4.c — Faded: the soft update’s trajectory

Setup. The target \barφ starts at the 0, the critic φ is at the 1, the τ = 0.005.

Question. What is the \barφ after the 100 steps? After the 1000 steps?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The soft update \barφ ← τφ + (1 − τ)\barφ = 0.005·1 + 0.995·\barφ. The trajectory is the \barφ_k = 1 − (0.995)^k (the exponential’s approach to the 1). The \barφ_{100} = 1 − 0.995^{100} ≈ 1 − 0.606 = 0.394, and the \barφ_{1000} = 1 − 0.995^{1000} ≈ 1 − 0.0067 = 0.993. The half-life is the log(0.5)/log(0.995) ≈ 138 steps.

The lesson. The soft update’s τ = 0.005 is the slow (the 138 steps’ half-life), and the target converges to the critic’s value over the ~1000 steps. The slow update is the stability’s source, and the fast critic’s change is the smoothed by the target’s slow tracking.

WE-L.4.d — Contrastive near-miss: the no target network

Correct solution. The target networks \barφ with the soft update.

Near-miss. The no target network (the critic φ’s every update is the target’s direct source).

Why the near-miss produces a plausible but incorrect number. The no target network makes the TD’s target track the critic’s every update, and the positive feedback’s loop (the larger Q → the larger target → the larger Q) diverges. The symptom is the Q’s blow-up (the NaN or the huge values), and the policy’s learning stops. The fix is the target networks (the slow update), which breaks the feedback’s loop.

The discrepancy, stated. The no target’s Q(s', a') and the target’s Q̄(s', a') differ by the update’s speed — the no target’s fast (the every step’s update), the target’s slow (the 0.5% per step). The slow update is the stability’s source, and the no is the divergence’s source.

Self-explanation 11. Before reading on: the no target’s divergence — if the learning rate is small (the 1e-5), does the no target work?

Answer. Partially — the small learning rate reduces the critic’s update’s speed, and the feedback’s loop is the slower (the less divergence). But the target’s slow update is the different mechanism (the target’s own stability), and the no target with the small rate is the workaround (the slower, the less stable). The target is the correct (the standard), and the no target is the simplification (the rarely works).

Part E — Failure Taxonomy, Lab, Cheat Sheet, Exercises, Exit Test

§L.5 — Failure taxonomy

§L.6 — The lab

§L.6.1 — Task

Implement SAC for the ArmEnv, with the twin critics, the target critics, the replay buffer, the reparameterization trick, and the automatic temperature; verify the convergence and the sample efficiency against PPO; break the SAC five ways and diagnose.

§L.6.2 — Conditions

Python 3.10+, PyTorch 2.2+, CPU. Seed 42, dtype = float64. Runtime under 15 minutes. The ArmEnv with the 2D position, the tanh-squashed Gaussian actor, the twin critics, the 10⁵ buffer (the reduced for the lab’s runtime), the lr = 3e-4, the γ = 0.99, the τ = 0.005, the B = 256, the $α_init$ = 0.2, the \bar H = −2.

§L.6.3 — Controls held fixed

The seed, dtype, the ArmEnv, the policy’s architecture, the hyperparameters are fixed. The variants are the no target, the single critic, the fixed α, the small buffer, and the no detach.

§L.6.4 — Metrics table

§L.6.5 — Pre-registered hypotheses

Each variant trains for 20,000 env steps and reports the mean of its last ten episode returns. These are the measured values.

• H1. The standard SAC converges to return > -0.5 within ~5,000 env steps. Measured: −1.3209 after 20,000 steps. Refuted. Standard SAC does not clear the bar on this budget. Two things explain it and both are properties of the lab, not of SAC: the episode is 10 steps of 0.1 · a movement toward a target at distance ≈ 0.54, so the achievable return is bounded near −0.5 only for a near-optimal controller; and 20,000 steps is roughly 2,000 episodes, which is a short run for SAC’s two-timescale actor-critic. The honest reading is that the bar was set for a longer run.
• H2. The no-target critic diverges — Q huge or NaN — within ~500 steps. Measured: final Q −1.4594, return −1.1747. Refuted. Removing the target network did not diverge. −1.4594 is an ordinary Q magnitude, and the no-target return is better than standard’s −1.3209. The mechanism is real in general but needs the critic to bootstrap off its own moving estimate for long enough; with a 256-batch, γ = 0.99 soft-update regime on a 10-step task, the target network’s τ = 0.005 smoothing is nearly redundant with the short horizon.
• H3. The single critic’s Q overestimates and the policy is suboptimal. Measured: single-critic return −1.1809, better than standard’s −1.3209. Refuted on the policy claim. The Q mean is −3.1585, which is lower than standard’s, so there is no visible overestimation either. Same cause as H2: the overestimation bias needs a harder credit-assignment problem than a dense-reward 2-D point mass.
• H4. The fixed α = 0.2 underperforms the automatic α by ~10%. Measured: fixed −2.8913 against automatic −1.3209 — 2.19× worse, not 10% worse. Holds, and far more strongly than predicted. The automatic temperature ends at α = 0.0155, so the fixed value is 13× too large; the policy stays near-uniform and never commits to a reaching action. This is the one ablation in the lab where the mechanism fires at full strength.
• H5. Removing the $Q_min$ detach doubles the actor’s gradient and makes training oscillate. Not implemented, and the claim is wrong. The lab’s standard run never detaches $Q_min$, because it must not: the actor’s gradient is the pathwise ∇_a $Q_min$ · ∇_θ a_θ, and detaching $Q_min$ deletes it. Measured directly at 8,000 steps, $Q_min$.detach() reaches −4.8900 against −1.0889. The detach that matters is on α; dropping it costs −1.3877 against −1.0889. See WE-L.3.d.

The summary of this block is uncomfortable and should be reported as such: three of five hypotheses were refuted, one held far more strongly than stated, and one described a bug that is not a bug. The pattern is consistent — SAC’s stabilisers are variance-reduction devices, and a dense-reward 2-D task does not generate the variance they suppress. The temperature is the exception because it is not a variance device: a badly scaled α changes the objective’s balance, not its noise, so it bites regardless of task difficulty.

§L.6.6 — The break-it sequence

1. The no target (predicted loud, measured silent). The no target network. Predicted symptom: the Q diverges within ~500 steps. Actual: Q ends at −1.4594, the return at −1.1747. Silent — no NaN, no blow-up, and the return is slightly better than standard.
2. The single critic (silent). The single Q. Predicted symptom: the Q overestimates and the policy is suboptimal. Actual: Q mean −3.1585 (no overestimation) and return −1.1809 (better than standard). Silent, and the predicted damage is absent.
3. The fixed α (silent, and expensive). The fixed α = 0.2. Predicted symptom: the return is lower. Actual: −2.8913 against −1.3209. Silent. The best-behaved break-it in the lesson: the predicted sign is right and the magnitude is 22× the predicted 10%.
4. The small buffer (silent). The buffer size 10⁴ (the stale data). Predicted symptom: the sample efficiency drops. Actual: not implemented — the lab builds a 100,000-capacity buffer and never varies it.
5. The detach on $Q_min$ (silent, catastrophic). Predicted symptom (as previously written): the gradient doubles. Actual: the gradient is deleted — −4.8900 against −1.0889 at 8,000 steps, a 4.5× return loss, because the only surviving term is the entropy bonus. Silent in the sense that it trains and its loss falls; what fails is the objective.

§L.6.7 — Acceptance criteria

§L.6.8 — What good looks like

A strong submission has five properties:

1. The SAC is implemented with the twin critics, the target critics, the soft update, the replay buffer, the reparameterization trick (with the tanh Jacobian’s correction), and the automatic α, and the sample efficiency is measured.
2. **The twin’s min is demonstrated (the Q’s less over-optimistic than the single), and the symptom’s comparison is shown.
3. **The automatic α’s adaptation is shown (the α’s trajectory), and the fixed α’s underperformance is measured.
4. **The break-its are predicted with the symptoms (the divergence, the overestimation, the suboptimal, the stale, the oscillation) before running.
5. **The sample efficiency’s comparison against the PPO (1.2.4) is measured (the SAC’s fewer env-steps), and the trade-off (the SAC’s replay buffer’s memory, the PPO’s simplicity) is discussed.

§L.6.9 — Runnable lab code

# lab_1_2_5.py — SAC with the twin critics, the target, the replay buffer, and the break-its
# Requires: Python 3.10+, PyTorch 2.2+

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import copy
from torch.distributions import Normal

torch.manual_seed(42); dtype, device = torch.float64, torch.device("cpu")

# --- The SimpleArmEnv (2D position, episodic) --------------------------
class SimpleArmEnv:
    def __init__(self, max_steps=10):
        self.$p_target$ = torch.tensor([0.5, 0.2], dtype=dtype)
        self.max_steps = max_steps
        self.state_dim = 2; self.action_dim = 2
    def reset(self):
        self.pos = torch.randn(2, dtype=dtype) * 0.1
        self.step_count = 0
        return self.pos.clone()
    def step(self, a):
        self.pos = self.pos + a * 0.1
        dist = torch.norm(self.pos - self.$p_target$).item()
        r = -dist; self.step_count += 1
        terminated = dist < 0.05
        truncated = self.step_count >= self.max_steps
        return self.pos.clone(), r, terminated, truncated

# --- The Actor (tanh-squashed Gaussian) ------------------------------
class Actor(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=64):
        super().__init__()
        self.fc = nn.Sequential(nn.Linear(state_dim, hidden), nn.ReLU(), nn.Linear(hidden, hidden), nn.ReLU())
        self.mean = nn.Linear(hidden, action_dim)
        self.log_std = nn.Linear(hidden, action_dim)
        self.action_dim = action_dim
    def forward(self, s):
        h = self.fc(s)
        mu = self.mean(h)
        log_std = self.log_std(h).clamp(-20, 2)
        std = log_std.exp()
        dist = Normal(mu, std)
        z = dist.rsample()
        a = torch.tanh(z)
        log_prob = dist.log_prob(z) - torch.log(1 - a.pow(2) + 1e-6)
        return a, log_prob.sum(-1)

# --- The twin critics ------------------------------------------------
class TwinCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=64):
        super().__init__()
        self.q1 = nn.Sequential(nn.Linear(state_dim + action_dim, hidden), nn.ReLU(), nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, 1))
        self.q2 = nn.Sequential(nn.Linear(state_dim + action_dim, hidden), nn.ReLU(), nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, 1))
    def forward(self, s, a):
        sa = torch.cat([s, a], dim=-1)
        return self.q1(sa).squeeze(-1), self.q2(sa).squeeze(-1)

# --- The replay buffer ----------------------------------------------
class ReplayBuffer:
    def __init__(self, capacity, state_dim, action_dim):
        self.capacity = capacity; self.ptr = 0; self.size = 0
        self.states = torch.zeros(capacity, state_dim, dtype=dtype)
        self.actions = torch.zeros(capacity, action_dim, dtype=dtype)
        self.rewards = torch.zeros(capacity, dtype=dtype)
        self.next_states = torch.zeros(capacity, state_dim, dtype=dtype)
        self.terminateds = torch.zeros(capacity, dtype=torch.bool)
        self.truncateds = torch.zeros(capacity, dtype=torch.bool)
    def add(self, s, a, r, $s_next$, term, trunc):
        self.states[self.ptr] = s; self.actions[self.ptr] = a; self.rewards[self.ptr] = r
        self.next_states[self.ptr] = $s_next$; self.terminateds[self.ptr] = term; self.truncateds[self.ptr] = trunc
        self.ptr = (self.ptr + 1) % self.capacity
        self.size = min(self.size + 1, self.capacity)
    def sample(self, batch_size):
        idx = torch.randint(0, self.size, (batch_size,))
        return (self.states[idx], self.actions[idx], self.rewards[idx], self.next_states[idx],
                self.terminateds[idx], self.truncateds[idx])

# --- SAC hyperparameters -------------------------------------------
LR = 3e-4; GAMMA = 0.99; TAU = 0.005; BATCH_SIZE = 256; HIDDEN = 64
BUFFER_SIZE = 100_000; ALPHA_INIT = 0.2; TARGET_ENTROPY = -2
HIDDEN = 64; MAX_STEPS_PER_EP = 10

def sac_train(variant="standard", $n_steps=20_000$):
    env = SimpleArmEnv()
    actor = Actor(2, 2, HIDDEN).to(device, dtype)
    critic = TwinCritic(2, 2, HIDDEN).to(device, dtype)
    critic_target = copy.deepcopy(critic).to(device, dtype)
    actor_opt = torch.optim.Adam(actor.parameters(), lr=LR)
    critic_opt = torch.optim.Adam(critic.parameters(), lr=LR)
    log_alpha = torch.tensor(np.log(ALPHA_INIT), dtype=dtype, requires_grad=True, device=device)
    alpha_opt = torch.optim.Adam([log_alpha], lr=LR)
    buffer = ReplayBuffer(BUFFER_SIZE, 2, 2)
    returns = []; alphas = []; $q_means$ = []
    s = env.reset()
    ep_return = 0.0
    for step in $range(n_steps$):
        with torch.no_grad():
            a, _ = actor(s)
            $a_np$ = a.numpy()
        $s_next$, r, term, trunc = env.$step(a_np$)
        buffer.add(s, a, torch.tensor(r, dtype=dtype), $s_next$, term, trunc)
        ep_return += r
        if term or trunc:
            returns.append(ep_return); ep_return = 0.0
            s = env.reset()
        else:
            s = $s_next$
        if buffer.size < BATCH_SIZE:
            continue
        # Sample from the buffer
        sb, ab, rb, $s_nextb$, termb, truncb = buffer.sample(BATCH_SIZE)
        # Actor update
        $a_new$, log_prob = actor(sb)
        q1, q2 = critic(sb, $a_new$)
        $q_min$ = torch.min(q1, q2)
        alpha = log_alpha.exp()
        actor_loss = (alpha.detach() * log_prob - $q_min$).mean()
        actor_opt.zero_grad(); actor_loss.backward(); actor_opt.step()
        # Critic update
        with torch.no_grad():
            $a_next$, log_prob_next = $actor(s_nextb$)
            q1_t, q2_t = $critic_target(s_nextb$, $a_next$)
            $q_min_t$ = torch.min(q1_t, q2_t)
            target_q = rb + GAMMA * (1 - termb.float()) * ($q_min_t$ - alpha * log_prob_next)
        q1_pred, q2_pred = critic(sb, ab)
        if variant == "single_critic":
            critic_loss = F.mse_loss(q1_pred, target_q)
        else:
            critic_loss = F.mse_loss(q1_pred, target_q) + F.mse_loss(q2_pred, target_q)
        critic_opt.zero_grad(); critic_loss.backward(); critic_opt.step()
        # Alpha update
        if variant == "fixed_alpha":
            pass  # the α is fixed
        else:
            alpha_loss = -(log_alpha * (log_prob + TARGET_ENTROPY).detach()).mean()
            alpha_opt.zero_grad(); alpha_loss.backward(); alpha_opt.step()
        # Soft target update
        if variant != "no_target":
            with torch.no_grad():
                for p, $p_t$ in zip(critic.parameters(), critic_target.parameters()):
                    $p_t$.data.mul_(1 - TAU); $p_t$.data.add_(TAU * p.data)
        if step % 1000 == 0 and len(returns) > 0:
            alphas.append(log_alpha.exp().item())
            with torch.no_grad():
                q1_e, q2_e = critic(torch.randn(100, 2, dtype=dtype), torch.randn(100, 2, dtype=dtype))
                $q_means$.append(((q1_e + q2_e) / 2).mean().item())
    return returns, alphas, $q_means$

print("=== Standard SAC ===")
$r_std$, $a_std$, $q_std$ = sac_train("standard", $n_steps=20_000$)
print(f"final return (last 10 eps): {np.$mean(r_std[-10$:]):.4f}")
print(f"final alpha: {a_std[-1]:.4f}")

print("\n=== Single critic ===")
$r_sc$, _, $q_sc$ = sac_train("single_critic", $n_steps=20_000$)
print(f"final return: {np.$mean(r_sc[-10$:]):.4f}")
print(f"final Q mean: {q_sc[-1]:.4f}")

print("\n=== Fixed alpha ===")
$r_fa$, _, _ = sac_train("fixed_alpha", $n_steps=20_000$)
print(f"final return: {np.$mean(r_fa[-10$:]):.4f}")

print("\n=== No target ===")
try:
    $r_nt$, _, $q_nt$ = sac_train("no_target", $n_steps=20_000$)
    print(f"final return: {np.$mean(r_nt[-10$:]):.4f}, final Q: {q_nt[-1]:.4f}")
except Exception as e:
    print(f"raised: {e}")

# --- Summary -------------------------------------------------------
print("\n--- lab 1.2.5 summary ---")
print(f"H1: standard final return = {np.$mean(r_std[-10$:]):.4f} (target > -0.5)")
print(f"H2: no-target Q = {q_nt[-1] if '$q_nt$' in dir() else 'N/A'} (should diverge)")
print(f"H3: single-critic return = {np.$mean(r_sc[-10$:]):.4f} (should be suboptimal)")
print(f"H4: fixed-alpha return = {np.$mean(r_fa[-10$:]):.4f} vs auto {np.$mean(r_std[-10$:]):.4f}")
print(f"H5: no-detach would double the gradient (see break-it description)")

§L.7 — The cheat sheet

§L.8 — Exercises with solutions

E 1.2.5.1 — The Boltzmann policy (numerical). The Q = [1.0, 0.5], the α = 0.1. Compute the π.

S 1.2.5.1. The exp(10) ≈ 22026, the exp(5) ≈ 148, the Z ≈ 22174, the π ≈ [0.993, 0.007]. The lesson: the small α sharpens the Boltzmann to the greedy.

E 1.2.5.2 — The soft value (numerical). The same Q, the α = 0.5. Compute the $V_soft$.

S 1.2.5.2. The π ≈ [0.731, 0.269], the $V_soft$ ≈ 1.157 (from WE-L.1.a). The lesson: the soft value exceeds the greedy 1.0 by the entropy’s bonus.

E 1.2.5.3 — The critic’s target (numerical). The r = 1.0, the min Q = 1.5, the α log π = −0.1, the γ = 0.99. Compute the target.

S 1.2.5.3. The y = 1.0 + 0.99·(1.5 − (−0.1)) = 1.0 + 0.99·1.6 = 2.584. The lesson: the twin’s min and the entropy’s subtraction, and the target is the sum.

E 1.2.5.4 — The pathwise’s variance (conceptual). The Q = a², the μ = θ, the σ = 1. Compare the pathwise’s variance and the score function’s.

S 1.2.5.4. The pathwise Var[2(θ+ε)] = 4, the score function Var[a³] = 15. The ratio 3.75, the pathwise’s lower variance. The lesson: the pathwise’s gradient is the low variance (the 3.75× fewer samples).

E 1.2.5.5 — The overestimation’s bias (numerical). The $Q_1$ = 1 + $ε_1$, the $Q_2$ = 1 + $ε_2$, the ε ~ N(0, 0.1²). What is the max’s bias? The min’s?

S 1.2.5.5. The max’s +0.056, the min’s −0.056. The lesson: the max overestimates, the min underestimates, and the SAC uses the min (the conservative).

E 1.2.5.6 — The target’s soft update (numerical). The Q = 1.0, the Q̄ = 0, the τ = 0.005. What is the Q̄ after the 138 steps (the half-life)?

S 1.2.5.6. The Q̄_{138} = 1 − 0.995^{138} ≈ 0.5 (the half of the 1.0). The lesson: the τ = 0.005 is the slow (the 138 steps’ half-life), and the target tracks the critic slowly.

E 1.2.5.7 — The no target’s divergence (diagnostic). The Q blows up (NaN) within the 500 steps. What is the cause?

S 1.2.5.7. The no target network (the TD’s target tracks the critic’s every update, the positive feedback’s loop). The fix is the target network’s soft update. The lesson: the target is the stability’s mechanism, and the no target is the divergence’s source.

E 1.2.5.8 — The automatic α (constructive, transfer probe). Write the automatic α’s loss and the gradient’s direction.

S 1.2.5.8.

$L_alpha$ = -alpha * (log_pi + target_entropy).detach().mean()
# The gradient: dL/d(alpha) = -(log_pi + target_entropy)
# If log_pi + target_entropy > 0 (the entropy is below the target),
# the gradient is negative, and the alpha increases (more exploration).

The loss’s gradient increases the α when the entropy is below the target (the more exploration), and decreases it when above. The lesson: the automatic α is the constrained optimization’s Lagrangian, and its gradient adapts the exploration to the task’s needs.

§L.9 — Exit test

You are ready for Lesson 2.1.1 (Robotic Perception) when you can, without notes:

1. State the maximum-entropy objective and the soft value functions.
2. State the soft Bellman equations and the Boltzmann policy.
3. State the α’s role and the automatic tuning’s gradient.
4. State the twin critics and the clipped double-Q’s overestimation’s mitigation.
5. State the target critics and the soft update’s τ = 0.005.
6. State the reparameterization trick and the pathwise’s low variance.
7. State the tanh squashing’s Jacobian’s correction.
8. State which tensor the actor’s loss detaches — the α — and why $Q_min$ must stay attached.
9. State the SAC’s hyperparameters $`lr = 3e-4`, `τ = 0.005`, `B = 256`, `α_init = 0.2`, `\bar H = −|A|`$.
10. State the SAC’s off-policy vs. the PPO’s on-policy.
11. Diagnose the fixed α’s suboptimality (−2.8913 vs −1.3209), and state honestly that the no-target, single-critic and no-smoothing ablations did not produce their predicted symptoms on this task.
12. State the sample efficiency’s advantage and the cost (the replay buffer’s memory).

If any item is not answerable without notes, return to the section that introduced it: 1 to §L.1, 2 to §L.1.1, 3 to §L.2, 4 to §L.4, 5 to §L.4, 6 to §L.3, 7 to §L.3.1, 8 to §L.3.1, 9 to §L.2, 10 to §L.2.1, 11 to §L.5, 12 to §L.2.1.

End of Lesson 1.2.5. The learner has now completed the off-policy RL’s foundation: the maximum-entropy objective, the soft value functions, the reparameterization trick, the twin critics, the target networks, the replay buffer, and the automatic temperature — the SAC’s algorithm and its stability’s mechanisms. The next module, 2.1 — Robotic Perception, moves to the vision — the camera models, the object detection, the 6D pose estimation, and the visual servoing — the perception’s half of the perceive-reason-act loop that the RL policies consume.