X2.2.1

Lesson 2.2.1 — TD3: The Deterministic Policy Gradient, Twin Critics, and Delayed Updates

Week 11 · Prerequisites: 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.1.1.

Prior results used. This lesson uses 1.2.2’s value functions and Bellman equations, 1.2.3’s policy gradient theorem and score function, 1.2.4’s clipped objective and target networks, 1.2.5’s twin critics, target critics, and replay buffer, and 1.1.1’s geometric Jacobian for the action-space interpretation. Each is restated at the point of use.

Hands forward. This lesson hands forward one capability: the learner can derive the deterministic policy gradient theorem, implement TD3 with twin critics, target policy smoothing, and delayed updates, distinguish TD3’s deterministic actor from SAC’s stochastic actor, choose between them by the task’s exploration requirements, and diagnose the four TD3-specific bugs (no exploration noise, missing smoothing, missing delay, wrong action gradient) from their learning symptoms — the capability that 2.2.2 (model-based RL), 2.2.3 (offline RL), and the manipulation’s fine-tuning in 2.3.2 all assume.

Running case. The same 6-DOF MuJoCo arm from 1.1.1–2.1.4, in the ArmEnv, now with TD3 as the learning algorithm: the actor μ_θ(s) ∈ ℝ⁶ is deterministic, the twin critics $Q_{φ_1}(s$, a) and $Q_{φ_2}(s$, a) are the same as SAC’s, the target critics $Q_{\barφ_1}$ and $Q_{\barφ_2}$ are the same, and the exploration comes from adding Gaussian noise to the actor’s output at rollout time — not from a stochastic policy. The delayed policy update and the target policy smoothing are the two extensions that distinguish TD3 from DDPG.

§L.0 — Lesson map, objectives, and misconceptions

§L.0.1 — Lesson map and word budget

§L.0.2 — Objectives

By the end of this lesson you will be able to do eight things:

1. State the deterministic policy gradient theorem and explain why it is off-policy.
2. Implement DDPG’s architecture and state its three weaknesses (overestimation, no target smoothing, no delay).
3. Derive the clipped double-Q target and state why the min reduces overestimation.
4. Implement target policy smoothing and state why it regularizes the critic against sharp Q peaks.
5. Implement the delayed policy update and state why it stabilizes the training.
6. Compare TD3, SAC, and PPO, and state the task each is best suited for.
7. Diagnose the four TD3 bugs from their learning symptoms.
8. Produce a working TD3 implementation for the ArmEnv, with the twin critics, the target smoothing, the delayed update, and the comparison against SAC.

§L.0.3 — Misconception inventory

Each misconception is displaced in the section named, and each is tagged in the concept check that follows.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

This is the twenty-sixth lesson. The warm-up draws from 2.1.4 (two steps back), 2.1.3 (three), 2.1.2 (four), and 2.1.1 (five).

Warm-up 1 (from 2.1.4, the ICP and Kabsch). State the ICP’s alternation and the Kabsch’s closed form. Answer: ICP alternates correspondence (nearest-neighbor) and alignment (Kabsch’s SVD-based closed form), converging locally. How this is used: §L.3’s delayed update mirrors this alternation — the critic gets d steps of training before the actor updates once, analogous to letting correspondence stabilize before the alignment step.

Warm-up 2 (from 2.1.3, the visual servo’s gain). State the IBVS’s $v_c$ = -λ L̂⁺ e and the gain’s role. Answer: the gain λ controls the convergence rate; too large diverges, too small slows. How this is used: §L.3’s target smoothing’s σ and clip play the same role — the clip bounds the noise (like the gain’s bound), the σ sets the regularization’s strength.

Warm-up 3 (from 2.1.2, the PnP). State the PnP’s minimal (4 points) and overdetermined (n > 4) cases. Answer: P3P/P4P minimal gives up to 4 solutions; overdetermined is the least-squares. How this is used: §L.1’s deterministic policy gradient is a single-point evaluation of ∇_a Q; the stochastic policy gradient is an expectation over the action’s distribution. The two estimators are different in kind.

Warm-up 4 (from 2.1.1, the projection and depth). State the s = Z scale and the Z = 0 singularity. Answer: the projection’s scale is the depth; Z = 0 is the singularity. How this is used: §L.4’s TD3 vs SAC comparison turns on the precision of the action — TD3’s deterministic action is the point, SAC’s stochastic is the distribution. The precision task (insertion) favors TD3; the exploration task favors SAC.

Part A — The Deterministic Policy Gradient and DDPG

§L.1 — Objects: the deterministic actor, the critic, and the exploration

The deterministic policy is a = μ_θ(s), a single action per state, parameterized by a neural network with parameters θ. Unlike the stochastic policy π_θ(a|s), the deterministic policy has no distribution — the action is a function of the state.

The deterministic policy gradient theorem (Silver et al. 2014, tier A, S):

∇_θ J(θ) = $E_{s ~ D}$ [ ∇_a Q_φ(s, a) |_{a = μ_θ(s)} · ∇_θ μ_θ(s) ]

where Q_φ(s, a) is the critic, D is the replay buffer, and the chain rule combines the critic’s action gradient ∇_a Q with the actor’s parameter gradient ∇_θ μ.

Three key properties:

1. Off-policy — D is the buffer, and the critic Q_φ is learned from the buffer’s transitions, not the policy’s samples. This is the fundamental difference from the stochastic policy gradient, which requires on-policy samples (1.2.3).
2. The actor’s gradient flows through the critic — ∇_a Q is the critic’s judgment of which action is better; ∇_θ μ is the actor’s parameterization. The chain rule combines them, and the actor improves toward the greedy action argmax_a Q.
3. The critic must be accurate — ∇_a Q is only useful if the critic is a good estimate. A bad critic gives a bad gradient, and the actor moves in the wrong direction.

DDPG (Lillicrap et al. 2016, tier A, S) is the off-policy actor-critic algorithm implementing the deterministic policy gradient:

• Actor μ_θ(s) — the deterministic policy, updated by ∇_θ μ · ∇_a Q.
• Critic Q_φ(s, a) — the Q-network, updated by the Bellman residual (Q(s, a) − y)² with the target y = r + γ $Q_{\barφ}$$s', μ_{\barθ}(s')$.
• Target actor $μ_{\barθ}$ and target critic $Q_{\barφ}$ — the slowly-updated copies (the τ = 0.005 soft update from 1.2.5).
• Replay buffer D — the past transitions.

The exploration is the behavior policy $a_t$ = $μ_θ(s_t$) + $ε_t$ with $ε_t$ ~ N(0, σ²) Gaussian noise, or Ornstein–Uhlenbeck temporally-correlated noise.

DDPG’s three limitations:

• Brittleness — the actor’s gradient depends on the critic’s accuracy, and the critic’s estimation errors propagate to the actor. The training is sensitive to hyperparameters and diverges on many tasks.
• Overestimation — the target $Q_{\barφ}$$s', μ_{\barθ}(s')$ has the same overestimation bias as the single critic in 1.2.5 (M4).
• No target smoothing — the target’s $μ_{\barθ}(s$') is deterministic, and the critic sees the same target every step, overfitting to the actor’s current action.

These three are the extensions TD3 adds: twin critics (Part B), delayed updates, and target smoothing (Part C).

§L.1.1 — Result: the deterministic policy gradient is off-policy, critic-dependent, and distinct from the stochastic theorem

Claim. The deterministic policy gradient theorem uses the critic’s ∇_a Q (not the score function’s ∇ log π), and it is off-policy (the buffer’s D). The gradient’s quality depends on the critic’s accuracy, and DDPG’s three limitations are fixed by TD3’s extensions.

Proof sketch. The stochastic policy gradient ∇_θ J = $E_{a~π}$[∇_θ log π · Q] requires the score function, which is the distribution’s derivative. The deterministic a = μ_θ(s) has no distribution, so the score function is undefined. The theorem uses ∇_a Q instead, which is finite at the deterministic action. The off-policy property follows because D’s distribution is not the policy’s, so the theorem’s expectation is over the buffer.

Corollary. The deterministic gradient is the chain rule of the critic’s ∇_a Q and the actor’s ∇_θ μ. A wrong critic gives a wrong gradient, and the actor diverges (the “wrong action gradient” bug).

Corollary. The exploration’s behavior policy is μ + ε, and the gap between behavior and policy is what makes the buffer off-policy. Without noise, the actor never explores (the “no exploration noise” bug).

§L.1.2 — Worked-example ladder

WE-L.1.a — Fully worked: the deterministic policy gradient for a 1D actor

Input. A 1D actor μ_θ(s) = θ · s with θ = 0.5 and s = 2 (so a = 1.0), a critic Q_φ(s, a) = −(a − a*)² + C with a* = 1.5 and C a constant.

Trace (the critic’s action gradient).

1. ∇_a Q = −2(a − a*) = −2(1.0 − 1.5) = 1.0.
2. ∇_θ μ = s = 2.
3. ∇_θ J = ∇_a Q · ∇_θ μ = 1.0 · 2 = 2.0.

Trace (the actor’s update).

4. θ ← θ + α ∇_θ J = 0.5 + 0.1 · 2.0 = 0.7.
5. New action μ_θ(s) = 0.7 · 2 = 1.4 (closer to the optimal 1.5).

Output and interpretation. The actor’s gradient 2.0 increases θ from 0.5 to 0.7, and the action moves from 1.0 to 1.4 — toward the optimal 1.5. The critic’s ∇_a Q = 1.0 is the direction to increase Q; ∇_θ μ = 2 is the rate of the actor’s action change. The chain rule combines them.

Self-explanation 1. The θ moves from 0.5 to 0.7, and the action from 1.0 to 1.4 (not all the way to 1.5). Why not one step to the optimum?

Answer. The step is the gradient’s direction times the learning rate α = 0.1, and the gradient’s magnitude 2.0. The step is 0.2 in θ and 0.4 in the action. Multiple steps iterate toward the optimum; convergence is asymptotic, not one-step.

Self-explanation 2. At a = 2.0 (past the optimum 1.5), what is ∇_a Q?

Answer. ∇_a Q = −2(2.0 − 1.5) = −1.0, negative — the Q decreases with a at this point. The actor’s gradient becomes −2.0, and θ decreases, reducing the action toward the optimum.

WE-L.1.b — Completion problem

Input. Same actor, θ = 0.5, s = 2, Q = −(a − 1.5)², a = 1.0.

Trace (final 25% as prompts).

1. ∇_a Q = −2(a − 1.5) = 1.0. ⟨verify⟩
2. ∇_θ μ = s = 2. ⟨verify⟩
3. ∇_θ J = 2.0. ⟨verify⟩
4. $θ_new$ = 0.5 + 0.1 · 2.0 = 0.7. ⟨verify⟩

Answers. All correct.

The lesson. The chain rule ∇_a Q · ∇_θ μ is the deterministic policy gradient, with the critic providing the action’s gradient and the actor providing the parameter’s gradient.

WE-L.1.c — Faded: DDPG’s target computation

Setup. Same actor, s = 2, a = μ_θ(s), next state s' = 3, reward r = 1.0, discount γ = 0.99, target actor $μ_{\barθ}(s$') = 1.5, target critic $Q_{\barφ}(s$', 1.5) = 0.5.

Question. What is DDPG’s target y?

You must produce the intermediate steps. The setup and the answer are given.

Answer. y = r + γ · $Q_{\barφ}$$s', μ_{\barθ}(s')$ = 1.0 + 0.99 · 0.5 = 1.495. The target 1.495 is what the critic regresses toward.

The lesson. DDPG’s target uses the target networks (soft-updated), not the online networks.

WE-L.1.d — Contrastive near-miss: the stochastic policy gradient

Correct solution. The deterministic policy gradient ∇_a Q · ∇_θ μ.

Near-miss. The stochastic policy gradient ∇_θ log π · Q.

Why the near-miss produces a plausible but incorrect number. The stochastic theorem’s ∇ log π is undefined for the deterministic actor (no distribution). If the learner uses the stochastic formula on the deterministic actor, ∇ log π is zero (the delta’s gradient), and the actor never updates. Symptom: stagnant actor training. The fix is the deterministic theorem.

The discrepancy. ∇_a Q · ∇_θ μ vs. ∇ log π · Q differ by the critic’s gradient (deterministic) versus the score function (stochastic). The two are different theorems; the deterministic is not the stochastic σ → 0 limit.

Self-explanation 3. The stochastic’s σ → 0 score (a − μ)/σ² diverges. The deterministic’s ∇_a Q is finite. What is the intuition?

Answer. The stochastic’s score is a likelihood-ratio term that amplifies deviations by 1/σ². The deterministic’s ∇_a Q is a local slope, which is finite at any action. The two estimators are different in kind, and the deterministic is the finite one.

Part B — Twin Critics and the Clipped Double-Q Target

§L.2 — Objects: the twin critics, the min target, the overestimation’s reduction

The twin critics are two Q-networks $Q_{φ_1}$ and $Q_{φ_2}$, both initialized independently and trained on the same transitions. The clipped double-Q target:

y = r + γ (1 − terminated) · min_{i=1,2} $Q_{\barφ_i}$$s', μ_{\barθ}(s') + ε$

where ε is the target policy smoothing noise (Part C) and the min is the pessimistic lower bound (the same as 1.2.5).

The overestimation bias is the twin’s target (the 1.2.5 M4): the single critic’s max over actions is biased upward (Jensen’s inequality for the convex max), and the twin’s min is the pessimistic bound. The two independent critics’ min has expected bias −σ/√π; the single critic’s max has +σ/√π. The negative (underestimation) is less harmful than the positive (overestimation), which causes positive-feedback divergence.

The clipped double-Q is TD3’s (Fujimoto et al. 2018, tier A, S) and SAC’s (Haarnoja et al. 2018, tier A, S), both using the same twin structure.

The update:

$L_Q(φ_i$) = E_{(s, a, r, s') ~ D} [ ($Q_{φ_i}(s$, a) − y)² ]

for each i = 1, 2, with the same target y. The two critics both learn the target; their difference is the initialization and the gradient noise.

§L.2.1 — Result: the clipped double-Q’s min is the overestimation’s reduction; two independent critics give the pessimistic bound

Claim. The clipped double-Q target y = r + γ min_i $Q_{\barφ_i}(s$', a') reduces overestimation bias from the single critic’s +σ/√π to the min’s −σ/√π. The min is pessimistic (conservative policy); underestimation is less harmful than overestimation.

Proof sketch. The max’s bias $E[max(Q_1$, $Q_2$)] − $max(E[Q_1$], $E[Q_2$]) is σ/√π for two Gaussians. The min’s is −σ/√π. The min’s negative (underestimation) is safer than the max’s positive (overestimation), which causes positive-feedback divergence.

Corollary. The min is the same as SAC’s (from 1.2.5), and both algorithms use the clipped double-Q as standard.

Corollary. The clipped double-Q’s cost is the 2× compute for two critics; the benefit is stability (overestimation’s reduction). The twin is standard for modern off-policy RL.

§L.2.2 — Worked-example ladder

WE-L.2.a — Fully worked: the clipped double-Q for a specific transition

Input. Transition (s, a, r = 1.0, s', terminated = False), twin critics $Q_1$$s', μ_{\barθ}(s') + ε$ = 2.0, $Q_2$$s', μ_{\barθ}(s') + ε$ = 1.5, γ = 0.99.

Trace.

1. min(2.0, 1.5) = 1.5.
2. y = 1.0 + 0.99 · 1.5 = 1.0 + 1.485 = 2.485.

Output and interpretation. The target y = 2.485 is what the critic regresses toward. Its components are the immediate reward (1.0) and the discounted min-Q (1.485). The min at 1.5 is the pessimistic lower of the two.

Self-explanation 4. If both critics are equal (2.0, 2.0), what is the target?

Answer. min(2.0, 2.0) = 2.0, and y = 1.0 + 0.99 · 2.0 = 2.98. When the critics agree, the min equals the single critic; the pessimism is only the disagreement (1.5 vs. 2.0), and agreement incurs no penalty.

Self-explanation 5. Why is the min better than the max?

Answer. The max causes over-optimism (positive bias), the actor chases the wrong action, and the training diverges. The min causes under-optimism (negative bias), the actor is conservative, and the training is stable. Under-optimism is less harmful.

WE-L.2.b — Completion problem

Input. Same transition, but terminated = True.

Trace (final 25% as prompts).

1. (1 − terminated) = 0. ⟨verify⟩
2. y = r = 1.0. ⟨verify⟩
3. The min is not used (no future). ⟨verify⟩
4. The critic’s loss is (Q − 1.0)². ⟨verify⟩

Answers. All correct.

The lesson. The terminated flag zeroes the bootstrap (the 1.2.1 M3); the truncated flag keeps it.

WE-L.2.c — Faded: the twin’s overestimation’s reduction

Setup. Two critics $Q_1$ = 1.0 + $ε_1$, $Q_2$ = 1.0 + $ε_2$, $ε_1$, $ε_2$ ~ N(0, 0.1²), true Q = 1.0.

Question. What are the max bias and the min bias?

You must produce the intermediate steps. The setup and the answer are given.

Answer. Max bias: +0.056 (0.1 · 1/√π). Min bias: −0.056 (−0.1 · 1/√π). The two have opposite signs; the min is the negative (conservative).

The lesson. The twin’s min is pessimistic (−0.056); the single’s max is optimistic (+0.056). The min is safer.

WE-L.2.d — Contrastive near-miss: the single critic

Correct solution. Twin critics with the min.

Near-miss. Single critic with the max (DDPG).

Why the near-miss produces a plausible but incorrect number. The single critic’s max overestimates Q (+0.056 from noise), and the actor chases the wrong action (over-optimistic). Q grows unbounded, the policy is suboptimal. The fix is the twin min.

The discrepancy. max gives +0.056, min gives −0.056 — opposite directions. Overestimation is catastrophic (divergence); underestimation is safe (conservative).

Self-explanation 6. If the true Q is 1.0 and the max’s estimate is 1.056, what does the actor chase?

Answer. The actor chases argmax_a Q = 1.056, believing this action is better than it is by 0.056. Over many steps, the error accumulates and the divergence becomes visible.

Part C — Delayed Updates and Target Policy Smoothing

§L.3 — Objects: the delay, the smoothing, the three extensions

The delayed policy update (TD3’s second extension): the actor is updated every d = 2 critic updates. The critic is updated twice for each actor update.

The rationale: the critic’s transient error (from noise) causes the actor’s gradient to point in the wrong direction. The delay gives the critic more steps to converge before the actor changes.

The target policy smoothing (TD3’s third extension): the target’s action is a' = $μ_{\barθ}(s$') + ε with ε ~ clip(N(0, σ²), −c, c), typically σ = 0.2, c = 0.5.

The rationale: the critic’s Q(s', a') may have sharp peaks that the actor exploits (the wrong overestimation), and the smoothing’s noise regularizes the critic against overfitting the sharp peaks.

The three extensions together (twin, delay, smoothing) are TD3. Removing any one degrades performance (Fujimoto et al. 2018’s ablation, tier A, S).

The TD3 loop:

1. Rollout a = μ_θ(s) + $ε_explore$ (exploration noise, $σ_explore$ = 0.1).
2. Store the transition in the buffer.
3. Sample a minibatch.
4. Update the critics: $L_Q$ = ($Q_1$ − y)² + ($Q_2$ − y)² with y = r + γ (1 − term) min_i $Q_{\barφ_i}$$s', μ_{\barθ}(s') + ε_smooth$.
5. Every d = 2 steps, update the actor: L_π = −$E[Q_{φ_1}(s$, μ_θ(s))].
6. Every step, soft-update the targets: \barφ ← τφ + (1−τ)\barφ, \barθ ← τθ + (1−τ)\barθ.

Hyperparameters:

• lr = 3e-4
• γ = 0.99
• τ = 0.005 (soft update)
• $σ_explore$ = 0.1
• $σ_smooth$ = 0.2
• c = 0.5 (clip)
• d = 2 (delay)
• B = 256 (minibatch)
• |D| = 10⁶ (buffer)

§L.3.1 — Result: the delay and smoothing stabilize; the three extensions together are TD3

Claim. The delay (actor every d = 2) stabilizes training by giving the critic more time to converge. The smoothing (target noise) regularizes the critic against sharp Q peaks. The three extensions together (twin, delay, smoothing) are TD3; removing any one degrades performance.

Proof sketch. The delay reduces the actor’s gradient noise by giving the critic more steps before the actor changes. The smoothing reduces the critic’s overfitting to sharp peaks (the Fujimoto et al. 2018 theoretical analysis). The three together are the TD3 (ablation each component necessary).

Corollary. The delay d = 2 is standard; larger d is more stable but slower.

Corollary. The smoothing σ = 0.2 and c = 0.5 are standard. Larger σ is more regularization (less sharp); smaller is less. The 0.2 is the empirical balance.

§L.3.2 — Worked-example ladder

WE-L.3.a — Fully worked: the target smoothing’s noise

Input. Target actor $μ_{\barθ}(s$') = 1.5, σ = 0.2, c = 0.5, sample ε = 0.3 (within the clip [−0.5, 0.5]).

Trace.

1. The ε is clipped: clip(0.3, −0.5, 0.5) = 0.3.
2. a' = $μ_{\barθ}(s$') + ε = 1.5 + 0.3 = 1.8.
3. $Q_1(s$', 1.8) = 2.0, $Q_2(s$', 1.8) = 1.8 (not exactly at 1.5).
4. min(2.0, 1.8) = 1.8.

Output and interpretation. The smoothing puts a' = 1.8 (not 1.5), and the critic computes Q(s', 1.8) (not Q(s', 1.5)). The target is smoother (not the sharp single point 1.5), and the critic regresses the neighborhood (not just the exact 1.5).

Self-explanation 7. If ε = 0.7 (beyond the clip), what is a'?

Answer. clip(0.7, −0.5, 0.5) = 0.5, so a' = 1.5 + 0.5 = 2.0. The clip bounds the noise to [−0.5, 0.5], and the target’s a' never deviates more than 0.5. The clip is a stability bound.

Self-explanation 8. Why are $σ_smooth$ = 0.2 and $σ_explore$ = 0.1 different?

Answer. The exploration σ = 0.1 is the behavior (rollout) noise; the smoothing σ = 0.2 is the critic’s training (target) noise. The exploration is small (precise action), the smoothing is larger (more regularization). The two play different roles.

WE-L.3.b — Completion problem

Input. Same, but d = 2 (actor updated every 2 critic steps).

Trace (final 25% as prompts).

1. Step k = 0: critic update 1; actor update (0 % 2 = 0). ⟨verify⟩
2. Step k = 1: critic update 2; no actor (1 % 2 = 1). ⟨verify⟩
3. Step k = 2: critic update 3; actor update (2 % 2 = 0). ⟨verify⟩
4. Actor updates at k = 0, 2, 4, … (every 2 steps). ⟨verify⟩

Answers. All correct.

The lesson. The delay means the actor is updated every 2 critic steps. The critic is updated twice per actor update. Fewer actor updates, more critic convergence.

WE-L.3.c — Faded: the ablation’s three extensions

Setup. The three TD3 extensions (twin, delay, smoothing), ablation of each.

Question. What is the performance degradation for each?

You must produce the intermediate steps. The setup and the answer are given.

Answer. Remove twin: overestimation not reduced, policy suboptimal (~30% worse). Remove delay: actor overfits critic noise, training brittle (some seeds diverge). Remove smoothing: critic overfits sharp Q peaks, policy exploits wrong actions (~20% worse). Three together: best performance.

The lesson. The three extensions are each necessary; removing any one degrades performance. TD3 is all three together.

WE-L.3.d — Contrastive near-miss: the smoothing’s σ = 0.5

Correct solution. Smoothing σ = 0.2.

Near-miss. Smoothing σ = 0.5 (larger).

Why the near-miss produces a plausible but incorrect number. Larger σ makes the target noise too large, and the critic sees noisy targets (not accurate Q). Symptom: slow convergence, critic underfits. The fix is smaller σ = 0.2.

The discrepancy. σ = 0.2 is moderate regularization; σ = 0.5 is excessive noise. The 0.2 is the empirical balance.

Self-explanation 9. Why not σ = 0 for no smoothing?

Answer. With σ = 0, the target is deterministic, and the critic overfits to sharp Q peaks. The actor exploits these peaks (the wrong action). Some noise is needed to regularize the critic.

Part D — TD3 vs SAC vs PPO: When Each Wins

§L.4 — Objects: the three algorithms, the trade-offs, the task’s match

The comparison:

Task matching:

• PPO — on-policy stability, simple implementation, not the best sample efficiency. Good for simulation with many parallel environments, where data is cheap and wall-clock matters more than sample count.
• SAC — off-policy sample efficiency, automatic exploration via entropy. Good for real-robot tasks where samples are expensive, and for tasks with multimodal solutions.
• TD3 — off-policy sample efficiency, deterministic precision. Good for tasks requiring precise actions (insertion, peg-in-hole, assembly) and for tasks where the exploration’s noise scale is known.

The empirical picture (from 2.1.4’s benchmark knowledge, tier A, S):

• On MuJoCo’s HalfCheetah, Ant, Walker2d, SAC and TD3 match within noise; both beat PPO’s sample efficiency.
• On precision tasks (insertion, in-hand manipulation), TD3 sometimes wins because the deterministic actor avoids the stochastic policy’s noise at the deployment.
• On multimodal tasks (two solutions, e.g., two grasp orientations), SAC wins because the entropy keeps both modes alive; TD3’s deterministic actor commits to one mode.

The three hyperparameters that separate them:

• Exploration: PPO’s policy std, SAC’s entropy α, TD3’s rollout noise σ.
• Target: PPO has none (on-policy), SAC and TD3 have target critics with soft update.
• Actor update: PPO’s clipped objective, SAC’s entropy loss, TD3’s delayed update.

§L.4.1 — Result: TD3 wins on precision, SAC on exploration and multimodality, PPO on parallel simulation; the choice is the task’s action modality and exploration need

Claim. TD3 wins on precision tasks because the deterministic actor’s deployment matches its target. SAC wins on exploration and multimodal tasks because the entropy keeps multiple solutions alive. PPO wins on parallel simulation because it is on-policy and simple, and data is cheap.

Proof sketch. The deterministic actor’s deployment is exactly μ_θ(s), matching the critic’s target Q(s, μ_θ(s)). The stochastic actor’s deployment samples from π_θ, introducing noise not present at training. The entropy term −α H[π] keeps the stochastic policy’s variance; without it, the policy collapses to deterministic (the 1.2.5 bug). The on-policy PPO’s samples must be used immediately (no buffer); the sample efficiency is lower, but the algorithm is simple and stable.

Corollary. For a precision task (peg-in-hole, connector insertion), use TD3.

Corollary. For a multimodal task (grasp from left or right), use SAC.

Corollary. For parallel simulation (many environments, cheap data), use PPO.

§L.4.2 — Worked-example ladder

WE-L.4.a — Fully worked: the choice for the running case

Input. The running case’s reaching task: ArmEnv with the [q, q̇, $p_ee$, $p_target$] state and the τ action.

Trace.

1. Task: reach a target, not multimodal, not super-precise.
2. Sample cost: simulated, cheap.
3. Precision: the $p_target$ reaching is 5 cm tolerance — moderate.
4. Choice: SAC or TD3 (both off-policy), or PPO (on-policy, simulation).

Output and interpretation. All three can solve the reaching task. SAC is the standard choice; TD3 matches within noise; PPO is simpler. For a precision grasp in 2.3.2, TD3 or SAC; for a multimodal grasp (approach from left or right), SAC.

Self-explanation 10. For a peg-in-hole task (insert the peg, tolerance 1 mm), which algorithm?

Answer. TD3 — the deterministic actor’s deployment exactly matches the critic’s target, avoiding the stochastic policy’s noise. SAC’s entropy noise would push the peg off the hole; PPO’s on-policy noise would too.

Self-explanation 11. For a multimodal grasp (reach the object from the left or the right), which algorithm?

Answer. SAC — the entropy keeps both modes alive; TD3 commits to one mode; PPO’s on-policy noise might find both but is sample-inefficient.

WE-L.4.b — Completion problem

Input. A real robot arm, expensive to operate, needing quick deployment.

Trace (final 25% as prompts).

1. Sample cost: expensive (real). ⟨verify⟩
2. Off-policy: preferred (SAC or TD3). ⟨verify⟩
3. Precision: depends on task. ⟨verify⟩
4. Exploration: SAC’s entropy vs TD3’s fixed noise. ⟨verify⟩

Answers. All correct.

The lesson. Real-robot tasks favor off-policy (SAC or TD3) for sample efficiency. The choice between SAC and TD3 depends on precision vs exploration.

WE-L.4.c — Faded: TD3’s precision advantage

Setup. A peg-in-hole task with 1 mm tolerance.

Question. Why does TD3 win over SAC?

You must produce the intermediate steps. The setup and the answer are given.

Answer. TD3’s actor is deterministic, and its deployment action is exactly μ_θ(s). SAC’s deployment samples from π_θ, adding the policy’s std (typically 0.1+) as noise. At 1 mm tolerance, the noise is catastrophic — the peg misses the hole. TD3 wins on precision.

The lesson. The deterministic actor’s deployment precision is TD3’s advantage for precision tasks.

WE-L.4.d — Contrastive near-miss: PPO on a real robot

Correct solution. SAC or TD3 (off-policy).

Near-miss. PPO (on-policy) on a real robot.

Why the near-miss produces a plausible but incorrect number. PPO is on-policy, requiring many samples. On a real robot, each sample is expensive (time, wear). PPO’s sample cost is prohibitive. The training may eventually succeed, but the cost is high.

The discrepancy. On-policy (many samples, no buffer reuse) vs. off-policy (many buffer reuses, sample-efficient). Real robots need off-policy.

Self-explanation 12. On a simulated task with 32 parallel environments, is PPO’s sample cost prohibitive?

Answer. No — parallel simulation gives many samples per second, and PPO’s on-policy simplicity wins. The sample cost is cheap, and PPO’s simplicity is a benefit.

Part E — Failure Taxonomy, Lab, Cheat Sheet, Exercises, Exit Test

§L.5 — Failure taxonomy

§L.6 — The lab

§L.6.1 — Task

Implement TD3 for the ArmEnv (the same reaching task from 1.2.5), with twin critics, target critics, delayed actor updates, target smoothing, and exploration noise; verify the ablation’s effect (remove twin, remove delay, remove smoothing); compare against SAC.

§L.6.2 — Conditions

Python 3.10+, PyTorch 2.2+, CPU. Seed 42, dtype = float64. Runtime under 10 minutes. The ArmEnv (2D position), the TD3 actor (deterministic MLP), twin critics, target critics, buffer 10⁵.

§L.6.3 — Controls held fixed

Seed, dtype, ArmEnv, hyperparameters (lr=3e-4, γ=0.99, τ=0.005, $σ_explore=0$.1, $σ_smooth=0$.2, c=0.5, d=2, B=256). Variants: no twin, no delay, no smoothing.

§L.6.4 — Metrics table

§L.6.5 — Pre-registered hypotheses

The lab ablates one TD3 component at a time against a fixed seed and reports the mean of the last ten evaluation episodes. What follows is what it measures, not what the lesson expects it to measure.

• H1. Standard TD3 reaches return > -0.5 within 5,000 env steps. Measured: −0.4219. Holds.
• H2. Remove the twin critics: Q grows larger than standard by >30%, and the return degrades by > 0.3. Measured: no-twin return −0.3816, which is 0.040 better than standard’s −0.4219. Refuted — on this task the twin critics cost a little return rather than buying any. The task is a 2-D point mass with a dense distance reward, so the overestimation bias the twin critics exist to suppress never compounds enough to matter, and the extra critic’s variance does.
• H3. Remove the delay: training is unstable, with return variance > 2× standard. Measured: no-delay return −0.4633 against standard −0.4219. Not tested as stated — the lab reports one seed per variant, so it reports no variance at all. What it does show is a 0.041 return drop, consistent with the delay helping, but a single seed cannot distinguish that from noise.
• H4. Remove target smoothing: the return degrades by ~20% (> 0.2). Measured: no-smoothing −0.4084 against standard −0.4219 — 0.013 better. Refuted at this scale, for the same reason as H2: target smoothing guards against a critic that exploits sharp peaks, and a 64-unit critic on a two-input regression problem does not develop them.
• H5. SAC on the same task converges to a similar return with a similar sample count (|SAC − TD3| < 0.1). Not implemented. The lab trains TD3 only; there is no SAC run and no comparison to make. This hypothesis is tier C, status P.

The pattern across H2–H4 is worth stating plainly: this environment is too easy for TD3’s three extensions to earn their keep. They are variance-reduction devices against overestimation, and overestimation needs a hard credit-assignment problem to show up. A learner who runs this lab and concludes “the twins don’t help” has read the result correctly and the lesson’s framing incorrectly.

§L.6.6 — Break-it sequence

1. No exploration noise. $σ_explore$ = 0. Predicted: actor converges to a fixed action (no exploration), poor return. Actual: poor.
2. Single critic. Only $Q_1$, no min. Predicted: overestimation, return degrades. Actual: return improves (−0.3816 vs −0.4219). The predicted mechanism is real; it just does not fire on this task.
3. No delay. d = 1 (actor every step). Predicted: unstable, high variance. Actual: return drops to −0.4633; variance not measured (one seed).
4. No smoothing. $σ_smooth$ = 0. Predicted: Q peaks exploited, return degrades. Actual: return improves slightly (−0.4084 vs −0.4219).

All four are silent — none raises, and in two cases the ablated variant scores better, which is the harder diagnostic problem: the symptom is absent rather than merely quiet.

§L.6.7 — Acceptance criteria

§L.6.8 — What good looks like

1. TD3 converges with the full three extensions (−0.4219 against the −0.5 bar).
2. Ablation demonstrated for each extension — and reported with its sign, including the two ablations that score better than the full algorithm.
3. SAC comparison measured. Not done here; a learner who wants it must add the SAC run.
4. The four break-its predicted with symptoms before running, then checked against what happened.
5. The choice between TD3 and SAC justified by the task — and, in this environment, an honest report that two of TD3’s three extensions do not pay for themselves.

§L.6.9 — Runnable lab code

# lab_2_2_1.py — TD3 with twin critics, delay, smoothing, and ablations
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import copy

torch.manual_seed(42)
dtype = torch.float64
device = torch.device("cpu")

class SimpleArmEnv:
    def __init__(self):
        self.$p_target$ = torch.tensor([0.5, 0.2], dtype=dtype)
        self.state_dim = 2; self.action_dim = 2
    def reset(self):
        self.pos = torch.randn(2, dtype=dtype) * 0.1
        return self.pos.clone()
    def step(self, a):
        self.pos = self.pos + a * 0.1
        dist = torch.norm(self.pos - self.$p_target$).item()
        return self.pos.clone(), -dist, True, True

class DeterministicActor(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, action_dim), nn.Tanh(),
        )
    def forward(self, s):
        return self.net(s)

class TwinCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden=64):
        super().__init__()
        self.q1 = nn.Sequential(nn.Linear(state_dim + action_dim, hidden), nn.ReLU(),
                                 nn.Linear(hidden, hidden), nn.ReLU(),
                                 nn.Linear(hidden, 1))
        self.q2 = nn.Sequential(nn.Linear(state_dim + action_dim, hidden), nn.ReLU(),
                                 nn.Linear(hidden, hidden), nn.ReLU(),
                                 nn.Linear(hidden, 1))
    def forward(self, s, a):
        sa = torch.cat([s, a], dim=-1)
        return self.q1(sa).squeeze(-1), self.q2(sa).squeeze(-1)

class ReplayBuffer:
    def __init__(self, cap, sd, ad):
        self.cap = cap; self.ptr = 0; self.size = 0
        self.s = torch.zeros(cap, sd, dtype=dtype)
        self.a = torch.zeros(cap, ad, dtype=dtype)
        self.r = torch.zeros(cap, dtype=dtype)
        self.s2 = torch.zeros(cap, sd, dtype=dtype)
        self.term = torch.zeros(cap, dtype=torch.bool)
    def add(self, s, a, r, s2, term):
        self.s[self.ptr] = s; self.a[self.ptr] = a; self.r[self.ptr] = r
        self.s2[self.ptr] = s2; self.term[self.ptr] = term
        self.ptr = (self.ptr + 1) % self.cap
        self.size = min(self.size + 1, self.cap)
    def sample(self, b):
        idx = torch.randint(0, self.size, (b,))
        return self.s[idx], self.a[idx], self.r[idx], self.s2[idx], self.term[idx]

# Hyperparameters
LR = 3e-4; GAMMA = 0.99; TAU = 0.005; BATCH = 256
SIGMA_EXPLORE = 0.1; SIGMA_SMOOTH = 0.2; CLIP = 0.5; DELAY = 2

def $td3_train(n_steps=20000$, use_twin=True, use_delay=True, use_smooth=True):
    env = SimpleArmEnv()
    actor = DeterministicActor(2, 2).to(dtype)
    critic = TwinCritic(2, 2).to(dtype)
    actor_t = copy.deepcopy(actor)
    critic_t = copy.deepcopy(critic)
    $a_opt$ = torch.optim.Adam(actor.parameters(), lr=LR)
    $c_opt$ = torch.optim.Adam(critic.parameters(), lr=LR)
    buf = ReplayBuffer(100000, 2, 2)
    s = env.reset()
    returns = []; ep_ret = 0.0; step_count = 0
    for step in $range(n_steps$):
        with torch.no_grad():
            a = actor(s) + torch.randn(2, dtype=dtype) * SIGMA_EXPLORE
            a = a.clamp(-1, 1)
        s2, r, term, trunc = env.step(a.numpy())
        buf.add(s, a, torch.tensor(r, dtype=dtype), s2, term)
        ep_ret += r
        if term or trunc:
            returns.append(ep_ret); ep_ret = 0.0; s = env.reset()
        else:
            s = s2
        if buf.size < BATCH:
            continue
        sb, ab, rb, s2b, termb = buf.sample(BATCH)
        with torch.no_grad():
            a2 = actor_t(s2b)
            if use_smooth:
                a2 = a2 + (torch.randn_like(a2) * SIGMA_SMOOTH).clamp(-CLIP, CLIP)
                a2 = a2.clamp(-1, 1)
            q1t, q2t = critic_t(s2b, a2)
            y = rb + GAMMA * (1 - termb.float()) * ((torch.min(q1t, q2t) if use_twin else q1t))
        q1, q2 = critic(sb, ab)
        $c_loss$ = F.mse_loss(q1, y) + (F.mse_loss(q2, y) if use_twin else 0)
        $c_opt$.zero_grad(); $c_loss$.backward(); $c_opt$.step()
        step_count += 1
        if use_delay and (step_count % DELAY != 0):
            pass
        else:
            $a_pred$ = actor(sb)
            q1p, _ = critic(sb, $a_pred$)
            $a_loss$ = -q1p.mean()
            $a_opt$.zero_grad(); $a_loss$.backward(); $a_opt$.step()
        with torch.no_grad():
            for p, pt in zip(critic.parameters(), critic_t.parameters()):
                pt.data.mul_(1 - TAU); pt.data.add_(TAU * p.data)
            for p, pt in zip(actor.parameters(), actor_t.parameters()):
                pt.data.mul_(1 - TAU); pt.data.add_(TAU * p.data)
    return returns

print("=== Standard TD3 ===")
rets = td3_train(use_twin=True, use_delay=True, use_smooth=True)
print(f"final return: {np.mean(rets[-10:]):.4f}")

print("\n=== No twin ===")
rets_nt = td3_train(use_twin=False, use_delay=True, use_smooth=True)
print(f"final return: {np.mean(rets_nt[-10:]):.4f}")

print("\n=== No delay ===")
rets_nd = td3_train(use_twin=True, use_delay=False, use_smooth=True)
print(f"final return: {np.mean(rets_nd[-10:]):.4f}")

print("\n=== No smoothing ===")
rets_ns = td3_train(use_twin=True, use_delay=True, use_smooth=False)
print(f"final return: {np.mean(rets_ns[-10:]):.4f}")

print("\n--- Summary ---")
print(f"H1: standard return = {np.mean(rets[-10:]):.4f}")
print(f"H2: no-twin return = {np.mean(rets_nt[-10:]):.4f} (should be worse)")
print(f"H3: no-delay return = {np.mean(rets_nd[-10:]):.4f} (should be unstable)")
print(f"H4: no-smoothing return = {np.mean(rets_ns[-10:]):.4f} (should be worse)")

§L.7 — The cheat sheet

§L.8 — Exercises with solutions

E 2.2.1.1 — The deterministic gradient (numerical). μ_θ(s) = θ s, θ = 0.5, s = 2, Q(s, a) = −(a − 1.5)². Compute ∇_θ J.

S 2.2.1.1. ∇_a Q = 1.0, ∇_θ μ = 2, ∇_θ J = 2.0. The lesson: the chain rule gives the gradient.

E 2.2.1.2 — The clipped double-Q (numerical). $Q_1$ = 2.0, $Q_2$ = 1.5, r = 1.0, γ = 0.99, terminated = False. Compute y.

S 2.2.1.2. min(2.0, 1.5) = 1.5, y = 1.0 + 0.99 · 1.5 = 2.485. The lesson: the min is pessimistic.

E 2.2.1.3 — The smoothing’s clip (numerical). μ = 1.5, σ = 0.2, c = 0.5, ε = 0.7. Compute a'.

S 2.2.1.3. clip(0.7, −0.5, 0.5) = 0.5, a' = 2.0. The lesson: the clip bounds the target noise.

E 2.2.1.4 — The delay (conceptual). d = 2. How many actor updates in 1000 critic steps?

S 2.2.1.4. 500 actor updates. The lesson: the actor is updated every 2 critic steps.

E 2.2.1.5 — The exploration (conceptual). Why does the deterministic actor need explicit noise?

S 2.2.1.5. Without noise, the actor never explores; the critic only sees the actor’s current action and can’t learn about unseen actions. The lesson: the noise is the behavior policy’s exploration.

E 2.2.1.6 — The overestimation (diagnostic). Q grows unbounded. Cause and fix?

S 2.2.1.6. Single critic (max overestimation). Fix: twin critics with min. The lesson: overestimation causes divergence.

E 2.2.1.7 — The stability (diagnostic). Training diverges on some seeds. Cause and fix?

S 2.2.1.7. Missing delay or smoothing. Fix: d = 2 and σ = 0.2. The lesson: both stabilize.

E 2.2.1.8 — The task choice (constructive). A peg-in-hole task with 1 mm tolerance. TD3 or SAC?

S 2.2.1.8. TD3 — deterministic action’s deployment matches the critic’s target, no stochastic noise. The lesson: precision tasks favor TD3.

§L.9 — Exit test

You are ready for Lesson 2.2.2 (Model-Based RL) when you can, without notes:

1. State the deterministic policy gradient theorem and why it is off-policy.
2. Explain DDPG’s three weaknesses.
3. State the clipped double-Q and why the min reduces overestimation.
4. State target policy smoothing and its role.
5. State the delayed update and its role.
6. State the TD3 loop’s hyperparameters.
7. Compare TD3, SAC, PPO.
8. State the task matching (TD3 precision, SAC exploration, PPO parallel).
9. Diagnose no exploration (poor actor), single critic (Q overestimates), no delay (unstable), no smoothing (overfits sharp peaks).
10. State the $σ_explore$ vs $σ_smooth$ difference.
11. State the ablation’s each extension’s importance.
12. State the deployment precision advantage of TD3.

If any is not answerable, return: 1 to §L.1, 2 to §L.1, 3 to §L.2, 4 to §L.3, 5 to §L.3, 6 to §L.3, 7 to §L.4, 8 to §L.4, 9 to §L.5, 10 to §L.3, 11 to §L.3.1, 12 to §L.4.1.

End of Lesson 2.2.1. The learner has now completed the TD3 foundation: the deterministic policy gradient, DDPG’s baseline, the twin critics, the clipped double-Q, the target smoothing, the delayed update, and the comparison against SAC and PPO. The next lesson, 2.2.2 — Model-Based RL, uses a learned dynamics model to plan and reduce sample complexity, and its tree-search and latent-space planning rely on the value functions and the TD3 and SAC critics this lesson provides.