X1.2.4

Lesson 1.2.4 — PPO: Trust Regions, Clipping, and the Modern Policy Gradient

Week 8 · Prerequisites: 1.2.1, 1.2.2, 1.2.3, 0.1.2, 0.3.2, 0.3.3.

Prior results used. This lesson uses 1.2.1’s ArmEnv (the observation/action spaces, the terminated/truncated distinction), 1.2.2’s value functions V^π/Q^π and the Bellman equations, 1.2.3’s policy gradient theorem, REINFORCE, the baseline/advantage, the Gaussian policy, and the tanh squashing’s Jacobian, 0.1.2’s autograd and torch.no_grad(), 0.3.2’s log-likelihood and its optimization, and 0.3.3’s filtering vs. smoothing (for the GAE’s causality). Each is restated at the point of use.

Hands forward. This lesson hands forward one capability: the learner can derive the PPO objective from the trust-region motivation, implement the clipped surrogate with the importance-sampling ratio, compute GAE(λ) advantages with the correct bias-variance trade-off, run the PPO loop with multiple epochs and minibatches, and diagnose the five PPO-specific bugs (ratio mis-scaling, clip’s asymmetry ignored, GAE’s λ mis-set, KL divergence blow-up, and value-loss dominating) from their learning symptoms — the capability that Module 2.2 (advanced RL), Module 3.1 (VLA fine-tuning), and the Phase 4 capstone all assume.

Running case. The same 6-DOF MuJoCo arm from 1.1.1–1.2.3, in the ArmEnv (the observation [q, q̇, p_ee, p_target], the reward −‖p_ee − p_target‖ − 0.01‖τ‖², the terminated/truncated), now with PPO as the learning algorithm: the actor π_θ(a | s) (a Gaussian with a learned mean and log-std), the critic V_φ(s) (a separate network), the rollout of N_envs × T_horizon transitions per iteration, and the multiple epochs of the clipped objective. The lesson’s central result is that the clipping is the pessimistic lower bound on the trust-region update, and it is what makes the multiple epochs stable where the vanilla policy gradient’s single step would diverge — the ratio r_t(θ) = π_θ(a_t | s_t)/π_{θ_old}(a_t | s_t) bounds the update’s step, and the min of the clipped and unclipped objectives removes the incentive to move the ratio outside the [1 − ε, 1 + ε] band.

§L.0 — Lesson map, objectives, and misconceptions

§L.0.1 — Lesson map and word budget

§L.0.2 — Objectives

By the end of this lesson you will be able to do eight things:

1. State the trust-region motivation: the vanilla policy gradient’s step size is uncontrolled, and a large step changes the policy’s distribution so much that the next rollout is off-policy (the old gradient’s assumption is violated), and the KL divergence is the measure of the change.
2. Derive the clipped surrogate objective L^{CLIP}(θ) = E_t[min(r_t(θ) A_t, clip(r_t(θ), 1−ε, 1+ε) A_t)] from the importance-sampling ratio and the pessimistic bound, and state why the min bounds the improvement’s estimate.
3. Compute the ratio r_t(θ) and the clipped and unclipped objectives by hand for a small example, and state the gradient’s zero for the advantage-positive and ratio > 1+ε case, and the advantage-negative and ratio < 1−ε case.
4. State the GAE(λ) advantage A^{GAE}_t = Σ_{l=0}^∞ (γλ)^l δ_{t+l} with the TD residual δ_t = r_t + γ V(s_{t+1}) − V(s_t), and compute it by hand for a 3-step trajectory.
5. State the λ’s role — the bias-variance trade-off (λ = 0 gives the low-variance, high-bias TD, λ = 1 gives the high-variance, unbiased Monte Carlo) — and the standard choice λ = 0.95.
6. Implement the PPO loop: the rollout (N_envs × T_horizon), the GAE advantages, the multiple epochs (typically K = 10), the minibatches (typically M = 32), the clipped objective, the value loss (the MSE to the returns), the entropy bonus, and the gradient clipping (the max_norm = 0.5).
7. State the PPO’s hyperparameter choices (the ε = 0.2, the γ = 0.99, the λ = 0.95, the K = 10, the M = 32, the lr = 3e-4, the entropy_coef = 0.01, the value_coef = 0.5) and the rationale for each.
8. Diagnose the five PPO-specific bugs — ratio mis-scaling (the ratio is the opposite of the expected), clip’s asymmetry ignored (the min is omitted, the pessimism is lost), GAE’s λ mis-set (λ = 1 gives the high-variance Monte Carlo, λ = 0 gives the high-bias TD), KL divergence blow-up (the policy changes too much, the ratio is far from 1), value loss dominating (the critic’s loss overwhelms the actor’s) — from their learning symptoms.

§L.0.3 — Misconception inventory

Each misconception is displaced in the section named, and each is tagged in the concept check that follows.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

This is the twentieth lesson. The prompt requires drawing from lessons two to five steps back; 1.2.3 is one step back and is excluded. The warm-up draws from 1.2.2 (two steps back), 1.2.1 (three), 1.1.4 (four), and 1.1.3 (five).

Warm-up 1 (from 1.2.2, the advantage and the value). State the advantage A(s, a) and its relationship to Q and V. Answer: A(s, a) = Q(s, a) − V(s); the advantage measures the excess of the action’s value over the state’s expected value, and its expectation under the policy is zero (E_{a ~ π}[A(s, a)] = 0). How this is used: §L.2’s PPO’s objective uses the advantage A_t (the baseline-subtracted return), and §L.3’s GAE computes the A_t from the TD residuals — the two are the same quantity, estimated differently (1.2.3’s G_t − V vs. 1.2.4’s GAE).

Warm-up 2 (from 1.2.1, the ArmEnv’s rollout). State the ArmEnv’s episode structure and the reset/step interface. Answer: the reset returns the initial observation, the step returns (obs, reward, terminated, truncated, info), and the episode ends at the terminated (success/collision) or the truncated (time limit). How this is used: §L.4’s PPO loop collects a rollout of N_envs × T_horizon transitions, and the terminated/truncated distinction is the bootstrap’s boundary (the 1.2.2’s warm-up 3’s issue).

Warm-up 3 (from 1.1.4, the pipeline’s stage contract). State the pick-and-place pipeline’s stage contract and the health checks. Answer: the seven stages with the (input, output, frame) contracts and the health checks (the reachability, the branch continuity, the tracking, the grasp’s force). How this is used: §L.4’s PPO’s training evaluates the policy on the ArmEnv (the simplified pipeline), and the health checks are the episode’s terminated (the success’s condition), the reward’s design, and the evaluation’s metric (1.1.4’s success rate).

Warm-up 4 (from 1.1.3, the PID’s stability). State the PD’s stability limit and the sample rate’s rule. Answer: the K_p and K_d must be positive (the stability), and the ω_n T_s ≤ 0.1 (the sample rate’s 10× rule). How this is used: §L.1’s trust-region motivation is the same concern in the policy’s space — the step size must be small enough (the KL’s ε) for the stability, and the too large step is the policy’s divergence (the 1.1.3’s instability’s analogue).

Part A — The Trust-Region Motivation

§L.1 — Objects: the step size, the KL divergence, and the trust-region

The vanilla policy gradient θ ← θ + α ∇_θ J(θ) has an uncontrolled step size — the learning rate α is a scalar, and the policy’s change is the α’s magnitude times the gradient’s direction and scale. For a large α, the policy π_θ changes a lot, and the next rollout’s transitions are sampled from a different distribution than the gradient’s assumption (the on-policy’s π_θ). The gradient’s estimate is biased for the new policy, and the training diverges (the large step, the bad update, the worse policy).

The KL divergence D_{KL}(π_{θ_old} ‖ π_θ) is the measure of the policy’s change:

The KL is non-negative (the Gibbs’ inequality), and it is zero iff the two policies are identical. The trust region is the set of policies within a KL’s radius δ of the old policy:

The TRPO (Trust Region Policy Optimization) solves the constrained optimization:

The objective is the importance-sampling’s surrogate (the π_θ / π_{θ_old}’s ratio times the advantage), and the constraint is the KL’s bound (the trust region). The TRPO’s solution uses the conjugate gradient (the Hessian-vector product) and the line search (the backtracking), and it is expensive (the second-order information, the many iterations).

The PPO (Proximal Policy Optimization) approximates the TRPO’s constrained problem with a clipped unconstrained objective, and it is cheaper (the first-order information, the standard SGD). The clipping is the practical enforcement of the trust region, and the min is the pessimistic bound that makes the clipping work.

§L.1.1 — Result: the uncontrolled step biases the gradient for the new policy, and the trust region bounds the step — the TRPO’s KL’s constraint is the exact bound, and the PPO’s clipping is the first-order approximation

Claim. For the vanilla policy gradient with the step α, the new policy π_{θ+αg} differs from the old π_θ by the α’s magnitude times the gradient’s scale, and the gradient’s estimate ∇ log π_θ · A is biased for the new policy (the transitions are sampled from the old). The KL’s constraint D_{KL}(π_{old} ‖ π_{new}) ≤ δ bounds the step, and the TRPO’s constrained optimization maximizes the surrogate within the trust region. The PPO’s clipping bounds the ratio r_t(θ) = π_θ / π_{old} to the [1 − ε, 1 + ε] band, which is the first-order approximation of the KL’s constraint (the KL’s second-order expansion around the π_{old}).

Proof sketch. The vanilla gradient’s step is the α · g, and the new policy’s parameter θ + αg is different by the α‖g‖. The KL between the π_{θ} and π_{θ+αg} is approximately ½ α² gᵀ F g (the second-order expansion with the Fisher information F), and the KL’s constraint ½ α² gᵀ F g ≤ δ gives the maximum α = √(2δ/(gᵀ F g)). The PPO’s clipping is the first-order version of the KL’s constraint — the ratio π_θ/π_{old} is bounded by the 1 ± ε, and the ε is the trust-region’s radius in the ratio’s space.

Corollary. The PPO’s clipping is a heuristic (the exact trust region is the KL’s constraint, and the PPO’s clipping is the approximation), and the practical PPO works because the ratio’s band is close to the KL’s bound for the small θ’s change (the Taylor’s expansion). The exact KL’s constraint’s PPO (the KL’s penalty instead of the clip) is the alternative (the PPO-KL), and the clip is the default (the simpler and the empirically competitive).

Corollary. The trust region’s radius δ (or the clip’s ε) is the step size’s control — the larger δ (or ε) allows the larger steps (the faster learning, the more instability), and the smaller allows the smaller (the slower, the more stable). The choice is the task’s trade-off, and the standard ε = 0.2 is the empirical balance.

§L.1.2 — Worked-example ladder

WE-L.1.a — Fully worked: the KL’s divergence for a Gaussian policy

Input. Two 1D Gaussian policies: the π_{old} = N(0, 1) and the π_θ = N(0.5, 1). Compute the KL’s divergence.

Trace.

1. The closed-form for the Gaussian’s KL: D_{KL}(N(μ_1, σ_1²) ‖ N(μ_2, σ_2²)) = log(σ_2/σ_1) + (σ_1² + (μ_1 − μ_2)²)/(2σ_2²) − ½.
2. With the μ_1 = 0, the μ_2 = 0.5, the σ_1 = σ_2 = 1: log(1) + (1 + 0.25)/2 − 0.5 = 0 + 0.625 − 0.5 = 0.125.
3. The KL is 0.125 (the nats).

Output and interpretation. The KL’s 0.125 is the divergence between the two Gaussians, and it is small (the 0.5’s mean shift is the σ’s 0.5). The trust-region’s δ = 0.01 would bound the μ’s shift to √(2·0.01) ≈ 0.14 (the 1D Gaussian’s approximation), and the 0.5’s shift is outside the trust region — the step is too large, and the next rollout’s transitions are off-policy.

Self-explanation 1. Before reading on: in step 3, the KL is 0.125 for a 0.5’s mean shift. What is the KL for a 1.0’s mean shift? For a 2.0’s?

Answer. The μ’s shift’s KL is (Δμ)²/2 for the same std: the 0.5’s shift gives 0.125, the 1.0’s gives 0.5, and the 2.0’s gives 2.0. The KL’s quadratic growth means the large shifts cost quadratically, and the trust-region’s δ = 0.01 bounds the shift to √(0.02) ≈ 0.14 — the small shift’s trust region.

Self-explanation 2. Before reading on: the KL’s δ = 0.01 and the ε = 0.2 (the PPO’s clip) — how are the two related?

Answer. The KL’s δ = 0.01 and the clip’s ε = 0.2 are the different parameterizations of the same trust region — the KL’s δ is the second-order (the Fisher’s scale), and the clip’s ε is the ratio’s first-order (the π_new/π_old’s range). The two are related by the local approximation ε ≈ √(2δ) (the ε = 0.2 corresponds to the δ ≈ 0.02), and the exact relation depends on the policy’s family. The practical PPO uses the clip’s ε (the simpler), and the TRPO uses the KL’s δ (the exact).

WE-L.1.b — Completion problem

Input. The two Gaussians from WE-L.1.a, but the σ_2 = 2 (the new policy’s std is 2× the old’s), the μ_2 = 0.5.

Trace (final 25% as prompts).

1. The KL’s formula: log(2/1) + (1 + 0.25)/(2·4) − 0.5 = 0.693 + 0.156 − 0.5 = 0.349. ⟨your turn⟩
2. The std’s change adds the 0.693 to the KL (the std’s ratio’s log). ⟨your turn⟩
3. The total KL is 0.349, larger than the 0.125 (the std’s change’s contribution). ⟨your turn⟩
4. The trust-region’s δ = 0.01 would bound the std’s change more tightly — the 0.349 is outside. ⟨your turn⟩

Answers.

1. 0.349. ✓
2. The std’s ratio’s log is the log(σ_2/σ_1) = log(2) = 0.693. ✓
3. The 0.349 > 0.125 (the std’s change adds the 0.693). ✓
4. The trust region’s δ = 0.01 bounds the std’s change to σ_2/σ_1 ≈ e^{0.01} ≈ 1.01 — the 2× change is outside. ✓

The lesson. The std’s change contributes the log(σ_2/σ_1) to the KL, and the std’s ratio’s log is the first term. The small δ bounds the std’s change tightly, and the large change (the 2×) is outside the trust region. The KL’s constraint bounds the std as well as the mean, and the PPO’s clip (the ratio’s range) bounds the same.

WE-L.1.c — Faded: the TRPO’s step size

Setup. The TRPO’s δ = 0.01, the gradient g with the ‖g‖ = 1, the Fisher F = I (the identity).

Question. What is the maximum step α?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The KL’s ½ α² gᵀ F g ≤ δ, with the F = I and the gᵀg = 1, so ½ α² ≤ 0.01, α ≤ √0.02 ≈ 0.141. The maximum step is 0.141, and the TRPO’s line search ensures the KL’s constraint.

The lesson. The TRPO’s step size is determined by the KL’s δ and the gradient’s scale (the gᵀ F g’s square root), and the step adapts to the gradient — the large gradient gives the small step (the KL’s bound), and the small gradient gives the large step (the same KL). The adaptive step is the trust region’s essence, and the fixed learning rate is the vanilla policy gradient’s limitation.

WE-L.1.d — Contrastive near-miss: the large learning rate

Correct solution. The trust-region’s δ = 0.01 (or the clip’s ε = 0.2), with the adaptive step.

Near-miss. The vanilla policy gradient with the α = 1.0 (the large learning rate).

Why the near-miss produces a plausible but incorrect number. The large α = 1.0 makes the policy’s step huge (the gradient’s scale times the 1.0), and the new policy is far from the old — the KL’s divergence is large, the rollout is off-policy, and the next gradient’s estimate is biased. The training diverges (the return decreases, the policy oscillates). The symptom is the large oscillation and the divergence, and the fix is the smaller learning rate (the 3e-4) or the trust-region’s adaptive step.

The discrepancy, stated. The fixed α and the adaptive α (the trust-region’s) differ by the step size’s adaptation — the fixed is easy (the 3e-4) but not the optimal (the large gradient’s large step is the divergence), and the adaptive is optimal (the KL’s bound) but complicated (the TRPO’s Hessian). The PPO’s clip is the middle (the fixed ε but the ratio’s range’s bound), and it is the practical choice.

Self-explanation 3. Before reading on: in the near-miss, if the α = 1.0 and the gradient’s norm is 0.001 (the tiny gradient), is the step small or large?

Answer. The step is α · ‖g‖ = 1.0 · 0.001 = 0.001 — the small (the tiny gradient’s small step). The near-miss’s danger is the large gradient (the norm 1.0), where the α = 1.0 gives the 1.0 step (the huge). The fixed α’s step’s scale is proportional to the gradient’s scale, and the large gradient’s large step is the divergence’s source. The trust-region’s adaptive α inversely proportional to the gradient’s scale (the √(2δ/(gᵀ F g))), and the large gradient’s small step is the stability’s reason.

Part B — The PPO Clipped Objective

§L.2 — Objects: the ratio, the clip, the min, and the pessimistic bound

The PPO’s clipped objective is:

where:

• r_t(θ) = π_θ(a_t | s_t) / π_{θ_old}(a_t | s_t) is the importance-sampling ratio — the new policy’s probability divided by the old’s.
• A_t is the advantage (from the GAE, the Part C).
• clip(r_t(θ), 1 − ε, 1 + ε) is the clipping of the ratio to the [1 − ε, 1 + ε] band (the ε = 0.2 standard).
• min(·, ·) is the pessimistic minimum of the unclipped and the clipped objectives.

The key properties:

1. For the A_t > 0 (the good action): the objective is the min(r_t A_t, clip(r_t) A_t) — the unclipped and the clipped match when the r_t ≤ 1 + ε, and the clipped caps the objective at the (1 + ε) A_t when the r_t > 1 + ε. The gradient is zero when the r_t > 1 + ε (the min’s flat region), so the policy is not incentivized to increase the ratio further — the trust region’s upper bound.
2. For the A_t < 0 (the bad action): the objective is the min(r_t A_t, clip(r_t) A_t) — the unclipped is r_t A_t (the positive when the r_t < 1, the negative when the r_t > 1), and the clipped is (1 − ε) A_t when the r_t < 1 − ε (the larger negative). The min chooses the larger negative (the pessimistic), and the gradient is zero when the r_t < 1 − ε (the policy is not incentivized to decrease the ratio further) — the trust region’s lower bound.
3. For the r_t ∈ [1 − ε, 1 + ε] (the trust region’s interior): the unclipped and the clipped coincide (the clip doesn’t change the value), and the objective is the standard policy gradient’s r_t A_t — the gradient is non-zero, and the learning proceeds.

The pessimistic bound: the min ensures the objective is a lower bound on the true policy gradient’s improvement (the TRPO’s theory), and the bound is the trust region’s enforcement. The clip alone (without the min) would allow the large ratio’s large objective, and the trust region’s bound would be violated.

§L.2.1 — Result: the PPO’s clip bounds the ratio to the trust region, and the min removes the incentive to move outside — the gradient is zero for the advantage-positive and ratio > 1+ε, and for the advantage-negative and ratio < 1−ε, and non-zero for the trust region’s interior

Claim. The PPO’s clipped objective L^{CLIP} is equal to the standard policy gradient’s r_t A_t for the r_t ∈ [1−ε, 1+ε], and it is flat (the gradient zero) for the r_t outside the band when the advantage’s sign would push the ratio further out (the A > 0 with the r > 1+ε, and the A < 0 with the r < 1−ε). The gradient is non-zero for the outside the band when the advantage’s sign would pull the ratio back (the A > 0 with the r < 1−ε, and the A < 0 with the r > 1+ε) — the clip’s min removes the incentive to move outside, and the trust region’s bound is enforced.

Proof sketch. The objective is the min(r_t A_t, clip(r_t) A_t). For the A_t > 0 and the r_t > 1 + ε: the clip(r_t) = 1 + ε, and the clip(r_t) A_t = (1 + ε) A_t < r_t A_t (the min), so the objective is (1 + ε) A_t — the constant (the gradient zero). For the A_t > 0 and the r_t < 1 + ε: the clip(r_t) = r_t, and the objective is r_t A_t (the gradient non-zero). The symmetric for the A_t < 0. The combination bounds the ratio’s incentive to the band.

Corollary. The PPO’s clip’s ε is the trust region’s radius, and the value must match the policy’s variance and the rollout’s length — the small ε (the 0.1) is the conservative (the slow learning), the large ε (the 0.3) is the aggressive (the risk of the divergence). The standard ε = 0.2 is the empirical balance.

Corollary. The multiple epochs of the PPO (the K = 10) reuse the same rollout, and the ratio r_t(θ) moves away from the 1 as the θ changes. The clip bounds the ratio’s movement, and the multiple epochs are safe — the REINFORCE’s single epoch is the alternative, and the sample efficiency’s gain is the PPO’s advantage.

§L.2.2 — Worked-example ladder

WE-L.2.a — Fully worked: the clipped objective for a specific advantage and ratio

Input. The A_t = 1.0 (the good action), the ε = 0.2, the ratio r_t = 1.5 (the large increase).

Trace.

1. The clip(r_t, 1 − ε, 1 + ε) = clip(1.5, 0.8, 1.2) = 1.2.
2. The unclipped objective: r_t · A_t = 1.5 · 1.0 = 1.5.
3. The clipped objective: 1.2 · 1.0 = 1.2.
4. The min of the two: min(1.5, 1.2) = 1.2.

Output and interpretation. The objective is 1.2 (the clipped), and the gradient is zero for the further increase of the ratio (the min’s flat). The policy is not incentivized to increase the ratio further (the 1.5 is capped at the 1.2), and the trust region’s upper bound is enforced.

Self-explanation 4. Before reading on: in step 4, the min chooses the 1.2 (the clipped). What if the A_t = -1.0 (the bad action) and the ratio r_t = 1.5?

Answer. The r_t = 1.5 for the bad action: the unclipped is 1.5 · (-1) = -1.5, the clipped is 1.2 · (-1) = -1.2, and the min is -1.5 (the unclipped is smaller). The gradient is non-zero — the policy is incentivized to decrease the ratio (the reducing the bad action’s probability), which is correct (the bad action should be avoided). The min allows the decrease (the pull back the ratio), and it is the pessimistic bound’s asymmetry — the good action’s large ratio is capped, and the bad action’s large ratio is penalized.

Self-explanation 5. Before reading on: in step 1, the clip(1.5, 0.8, 1.2) = 1.2 — what is the clip’s definition?

Answer. The clip clip(x, a, b) = max(a, min(b, x)) — the max of the a and the min of the b and the x. The clip(1.5, 0.8, 1.2) = max(0.8, min(1.2, 1.5)) = max(0.8, 1.2) = 1.2. The clip bounds the x to the [a, b]’s range, and the 1.5 is clipped to the 1.2 (the upper bound). The ratio’s lower bound is the 0.8 (the 1 − ε), and the upper is the 1.2 (the 1 + ε).

WE-L.2.b — Completion problem

Input. The A_t = 0.5, the r_t = 0.5 (the decreased ratio), the ε = 0.2.

Trace (final 25% as prompts).

1. The clip(0.5, 0.8, 1.2) = 0.8 (the lower bound). ⟨your turn⟩
2. The unclipped objective: 0.5 · 0.5 = 0.25. ⟨your turn⟩
3. The clipped objective: 0.8 · 0.5 = 0.4. ⟨your turn⟩
4. The min of the two: min(0.25, 0.4) = 0.25 (the unclipped is smaller). ⟨your turn⟩
5. The gradient: non-zero (the ratio’s decrease is incentivized to be reversed — the good action’s probability should be increased). ⟨your turn⟩

Answers.

1. 0.8. ✓
2. 0.25. ✓
3. 0.4. ✓
4. 0.25 (the unclipped). ✓
5. The gradient is non-zero — the policy is incentivized to increase the ratio (the good action’s probability). ✓

The lesson. For the A_t > 0 and the r_t < 1 − ε, the unclipped is the r_t · A_t (the small), and the clipped is the (1 − ε) A_t (the larger), and the min chooses the unclipped — the objective is the r_t · A_t, and the gradient is non-zero (the policy should increase the ratio). The min’s asymmetry allows the pull back (the good action’s decrease is reversed) and caps the push out (the good action’s increase is bounded).

WE-L.2.c — Faded: the multiple epochs’ safety

Setup. The PPO’s K = 10 epochs, the same rollout, the small ε = 0.2, the actor’s learning rate 3e-4.

Question. Does the ratio stay within the [0.8, 1.2] across the 10 epochs? What is the safety’s mechanism?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The ratio starts at the 1.0 (the first epoch, the same policy), and the gradient’s step moves the ratio slightly (the small 3e-4’s learning rate). The ratio stays close to the 1.0 for the first few epochs, and the clip’s bound caps the objective if the ratio moves outside the [0.8, 1.2]. The safety’s mechanism is the clip’s cap — the objective’s gradient is zero for the further movement outside, and the policy’s step is bounded.

The lesson. The multiple epochs’s safety is the clip’s cap — the ratio moves toward the band’s edge, and the clip stops the objective’s increase (the gradient zero). The 10 epochs reuse the same rollout (the sample efficiency), and the clip bounds the change (the trust region). The two together are the PPO’s essence.

WE-L.2.d — Contrastive near-miss: the clip without the min

Correct solution. The PPO’s min(r_t A_t, clip(r_t) A_t) — the pessimistic bound.

Near-miss. The objective clip(r_t) A_t (the clip alone, without the min).

Why the near-miss produces a plausible but incorrect number. The clip-alone objective allows the large ratio’s large objective — the gradient is non-zero for the r_t > 1 + ε (the advantage-positive), and the policy is incentivized to push the ratio further (the objective is clip(r_t) A_t = (1 + ε) A_t, the constant, but the gradient is the unclipped’s r_t A_t’s large gradient). The clip-alone doesn’t bound the objective’s gradient, and the ratio moves outside the band — the trust region is violated. The symptom is the unstable training (the large ratio, the divergence), and the fix is the min (the pessimistic bound).

The discrepancy, stated. The clip(r_t) A_t and the min(r_t A_t, clip(r_t) A_t) differ by the min — the clip-alone caps the value but not the gradient, and the min caps the gradient (the flat for the outside the band). The min is the key (the pessimism), and the clip-alone is the naive (the value’s cap without the gradient’s cap).

Self-explanation 6. Before reading on: in the near-miss, if the learning rate is small (the 1e-5), does the clip-alone work?

Answer. Partially — the small learning rate reduces the ratio’s movement, and the clip-alone’s ratio stays close to the 1.0 (the band’s interior). The clip-alone’s failure is the large learning rate (the ratio’s large movement), and the small rate avoids it. The min is needed for the robust training (the large rate’s safety), and the clip-alone with the small rate is the limited workaround.

Part C — Generalized Advantage Estimation (GAE)

§L.3 — Objects: the TD residual, the λ-weighted sum, and the bias-variance trade-off

The TD residual at the step t is:

It is the difference between the one-step TD target (r_t + γ V(s_{t+1})) and the current value estimate V(s_t). The δ_t’s expectation is the advantage A(s_t, a_t) (up to the bias from the value’s inaccuracy).

The GAE(λ) advantage is the exponentially-weighted sum of the TD residuals:

where λ ∈ [0, 1] is the GAE’s parameter (the bias-variance trade-off).

The special cases:

• λ = 0: the A^{GAE}_t = δ_t — the one-step TD residual (the low variance, the high bias from the value’s inaccuracy).
• λ = 1: the A^{GAE}_t = Σ_{l=0}^{∞} γ^l δ_{t+l} = G_t − V(s_t) (the telescoping sum) — the Monte Carlo advantage (the high variance, the unbiased if the value is exact at the terminal, but the value’s error is the bias).

The λ’s role: the interpolation between the TD and the MC, and the standard λ = 0.95 is the empirical balance (the moderate bias, the moderate variance).

The TD residual’s computation requires the value V(s_{t+1}), and the bootstrap’s boundary is the truncated’s state (the terminated’s V = 0, the truncated’s V(s_T)’s bootstrap) — the 1.2.1’s terminated/truncated distinction’s importance in the PPO’s implementation.

§L.3.1 — Result: the GAE’s λ interpolates the TD’s low variance high bias and the MC’s high variance low bias, and the standard λ = 0.95 balances the two — the advantage’s estimation is the policy gradient’s quality

Claim. The GAE(λ) advantage A^{GAE}_t = Σ_{l≥0} (γλ)^l δ_{t+l} satisfies A^{GAE}_t = (1 − λ) Σ_{l≥0} λ^l A^{(l)}_t (the λ-weighted sum of the l-step advantages), where A^{(l)}_t = Σ_{k=0}^{l-1} γ^k r_{t+k} + γ^l V(s_{t+l}) − V(s_t). The λ = 0 gives the one-step TD (the low variance, the high bias from the V’s error), the λ = 1 gives the MC (the high variance, the unbiased if the V is exact), and the intermediate λ balances the two.

Proof sketch. The telescoping of the (γλ)^l δ_{t+l} sum gives the (1 − λ) Σ λ^l A^{(l)}_t, and the weights (1 − λ) λ^l sum to the 1 — the λ-weighted average of the l-step advantages. The l-step advantage’s variance grows with the l (the longer the horizon, the more the random rewards), and the bias decreases with the l (the longer the horizon, the less the V’s error’s influence). The λ’s weights balance the two — the small λ weights the small l (the low variance, the high bias), and the large λ weights the large l (the high variance, the low bias).

Corollary. The λ = 0.95 is the standard because the bias and the variance are balanced for the typical tasks — the task’s horizon and the value’s accuracy determine the optimal λ, and the 0.95 is the empirical default (the 0.9 for the shorter horizons, the 0.99 for the longer).

Corollary. The GAE’s computation requires the value V(s_{t+1}) for each step, and the bootstrap’s boundary is the terminated’s V = 0 (the no future) and the truncated’s V(s_T) (the future’s bootstrap). The terminated/truncated’s distinction (the 1.2.1’s misconception M3) is critical for the GAE — the wrong bootstrap biases the advantage, and the policy learns the wrong behavior at the episode’s boundary.

§L.3.2 — Worked-example ladder

WE-L.3.a — Fully worked: the GAE for a 3-step trajectory

Input. A 3-step trajectory with the rewards [1, 2, 3], the values V(s_0) = 0.5, V(s_1) = 1.0, V(s_2) = 1.5, V(s_3) = 2.0 (the terminal), the γ = 0.9, the λ = 0.5.

Trace (the TD residuals).

1. δ_0 = r_0 + γ V(s_1) − V(s_0) = 1 + 0.9·1 − 0.5 = 1.4.
2. δ_1 = r_1 + γ V(s_2) − V(s_1) = 2 + 0.9·1.5 − 1 = 2.35.
3. δ_2 = r_2 + γ V(s_3) − V(s_2) = 3 + 0.9·2 − 1.5 = 3.3.

Trace (the GAE advantages).

4. A^{GAE}_2 = δ_2 = 3.3.
5. A^{GAE}_1 = δ_1 + γλ δ_2 = 2.35 + 0.9·0.5·3.3 = 2.35 + 1.485 = 3.835.
6. A^{GAE}_0 = δ_0 + γλ δ_1 + (γλ)² δ_2 = 1.4 + 0.45·2.35 + 0.2025·3.3 = 1.4 + 1.0575 + 0.668 = 3.1255.

Output and interpretation. The advantages [3.13, 3.84, 3.3] are the λ-weighted sums, and they are larger than the TD residuals [1.4, 2.35, 3.3] (the λ’s λ < 1’s weights and the future residuals’ addition). The λ = 0.5 is the middle (the moderate variance), and the λ = 1 would give the MC’s G_t − V(s_t) (the larger variance), and the λ = 0 would give the TD’s δ_t (the smaller variance).

Self-explanation 7. Before reading on: in step 6, the A^{GAE}_0 = 3.13 is larger than the δ_0 = 1.4 — what is the addition’s source?

Answer. The addition is the γλ δ_1 + (γλ)² δ_2 = 1.0575 + 0.668 = 1.7255, and the total is 1.4 + 1.7255 = 3.1255. The addition is the future residuals’ weighted contribution — the λ’s weights (0.45 and 0.2025), and the future residuals’ information (the steps 1 and 2’s δ). The GAE’s advantage is not just the current step’s residual, and it is the future’s information’s inclusion — the λ’s role.

Self-explanation 8. Before reading on: in step 1, the δ_0 = 1.4 — what is the V(s_1)’s contribution to the residual?

Answer. The V(s_1)’s contribution is the γ V(s_1) = 0.9 · 1.0 = 0.9 — the discounted future value’s inclusion. The residual δ_0 = r_0 + γ V(s_1) − V(s_0) compares the one-step TD target (r_0 + γ V(s_1) = 1.9) with the current value (V(s_0) = 0.5), and the difference is the surprise — the actual outcome’s excess over the expected (the 1.9 − 0.5 = 1.4). The TD residual’s interpretation is the surprise, and the GAE’s sum is the weighted sum of the surprises.

WE-L.3.b — Completion problem

Input. The same trajectory, but the λ = 1.0 (the MC).

Trace (final 25% as prompts).

1. A^{GAE}_2 = δ_2 = 3.3 (the same, since it is the last step). ⟨your turn⟩
2. A^{GAE}_1 = δ_1 + γ·1·δ_2 = 2.35 + 0.9·3.3 = 2.35 + 2.97 = 5.32. ⟨your turn⟩
3. A^{GAE}_0 = δ_0 + γ δ_1 + γ² δ_2 = 1.4 + 0.9·2.35 + 0.81·3.3 = 1.4 + 2.115 + 2.673 = 6.188. ⟨your turn⟩
4. The MC’s advantage G_0 − V(s_0) = (1 + 0.9·2 + 0.81·3) − 0.5 = 5.23 − 0.5 = 4.73… wait, this doesn’t match 6.188. Let me recompute. G_0 = r_0 + γ r_1 + γ² r_2 = 1 + 0.9·2 + 0.81·3 = 1 + 1.8 + 2.43 = 5.23. The G_0 − V(s_0) = 5.23 − 0.5 = 4.73. But the A^{GAE}_0 with λ = 1 should equal the G_0 − V(s_0) by the telescoping. Let me check: the δ_0 + γ δ_1 + γ² δ_2 = 1.4 + 0.9·2.35 + 0.81·3.3 = 1.4 + 2.115 + 2.673 = 6.188. Hmm, the telescoping should give 4.73. Let me verify: δ_0 + γ δ_1 + γ² δ_2 = (r_0 + γ V_1 − V_0) + γ(r_1 + γ V_2 − V_1) + γ²(r_2 + γ V_3 − V_2) = r_0 + γ r_1 + γ² r_2 + γ V_1 − V_0 + γ² V_2 − γ V_1 + γ³ V_3 − γ² V_2 = G_0 − V_0 + γ³ V_3. So the A^{GAE}_0 = G_0 − V_0 + γ³ V_3. With the V_3 = 2.0 and the γ = 0.9, γ³ = 0.729, γ³ V_3 = 1.458. So the A^{GAE}_0 = 5.23 − 0.5 + 1.458 = 6.188. ✓ The extra γ³ V_3 is the terminal’s value’s contribution — the bootstrap at the terminal (if the V_3 ≠ 0).
5. The terminal’s value V_3 = 2.0 (the non-zero) adds the γ³ V_3 = 1.458 to the GAE, and the MC’s G_0 − V_0 = 4.73 is smaller — the terminal’s bootstrap is the difference.
6. The terminated’s correct handling is the V_3 = 0 (the no future), and the MC’s G_0 − V_0 = 4.73 is the correct advantage. The truncated’s correct handling is the V_3 ≠ 0 (the bootstrap), and the GAE’s 6.188 is the correct. The two are different (the terminated vs. the truncated), and the 1.2.1’s distinction is the key.

Answers.

1. 3.3. ✓
2. 5.32. ✓
3. 6.188 (with the terminal’s V_3 = 2.0). ✓
4. The MC’s 4.73 (the no terminal’s value), the GAE’s 6.188 (the terminal’s bootstrap), the two differ by the γ³ V_3 = 1.458. ✓
5. The terminal’s value’s contribution. ✓
6. The terminated’s V = 0, the truncated’s V = V(s_T). ✓

The lesson. The GAE’s λ = 1 (the MC) with the terminal’s non-zero V_3 adds the γ^T V_T to the advantage, and the MC’s true G_0 − V_0 is the smaller (the V_T = 0 for the terminated). The terminated/truncated’s distinction (the 1.2.1’s M3) is critical for the GAE’s correctness, and the wrong bootstrap biases the advantage by the γ^T V_T’s magnitude.

WE-L.3.c — Faded: the λ’s effect

Setup. The same trajectory, the λ = 0 (the TD).

Question. What are the GAE advantages for the λ = 0?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The A^{GAE}_t = δ_t for the all t (the λ = 0), so the [δ_0, δ_1, δ_2] = [1.4, 2.35, 3.3]. The TD’s advantages are the smallest (the low variance), and the bias is the high (the V’s error).

The lesson. The λ = 0 (the TD) is the low variance high bias, and the λ = 1 (the MC) is the high variance low bias. The λ = 0.5 (the intermediate) balances the two. The choice is the task’s trade-off, and the standard 0.95 is the empirical balance for the typical tasks.

WE-L.3.d — Contrastive near-miss: the wrong bootstrap

Correct solution. The terminated’s V(s_T) = 0 (the no future), the truncated’s V(s_T) = V(s_T) (the bootstrap).

Near-miss. The terminated’s V(s_T) = V(s_T) (the wrong bootstrap), the truncated’s V(s_T) = 0 (the wrong no-bootstrap).

Why the near-miss produces a plausible but incorrect number. The terminated’s wrong V(s_T) = V(s_T) adds the γ^T V(s_T) to the advantage (the positive), and the policy learns the wrong behavior at the episode’s end — the terminal’s value is inflated, and the policy thinks the terminal is better than it is. The symptom is the policy’s hovering at the terminal (the inflated value attracts the policy), and the fix is the terminated’s correct V = 0. The truncated’s wrong V = 0 loses the future’s bootstrap, and the policy thinks the truncated is worse than it is. The symptom is the short-sighted behavior (the policy avoids the time-limit’s approach), and the fix is the truncated’s correct V = V(s_T).

The discrepancy, stated. The terminated’s V = 0 and the truncated’s V = V(s_T) differ by the bootstrap’s presence — the terminated’s no future (the V = 0), the truncated’s future (the V = V(s_T)). The conflation (the terminated with the truncated’s V, or vice versa) biases the advantage by the γ^T V(s_T)’s magnitude, and the policy learns the wrong behavior at the episode’s boundary.

Self-explanation 9. Before reading on: in the near-miss, if the truncated’s V(s_T) is close to the 0, does the wrong no-bootstrap still matter?

Answer. Less — if the V(s_T) ≈ 0, the bootstrap’s contribution is small, and the wrong no-bootstrap is close to the correct (the V = V(s_T) ≈ 0 is similar to the V = 0). The bootstrap’s contribution is the γ^T V(s_T), and the magnitude determines the error — the small V(s_T) gives the small error, and the large V(s_T) gives the large. The correctness depends on the value’s magnitude at the truncated’s state, and the wrong handling’s cost is the γ^T V(s_T).

Part D — The PPO Loop and the Implementation

§L.4 — Objects: the rollout, the multiple epochs, the losses, and the hyperparameters

The PPO loop:

1. Collect a rollout of N_envs × T_horizon transitions by running the current policy π_θ in the N_envs parallel ArmEnvs (the 1.2.1’s N_envs’s parallelism).
2. Compute the GAE advantages A_t and the value targets V_target_t (the A_t + V(s_t) for the value’s regression).
3. For the K epochs:
   1. Shuffle the rollout’s N = N_envs × T_horizon transitions.
   2. For each minibatch of M transitions:
      1. Compute the ratio r_t(θ) = π_θ(a_t | s_t) / π_{θ_old}(a_t | s_t).
      2. Compute the clipped objective L^{CLIP}.
      3. Compute the value loss L^{VF} = (V_φ(s_t) − V_target_t)².
      4. Compute the entropy H[π_θ(· | s_t)].
      5. Total loss: L = −L^{CLIP} + c_1 L^{VF} − c_2 H (the c_1 = 0.5, the c_2 = 0.01).
      6. Backpropagate the L’s gradient to θ and φ.
      7. Clip the gradient to the max_norm = 0.5 (the stability).
      8. Optimizer’s step (the Adam, the lr = 3e-4).
4. Discard the old rollout (the new rollout from the updated policy).
5. Repeat until the budget is exhausted or the return converges.

The hyperparameters:

• ε = 0.2 — the clip’s range (the trust region’s radius).
• γ = 0.99 — the reward’s discount (the task’s horizon).
• λ = 0.95 — the GAE’s bias-variance trade-off.
• K = 10 — the epochs (the rollout’s reuse).
• M = 32 — the minibatch’s size.
• lr = 3e-4 — the learning rate (the Adam’s standard).
• c_1 = 0.5 — the value loss’s coefficient.
• c_2 = 0.01 — the entropy’s bonus’s coefficient.
• max_norm = 0.5 — the gradient’s clip’s norm.
• N_envs = 8 — the parallel environments.
• T_horizon = 128 — the rollout’s length per environment (the N = 8 · 128 = 1024 transitions per iteration).

The rationale for each is the empirical balance, and the tuning depends on the task (the longer horizon for the larger γ’s task, the larger K for the smaller ε’s conservative clip).

§L.4.1 — Result: the PPO loop’s multiple epochs reuse the rollout and the clip bounds the policy’s change, and the value loss’s coefficient must balance the actor’s gradient — the entropy’s bonus maintains the exploration

Claim. The PPO’s multiple epochs (the K = 10) reuse the same rollout and the same transitions, and the sample efficiency is the K× the vanilla policy gradient’s. The clip bounds the ratio’s movement (the ε = 0.2), and the multiple epochs are safe (the policy’s change is bounded). The value loss’s coefficient c_1 = 0.5 balances the critic’s gradient and the actor’s, and the entropy’s bonus c_2 = 0.01 maintains the exploration (the policy’s std doesn’t collapse).

Proof sketch. The multiple epochs’s sample efficiency is the K× (the K gradient’s steps per rollout, the K× the samples’ use). The clip’s bound prevents the ratio’s large movement (the trust region’s enforcement), and the multiple epochs’s safety is the consequence. The value loss’s coefficient 0.5 matches the actor’s gradient’s scale (the value’s MSE’s gradient is larger than the actor’s policy gradient’s, and the 0.5 scales it down), and the entropy’s bonus 0.01 adds the exploration’s incentive (the negative of the entropy’s loss, the policy’s std stays large).

Corollary. The value loss’s coefficient’s mismatch (the c_1 = 1.0 or the c_1 = 0.1) unbalances the actor and the critic — the large c_1 makes the critic’s gradient dominate (the actor stops learning), and the small c_1 makes the critic underfit (the advantage’s quality drops). The 0.5 is the empirical balance, and the task’s tuning is the refinement.

Corollary. The entropy’s bonus’s absence (c_2 = 0) collapses the policy’s std (the policy converges to the deterministic, the exploration stops, the local optimum), and the symptom is the stagnant return (the policy stuck). The 0.01 is the small bonus, and the larger c_2 maintains more exploration (the slower convergence, the higher return for the multimodal tasks).

§L.4.2 — Worked-example ladder

WE-L.4.a — Fully worked: the PPO’s iteration for the ArmEnv

Input. The ArmEnv with the N_envs = 8, the T_horizon = 128, the K = 10, the M = 32, the ε = 0.2, the γ = 0.99, the λ = 0.95.

Trace (the iteration).

1. The rollout: 8 × 128 = 1024 transitions per iteration.
2. The GAE advantages: the A_t for each of the 1024 transitions.
3. The value targets: the V_target_t = A_t + V(s_t).
4. The K = 10 epochs × the 1024/32 = 32 minibatches = the 320 gradient’s steps per iteration.
5. The losses: the L^{CLIP}, the L^{VF}, the H, the total L.
6. The gradient’s clip: the max_norm = 0.5.
7. The optimizer’s step (the Adam, the 3e-4).

Output and interpretation. The iteration processes the 1024 transitions in the 320 gradient’s steps (the K× the rollout’s use), and the clip bounds the ratio’s movement. The sample efficiency is the 320/1024 = 0.3125 gradient’s steps per transition, and the vanilla policy gradient’s 1/1024 = 0.00098 is the 320× less — the PPO’s advantage.

Self-explanation 10. Before reading on: in step 4, the 320 gradient’s steps per iteration — is this the PPO’s sample efficiency’s source?

Answer. Yes — the multiple epochs (the K = 10) reuse the same rollout, and the 320 gradient’s steps per 1024 transitions are the PPO’s sample efficiency. The vanilla policy gradient’s single step per rollout is the 320× less efficient (the same data, the fewer updates), and the PPO’s advantage is the reuse’s efficiency. The clip’s bound makes the reuse safe (the policy’s change is bounded), and the two together are the PPO’s essence.

WE-L.4.b — Completion problem

Input. The same ArmEnv, but the K = 1 (the single epoch) and the K = 20 (the many epochs).

Trace (final 25% as prompts).

1. The K = 1’s sample efficiency is the vanilla policy gradient’s (1024/32 = 32 gradient’s steps per 1024 transitions), and the learning is slow. ⟨your turn⟩
2. The K = 20’s sample efficiency is the 2× the K = 10’s, but the ratio’s movement is larger (the more gradient’s steps), and the clip’s cap is needed more often. ⟨your turn⟩
3. The optimal K depends on the task — the smaller K for the unstable tasks, the larger for the stable. ⟨your turn⟩

Answers.

1. 32 gradient’s steps, the slow learning. ✓
2. The 2× efficiency, the larger ratio’s movement, the more frequent clip. ✓
3. The optimal K depends on the task. ✓

The lesson. The K’s trade-off — the larger K is the more efficient (the more gradient’s steps per rollout), but the ratio’s movement is larger (the more epochs’ accumulated change), and the clip’s cap is needed more often. The standard K = 10 is the empirical balance, and the task’s tuning is the refinement.

WE-L.4.c — Faded: the value loss’s coefficient

Setup. The same ArmEnv, the c_1 = 0.5 (the standard) vs. the c_1 = 5.0 (the large).

Question. What is the actor’s learning’s difference?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The large c_1 = 5.0 makes the critic’s gradient 10× the actor’s, and the shared backbone (if the actor and the critic share the network) is dominated by the critic’s gradient. The actor’s learning is slow (the shared backbone’s features are optimized for the value, not the policy). The symptom is the slow policy improvement and the value’s convergence (the critic wins). The fix is the separate networks (the actor and the critic independent) or the smaller c_1 (the 0.5).

The lesson. The value loss’s coefficient c_1 balances the actor and the critic, and the large c_1 dominates the shared backbone (the actor’s learning suffers). The separate networks avoid the issue, and the shared networks need the balanced c_1 = 0.5. The PPO’s standard is the separate networks, and the shared is the samples’ efficiency’s trade-off.

WE-L.4.d — Contrastive near-miss: the no gradient clip

Correct solution. The gradient’s clip (max_norm = 0.5).

Near-miss. The no gradient’s clip (the raw gradient).

Why the near-miss produces a plausible but incorrect number. The raw gradient’s large norm (the rare large advantage, the surprising transition) moves the policy’s parameters huge, and the ratio jumps outside the clip’s band (the clip’s cap is saturated). The symptom is the occasional large jump (the loss’s spike, the return’s drop), and the training is unstable. The gradient’s clip bounds the step, and the stability is the result.

The discrepancy, stated. The raw gradient and the clipped gradient differ by the norm’s bound (0.5), and the large gradient is scaled to the 0.5 (the norm’s preservation), and the step’s size is bounded. The clip’s cap is the stability’s mechanism, and its absence is the spike’s source.

Self-explanation 11. Before reading on: in the near-miss, if the advantage’s magnitude is small (the typical case), does the gradient’s clip matter?

Answer. Less — if the advantage’s magnitude is small, the gradient’s norm is below the 0.5, and the clip is inactive (the identity). The clip’s matter is the rare large gradient (the large advantage, the surprising transition), and the clip bounds the rare large step. The clip’s effect is the tail’s robustness, not the typical’s behavior — the small gradient is unchanged, and the large gradient is capped.

Part E — Failure Taxonomy, Lab, Cheat Sheet, Exercises, Exit Test

§L.5 — Failure taxonomy

§L.6 — The lab

§L.6.1 — Task

Implement the PPO for the ArmEnv, with the clipped objective, the GAE, the multiple epochs, the value loss, the entropy’s bonus, and the gradient’s clip; verify the convergence and the hyperparameters’ effect; break the PPO four ways and diagnose.

§L.6.2 — Conditions

Python 3.10+, PyTorch 2.2+, CPU. Seed 42, dtype = float64. Runtime under 10 minutes. The ArmEnv with the 2D position, the Gaussian policy (nn.Linear(4, 64) → mean, nn.Parameter for the log-std), the separate critic (nn.Linear(4, 64) → 1), the rollout N = 1024 (the N_envs = 8, the T_horizon = 128), the K = 10, the M = 32, the ε = 0.2, the γ = 0.99, the λ = 0.95, the lr = 3e-4, the c_1 = 0.5, the c_2 = 0.01, the max_norm = 0.5.

§L.6.3 — Controls held fixed

The seed, dtype, the ArmEnv, the policy’s architecture, the hyperparameters are fixed. The variants are the ε = 0 (the no clip), the λ = 1 (the MC), the K = 1 (the single epoch), and the c_2 = 0 (the no entropy).

§L.6.4 — Metrics table

§L.6.5 — Pre-registered hypotheses

Each variant runs for 100 iterations and reports the mean of its last ten deterministic evaluations. The values below are measured, not intended.

• H1. The standard PPO (ε = 0.2, λ = 0.95, K = 10) converges to return > -0.5 within ~200 iterations. Measured: −0.0940 at 100 iterations, with a clip fraction of 0.222. Holds.
• H2. ε = 0 (no clip) diverges — the return decreases or oscillates — while ε = 0.2 is stable. Measured: no-clip −0.0039 against standard −0.0940. Refuted. Removing the clip improved the return. The mechanism the lesson describes is real, but it needs a task where the unclipped ratio actually moves far from 1 between the rollout and the update; on this 2-D point mass the policy barely changes within a batch, so the clip only truncates useful gradient. The clip fraction of 0.222 shows the clip is engaging — this is not a case of a switch that does nothing, but of a constraint that costs more than it saves here.
• H3. K = 10 is ~10× more sample-efficient than K = 1. Measured: K = 10 reaches −0.0940, K = 1 reaches −0.3593 — a gap of 0.265 return for the same number of environment steps. Holds in direction, and the gap is large, but the “10×” is not a ratio the lab computes: both variants consume the same number of env steps and differ only in reuse, so the correct statement is that the extra nine epochs buy 0.265 return on a fixed sample budget.
• H4. c_2 = 0 collapses the policy std and the return stalls; c_2 = 0.01 maintains exploration. Measured: no-entropy −0.4989 against standard −0.0940. Holds on the return; the std collapse itself is not printed by the lab.
• H5. λ = 1 (the Monte-Carlo advantage) is noisier than λ = 0.95. Measured: λ = 1 reaches −2.1154 against −0.0940 — 22× worse, far beyond a noise difference. Holds, and more strongly than the hypothesis claims: this is not merely a noisier curve but a failure to learn, because a Monte-Carlo advantage on a 10-step horizon with a dense distance reward has variance that swamps the signal.

§L.6.6 — The break-it sequence

1. The clip’s min missing (silent). The L = r_t · A_t (the no clip, the no min) with the K = 10. Predicted symptom: the training diverges. Actual: no divergence — the return reaches −0.0039, better than standard. Silent, but the predicted damage is absent, not merely quiet.
2. The GAE’s λ = 1 (silent). The λ = 1 (the MC advantage). Predicted symptom: the variance is large, and the sample efficiency drops. Actual: as predicted — the return falls to −2.1154. Silent.
3. The K = 1 (silent). The single epoch. Predicted symptom: the sample efficiency drops. Actual: as predicted — −0.3593 against −0.0940. Silent.
4. The c_2 = 0 (silent). The no entropy’s bonus. Predicted symptom: the return stalls. Actual: as predicted — −0.4989. Silent.
5. The terminated’s wrong bootstrap (silent). A terminated step bootstraps from V(s') of the next episode instead of 0. Predicted symptom: the policy hovers at the terminal, crediting value it never reached. Actual: the return falls to −0.6809, a 0.587 loss, and nothing raises. Silent, and the most expensive of the five.

The bootstrap break-it is worth dwelling on: it is the bug this lab’s own GAE had before the compute_gae repair. The original inline loop zeroed the carry on done but still added gamma * next_val from the next episode’s first value, which is exactly the wrong-bootstrap above. It cost the standard run 0.50 return (−0.5946 before the repair, −0.0940 after) and produced no error, no warning, and no NaN — only a policy that learned more slowly than it should have.

§L.6.7 — Acceptance criteria

§L.6.8 — What good looks like

A strong submission has five properties:

1. The clipped objective is implemented with the ratio, the clip, and the min, and the pessimism is demonstrated (the clip-fraction’s value).
2. The GAE is implemented with the correct λ and the correct terminated/truncated bootstrap.
3. The multiple epochs are implemented with the K = 10 and the shuffled minibatches, and the sample efficiency is measured.
4. The value loss and the entropy are balanced (c_1 = 0.5, c_2 = 0.01), and the actor’s learning is demonstrated (the std’s evolution, the return’s growth).
5. The five break-its are predicted with the symptoms (the divergence, the high variance, the low efficiency, the std collapse, the hovering) before running.

§L.6.9 — Runnable lab code

# lab_1_2_4.py — PPO with the clipped objective, the GAE, and the break-its
# Requires: Python 3.10+, PyTorch 2.2+

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
from torch.distributions import Normal

torch.manual_seed(42); dtype, device = torch.float64, torch.device("cpu")

# --- The SimpleArmEnv (2D position, episodic) --------------------------
class SimpleArmEnv:
    def __init__(self, max_steps=10):
        self.p_target = torch.tensor([0.5, 0.2], dtype=dtype)
        self.max_steps = max_steps
        self.state_dim = 2; self.action_dim = 2
    def reset(self):
        self.pos = torch.randn(2, dtype=dtype) * 0.1
        self.step_count = 0
        return self.pos.clone()
    def step(self, a):
        self.pos = self.pos + a * 0.1
        dist = torch.norm(self.pos - self.p_target).item()
        r = -dist; self.step_count += 1
        terminated = dist < 0.05
        truncated = self.step_count >= self.max_steps
        return self.pos.clone(), r, terminated, truncated

# --- The Actor-Critic -----------------------------------------------
class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.actor = nn.Sequential(
            nn.Linear(state_dim, 64), nn.Tanh(),
            nn.Linear(64, action_dim),
        )
        self.critic = nn.Sequential(
            nn.Linear(state_dim, 64), nn.Tanh(),
            nn.Linear(64, 1),
        )
        self.log_std = nn.Parameter(torch.ones(action_dim) * -1.0)
    def forward(self, s):
        mu = self.actor(s)
        std = self.log_std.exp()
        return Normal(mu, std), self.critic(s).squeeze(-1)

# --- PPO hyperparameters ---------------------------------------------
N_ENVS = 8; T_HORIZON = 128; K_EPOCHS = 10; MINIBATCH = 32
EPS_CLIP = 0.2; GAMMA = 0.99; LAMBDA = 0.95
LR = 3e-4; C1 = 0.5; C2 = 0.01; MAX_NORM = 0.5

def compute_gae(rewards, values, dones, truncateds, boot_val, last_val,
                gamma=GAMMA, lam=LAMBDA, wrong_bootstrap=False):
    """GAE with the terminated/truncated distinction.

    rewards, values, dones, truncateds, boot_val are (T, N_ENVS).
    `boot_val[t][i]` is V(s_T) for an episode truncated at step t -- the value of
    the terminal observation itself, captured before the env was reset, because
    the rollout's `values[t + 1]` belongs to the *next* episode.
    A terminated step bootstraps from 0; a truncated step bootstraps from
    boot_val, because a time limit cuts the episode without ending the task.

    `wrong_bootstrap=True` is the break-it: a terminated step bootstraps from
    the critic's own estimate of the *next* episode's first observation instead
    of 0, which credits the agent for value it never reached.
    """
    T = rewards.shape[0]
    advantages = torch.zeros(T, N_ENVS, dtype=dtype)
    gae = torch.zeros(N_ENVS, dtype=dtype)
    for t in reversed(range(T)):
        next_val = last_val if t == T - 1 else values[t + 1]
        next_val = torch.where(truncateds[t], boot_val[t], next_val)
        if not wrong_bootstrap:
            next_val = torch.where(dones[t], torch.zeros_like(next_val), next_val)
        delta = rewards[t] + gamma * next_val - values[t]
        gae = delta + gamma * lam * torch.where(dones[t], torch.zeros_like(gae), gae)
        advantages[t] = gae
    return advantages

def ppo_iteration(actor_critic, optimizer, envs, use_clip=True, use_min=True,
                  lam=LAMBDA, k_epochs=K_EPOCHS, c2=C2, wrong_bootstrap=False):
    # Collect rollout
    obs_list = []; act_list = []; rew_list = []; done_list = []; trunc_list = []; val_list = []; logp_list = []
    boot_list = []
    obs = torch.stack([envs[i].reset() for i in range(N_ENVS)])
    for t in range(T_HORIZON):
        with torch.no_grad():
            dist, val = actor_critic(obs)
            act = dist.sample()
            logp = dist.log_prob(act).sum(-1)
        new_obs_list = []; rew_arr = torch.zeros(N_ENVS, dtype=dtype)
        done_arr = torch.zeros(N_ENVS, dtype=torch.bool); trunc_arr = torch.zeros(N_ENVS, dtype=torch.bool)
        boot_arr = torch.zeros(N_ENVS, dtype=dtype)
        for i in range(N_ENVS):
            new_obs, r, done, trunc = envs[i].step(act[i])
            rew_arr[i] = r; done_arr[i] = done; trunc_arr[i] = trunc
            if trunc:
                # V(s_T) at the terminal observation, before the reset discards it.
                with torch.no_grad():
                    _, boot_arr[i] = actor_critic(new_obs.unsqueeze(0))
            if done or trunc:
                new_obs = envs[i].reset()
            new_obs_list.append(new_obs)
        obs_list.append(obs); act_list.append(act); rew_list.append(rew_arr)
        done_list.append(done_arr); trunc_list.append(trunc_arr); val_list.append(val); logp_list.append(logp)
        boot_list.append(boot_arr)
        obs = torch.stack(new_obs_list)
    # GAE
    with torch.no_grad():
        _, last_val = actor_critic(obs)
    advantages = compute_gae(torch.stack(rew_list), torch.stack(val_list),
                             torch.stack(done_list), torch.stack(trunc_list),
                             torch.stack(boot_list), last_val, lam=lam,
                             wrong_bootstrap=wrong_bootstrap)
    returns = advantages + torch.stack(val_list)
    # Flatten
    obs_flat = torch.stack(obs_list).view(-1, 2)
    act_flat = torch.stack(act_list).view(-1, 2)
    logp_old_flat = torch.stack(logp_list).view(-1)
    adv_flat = advantages.view(-1)
    ret_flat = returns.view(-1)
    adv_flat = (adv_flat - adv_flat.mean()) / (adv_flat.std() + 1e-8)
    # Multiple epochs
    N = obs_flat.shape[0]
    clip_fracs = []
    for _ in range(k_epochs):
        idx = torch.randperm(N)
        for start in range(0, N, MINIBATCH):
            mb = idx[start:start + MINIBATCH]
            dist, val = actor_critic(obs_flat[mb])
            logp = dist.log_prob(act_flat[mb]).sum(-1)
            entropy = dist.entropy().sum(-1).mean()
            ratio = (logp - logp_old_flat[mb]).exp()
            surr1 = ratio * adv_flat[mb]
            surr2 = torch.clamp(ratio, 1 - EPS_CLIP, 1 + EPS_CLIP) * adv_flat[mb]
            if use_clip and use_min:
                policy_loss = -torch.min(surr1, surr2).mean()
            elif use_clip and not use_min:
                policy_loss = -surr2.mean()  # clip without min
            else:
                policy_loss = -surr1.mean()  # no clip
            value_loss = F.mse_loss(val, ret_flat[mb])
            loss = policy_loss + C1 * value_loss - c2 * entropy
            optimizer.zero_grad(); loss.backward()
            torch.nn.utils.clip_grad_norm_(actor_critic.parameters(), MAX_NORM)
            optimizer.step()
            if use_clip:
                with torch.no_grad():
                    clip_fracs.append(((ratio - 1).abs() > EPS_CLIP).float().mean().item())
    return advantages.mean().item(), advantages.std().item(), np.mean(clip_fracs) if clip_fracs else 0.0

# --- Training loop --------------------------------------------------
def train_ppo(variant="standard", n_iter=200):
    envs = [SimpleArmEnv() for _ in range(N_ENVS)]
    ac = ActorCritic(2, 2).to(device, dtype)
    opt = torch.optim.Adam(ac.parameters(), lr=LR)
    returns = []; clip_fracs = []
    for it in range(n_iter):
        if variant == "standard":
            adv_mean, adv_std, cf = ppo_iteration(ac, opt, envs)
        elif variant == "no_clip":
            adv_mean, adv_std, cf = ppo_iteration(ac, opt, envs, use_clip=False)
        elif variant == "clip_no_min":
            adv_mean, adv_std, cf = ppo_iteration(ac, opt, envs, use_min=False)
        elif variant == "lambda_1":
            adv_mean, adv_std, cf = ppo_iteration(ac, opt, envs, lam=1.0)
        elif variant == "k1":
            adv_mean, adv_std, cf = ppo_iteration(ac, opt, envs, k_epochs=1)
        elif variant == "no_entropy":
            adv_mean, adv_std, cf = ppo_iteration(ac, opt, envs, c2=0.0)
        elif variant == "wrong_bootstrap":
            adv_mean, adv_std, cf = ppo_iteration(ac, opt, envs, wrong_bootstrap=True)
        # Evaluate
        with torch.no_grad():
            eval_returns = 0.0
            for _ in range(5):
                s = envs[0].reset(); done = False; trunc = False
                while not (done or trunc):
                    dist, _ = ac(s)
                    a = dist.mean  # deterministic
                    s, r, done, trunc = envs[0].step(a)
                    eval_returns += r
            returns.append(eval_returns / 5)
            clip_fracs.append(cf)
    return returns, clip_fracs

print("=== Standard PPO ===")
rets_std, cf_std = train_ppo("standard", n_iter=100)
print(f"final return: {np.mean(rets_std[-10:]):.4f}, clip fraction: {np.mean(cf_std):.3f}")

print("\n=== No clip ===")
rets_nc, _ = train_ppo("no_clip", n_iter=100)
print(f"final return: {np.mean(rets_nc[-10:]):.4f}")

print("\n=== Clip without min ===")
rets_cnm, _ = train_ppo("clip_no_min", n_iter=100)
print(f"final return: {np.mean(rets_cnm[-10:]):.4f}")

print("\n=== Lambda = 1 (MC) ===")
rets_l1, _ = train_ppo("lambda_1", n_iter=100)
print(f"final return: {np.mean(rets_l1[-10:]):.4f}")

print("\n=== K = 1 ===")
rets_k1, _ = train_ppo("k1", n_iter=100)
print(f"final return: {np.mean(rets_k1[-10:]):.4f}")

print("\n=== No entropy ===")
rets_ne, _ = train_ppo("no_entropy", n_iter=100)
print(f"final return: {np.mean(rets_ne[-10:]):.4f}")

print("\n=== Wrong terminal bootstrap (break-it 5) ===")
rets_wb, _ = train_ppo("wrong_bootstrap", n_iter=100)
print(f"final return: {np.mean(rets_wb[-10:]):.4f}")

# --- Summary -------------------------------------------------------
std = np.mean(rets_std[-10:])
print("\n--- lab 1.2.4 summary ---")
print(f"H1: standard final return = {std:.4f} (target > -0.5)")
print(f"H2: no-clip = {np.mean(rets_nc[-10:]):.4f} vs standard {std:.4f} (predicted: diverges)")
print(f"H3: K=10 vs K=1: {std:.4f} vs {np.mean(rets_k1[-10:]):.4f} (predicted: K=1 ~10x worse)")
print(f"H4: no-entropy = {np.mean(rets_ne[-10:]):.4f} (predicted: worse)")
print(f"H5: λ=1 = {np.mean(rets_l1[-10:]):.4f} (predicted: noisier)")
print(f"B5: wrong terminal bootstrap = {np.mean(rets_wb[-10:]):.4f} vs standard {std:.4f} "
      f"(predicted: silent, hovering)")

§L.7 — The cheat sheet

§L.8 — Exercises with solutions

E 1.2.4.1 — The clipped objective (numerical). The A_t = 0.5, the r_t = 1.5, the ε = 0.2. Compute the min(r_t A_t, clip(r_t) A_t).

S 1.2.4.1. The clip(1.5, 0.8, 1.2) = 1.2. The r_t A_t = 0.75, the clip A_t = 0.6. The min is 0.6. The lesson: the good action’s large ratio is capped at the 1.2, and the objective is the pessimistic 0.6.

E 1.2.4.2 — The GAE (numerical). A 2-step trajectory with the rewards [1, 2], the values V(s_0) = 0.5, V(s_1) = 1.0, V(s_2) = 0 (the terminal), the γ = 0.9, the λ = 0.95. Compute the GAE advantages.

S 1.2.4.2. The δ_0 = 1 + 0.9·1 − 0.5 = 1.4, the δ_1 = 2 + 0.9·0 − 1 = 1.0. The A^{GAE}_1 = 1.0, the A^{GAE}_0 = 1.4 + 0.9·0.95·1.0 = 1.4 + 0.855 = 2.255. The lesson: the terminal’s V(s_2) = 0 (the terminated), and the GAE’s λ = 0.95 weights the future’s residual at the 0.855.

E 1.2.4.3 — The ratio’s value (conceptual). The π_θ(a|s) = 0.3, the π_{θ_old}(a|s) = 0.2. What is the ratio? What is the clip’s value with the ε = 0.2?

S 1.2.4.3. The ratio r_t = 0.3/0.2 = 1.5. The clip clip(1.5, 0.8, 1.2) = 1.2. The lesson: the ratio is the new/old’s probability, and the 1.5 is outside the band (the clip’s 1.2).

E 1.2.4.4 — The λ’s effect (conceptual). The λ = 0 vs. the λ = 1 — which has the lower variance? Which has the lower bias?

S 1.2.4.4. The λ = 0 (the TD) has the lower variance (the one-step’s residual) and the higher bias (the V’s error’s influence). The λ = 1 (the MC) has the higher variance (the full trajectory’s randomness) and the lower bias (the V’s error’s smaller influence). The lesson: the λ interpolates the two, and the 0.95 balances them.

E 1.2.4.5 — The multiple epochs (conceptual). The K = 10 vs. the K = 1. What is the sample efficiency’s ratio?

S 1.2.4.5. The K = 10 reuses the rollout 10×, and the sample efficiency is the 10× the K = 1’s (the same return with the 10× fewer iterations). The lesson: the multiple epochs are the PPO’s sample efficiency’s source, and the clip’s bound makes the reuse safe.

E 1.2.4.6 — The terminated’s wrong bootstrap (diagnostic). The policy hovers at the terminal (the inflated value). What is the cause?

S 1.2.4.6. The wrong bootstrap (the terminated’s V(s_T) = V(s_T) instead of the 0) inflates the terminal’s value, and the policy thinks the terminal is better than it is — the hovering is the inflated value’s attraction. The fix is the terminated’s V = 0. The lesson: the terminated/truncated’s distinction (the 1.2.1’s M3) is critical for the GAE’s correctness, and the wrong bootstrap biases the advantage.

E 1.2.4.7 — The entropy’s collapse (diagnostic). The c_2 = 0 (the no entropy) collapses the std to the tiny, and the return stalls. What is the cause, and what is the fix?

S 1.2.4.7. The no entropy’s bonus allows the policy converging to the deterministic (the std → 0), and the exploration stops. The policy stuck at the local optimum (the stagnant return). The fix is the larger c_2 (the 0.01), which maintains the exploration. The lesson: the entropy’s bonus is the exploration’s incentive, and its absence is the local optimum’s source.

E 1.2.4.8 — The GAE’s implementation (constructive, transfer probe). Write the GAE’s pseudocode with the correct terminated/truncated bootstrap.

S 1.2.4.8.

advantages = zeros(T)
gae = 0
for t in reversed(range(T)):
    if terminated[t]:
        next_val = 0
        gae = 0  # reset the GAE at the terminal
    elif truncated[t]:
        next_val = V(s_{t+1})  # bootstrap
    else:
        next_val = V(s_{t+1})
    delta = r[t] + gamma * next_val - V(s[t])
    gae = delta + gamma * lambda * gae
    if terminated[t]:
        gae = delta  # reset after
    advantages[t] = gae
returns = advantages + V(s)

The terminated’s next_val = 0 and the gae’s reset (the no future’s bootstrap), and the truncated’s next_val = V(s_{t+1}) (the bootstrap). The lesson: the two are different (the 0 vs. the V), and the distinction is the GAE’s correctness.

§L.9 — Exit test

You are ready for Lesson 1.2.5 (SAC) when you can, without notes:

1. State the trust-region motivation and the KL’s constraint.
2. State the PPO’s clipped objective with the ratio, the clip, and the min.
3. Explain the min’s pessimism (the gradient’s zero for the advantage-positive and ratio > 1+ε).
4. Compute the clipped objective by hand for a specific A_t and r_t.
5. State the GAE(λ)‘s formula and the TD residual.
6. State the λ’s bias-variance trade-off (λ = 0 the TD, λ = 1 the MC).
7. State the terminated/truncated’s bootstrap (V = 0 vs. V = V(s_T)).
8. State the PPO loop’s hyperparameters (ε = 0.2, λ = 0.95, K = 10, M = 32, lr = 3e-4, c_1 = 0.5, c_2 = 0.01).
9. State the entropy’s bonus’s role and its absence’s symptom (the std collapse).
10. Diagnose the no-clip’s divergence, the λ = 1’s high variance, the K = 1’s low efficiency, the no-entropy’s stall, and the wrong bootstrap’s hovering from their symptoms.
11. State the value loss’s coefficient’s balance and its mismatch’s symptom (the actor’s slow learning).
12. State the gradient’s clip’s role and its absence’s symptom (the spike).

If any item is not answerable without notes, return to the section that introduced it: 1 to §L.1, 2 to §L.2, 3 to §L.2.1, 4 to WE-L.2.a, 5 to §L.3, 6 to §L.3.1, 7 to §L.3.1 and E 1.2.4.8, 8 to §L.4, 9 to §L.4.1, 10 to §L.5, 11 to §L.4.1, 12 to WE-L.4.d.

End of Lesson 1.2.4. The learner has now completed the PPO foundation: the trust-region motivation, the clipped objective, the GAE’s advantage estimation, and the loop’s implementation. The next lesson, 1.2.5 — SAC: Maximum Entropy, the Twin Critics, and the Off-Policy Alternative, introduces the off-policy RL with the soft actor-critic’s maximum-entropy objective, the replay buffer, and the twin critics’ overestimation’s mitigation — the algorithm behind the modern robot learning’s best sample efficiency and the VLA fine-tuning’s standard.