X0.1.2

Lesson 0.1.2 — Autograd and the Computational Graph: What .backward() Actually Does, and Why .detach() and torch.no_grad() Are Not Interchangeable

Week 1 · Prerequisites: 0.1.1.

Prior results used. This lesson uses 0.1.1’s five-attribute tensor model — specifically the requires_grad attribute and the .item() operation that drops a tensor’s graph ancestry — and restates each at the point of use.

Hands forward. This lesson hands forward one capability: the learner can trace any forward pass, identify which tensors are leaves and which are not, call .backward(), read the accumulated gradients on the leaves, and state exactly which tensors must have their gradient flow stopped — the capability that Lesson 0.1.3 (the training pipeline) and every subsequent learning lesson assumes.

Running case. The same 6-DOF MuJoCo arm across 8 parallel environments from Lesson 0.1.1; every example in this lesson is an instance of computing gradients of a scalar loss with respect to the parameters of a small policy network that maps (batch, 6) observations to (batch, 6) actions.

§L.0 — Lesson map, objectives, and misconceptions

§L.0.1 — Lesson map

§L.0.2 — Objectives

By the end of this lesson you will be able to do eight things:

1. State the difference between a leaf tensor and a non-leaf tensor, and for any tensor in a forward pass, decide which it is from the code that created it.
2. Trace a forward pass through a small network, naming the grad_fn of each intermediate tensor, and predict which leaves will receive gradients after .backward().
3. Compute the gradient of a small loss by hand — chain rule applied step by step, with arithmetic shown — and verify that the value matches the .grad that PyTorch produces.
4. Diagnose a gradient-accumulation bug (missing zero_grad) from the symptom of gradients that are an integer multiple of the expected value, and state the mechanism that produced the multiple.
5. Distinguish .detach() from torch.no_grad() by their effect — one is surgical and one is contextual — and select the correct one for a given gradient-stopping requirement.
6. Apply the target-network pattern to a small Q-learning update, and state which tensor must be detached and why the update is wrong without it.
7. Produce a written “gradient audit” of a training loop: the sequence of tensors that carry gradients, the sequence that do not, and the tensors whose gradient flow is stopped.
8. Fix three silent gradient bugs in a training loop: missing zero_grad, missing target detach, and missing no_grad in inference, using the minimum change for each.

§L.0.3 — Misconception inventory

Each misconception is displaced in the section named, and each is tagged in the concept check that follows it.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

The prompt for this curriculum specifies that retrieval warm-ups draw from two to five lessons back rather than the immediately preceding lesson, because the immediately preceding lesson is still in working memory and does not constitute a spaced retrieval. This is the second lesson of the curriculum, so there is no two-back lesson to draw from; the warm-up therefore draws from Lesson 0.1.1 (the only prior lesson) and from the learner’s Python/NumPy background, and every lesson from 0.1.3 onward will draw from the full 2–5-back window.

Warm-up 1 (from 0.1.1, attribute model). A tensor has five attributes: shape, dtype, device, requires_grad, and strides. Which of the five does .item() change? Answer: .item() does not change any attribute — it produces a Python scalar that drops the reference to the tensor and its graph, so the tensor remains as it was, and the Python scalar carries none of the five attributes. How this is used: §L.4 uses this fact to explain why .item() is the correct operation for a logging buffer and why the tensor’s graph retention, not its device, is the bug it prevents.

Warm-up 2 (from 0.1.1, view/copy). Is x.clone() a view or a copy? Is x.reshape(s) a view or a copy? Answer: clone() returns a copy with its own buffer; reshape returns a view when x is contiguous and a copy when it is not. How this is used: §L.3’s .detach() shares storage — like a view — and the learner who assumed .detach() copies will hit exactly the aliasing bug that clone() avoids, which is why the clone().detach() pattern is introduced in §L.3.2.

Warm-up 3 (from Python background, operator semantics). In Python, a += b and a = a + b differ for lists (in-place vs. rebinding) but not for ints (both rebind). What is the general question you ask to decide which one a language construct performs? Answer: You ask whether the construct mutates the object’s state or rebinds the name to a new object. How this is used: §L.2’s accumulation is the gradient analogue: .backward() mutates .grad (adds to it), while a second = assignment would rebind, and the distinction is what makes optimizer.zero_grad() necessary rather than optional.

Warm-up 4 (from 0.1.1, data_ptr). Two tensors are views of the same buffer. What single call tells you they share memory? Answer: Comparing a.data_ptr() == b.data_ptr() — equal pointers indicate shared storage. How this is used: §L.3’s .detach() shares storage, and the learner can verify this with the same data_ptr() check they learned in 0.1.1, which is the point of the lab’s Step 8.

Part A — The Dynamic Computational Graph

§L.1 — Objects: leaves, non-leaves, and the graph

A computational graph in PyTorch is a directed acyclic graph built during the forward pass. Its nodes are tensors, and its edges are the operations that produced each tensor from its inputs. The graph is built dynamically: it is not declared before the forward pass, but is produced by the operations as they execute, which means a different iteration can produce a different graph.

The two kinds of node are:

• A leaf tensor is a tensor that was created directly by the user, not as the result of an operation on another gradient-tracking tensor. Model parameters (nn.Parameter, which wraps a tensor with requires_grad=True) are leaves. Inputs and targets are leaves with requires_grad=False. A tensor created by torch.randn(…, requires_grad=True) is a leaf. .to(device) does not preserve leaf status in general: a real transfer — a device change, or a dtype change — is itself an operation, so the result is a non-leaf with a ToCopyBackward0 node (p.to(torch.float64) has is_leaf = False). The one case that looks like an exception is to('cpu') on a tensor already on CPU, which returns the same object without performing a copy; there the leaf status is unchanged because no operation happened.
• A non-leaf tensor is the result of an operation on a tensor that requires grad. It carries a grad_fn attribute referencing the operation that created it, and it holds a reference to its inputs and to the saved tensors needed to compute the backward pass.

The formal distinction is t.is_leaf: it is True if and only if t was not created by an operation on a gradient-tracking tensor, which is the operational definition of “a tensor the user created directly.” This distinction matters because only leaves accumulate their gradients into .grad when .backward() is called. A non-leaf’s gradient is computed transiently during the backward pass and discarded, unless the user has called .retain_grad() on it.

The second distinction is t.grad_fn: it is None for leaves and for non-leaves that were produced by operations under torch.no_grad() or from operands that did not require grad; it is a torch.autograd.Function object otherwise. The grad_fn is the graph edge, and its type names the operation that created the tensor.

§L.1.1 — Result: the graph is built by the forward pass, and every operation on a requires_grad=True tensor adds a node

Claim. For every operation f in the differentiable subset of the tensor algebra, if any operand x of f satisfies x.requires_grad == True and the operation is not executed under torch.no_grad(), then the output y = f(x, ...) satisfies y.requires_grad == True and y.grad_fn != None, and the graph gains an edge from y to each operand.

Proof sketch. By induction on the length of the forward pass. The base case is a leaf with requires_grad=True, which has no grad_fn and needs none: its gradient is accumulated directly into .grad when the backward pass reaches it. For the inductive step, the autograd dispatch for every differentiable operation checks the requires_grad of its operands, and if any is True, wraps the output in a grad_fn node that stores (a) a reference to the function’s backward, and (b) the saved tensors and output metadata that the backward will need. The composition of these wrapped operations is the graph, and the graph is a DAG because the forward pass is a DAG of operations (no operation is its own ancestor).

Corollary. The graph is freed when .backward() completes, because the saved tensors’ references are dropped. A second .backward() on the same loss will raise RuntimeError: Trying to backward through the graph a second time unless the caller passed retain_graph=True. This corollary is the subject of §L.2.

Corollary. The graph’s memory cost is the sum of the saved tensors’ sizes, which is approximately the size of the forward pass’s intermediate activations. For a VLA forward pass on a long instruction with a high-resolution image, this is large; the memory is why torch.no_grad() in inference is not a micro-optimization, and why gradient checkpointing exists.

§L.1.2 — Worked-example ladder

WE-L.1.a — Fully worked: a linear layer’s graph, by hand and in notation

Input. A single observation of shape (1, 3) — three joint angles — and a single linear layer of shape (3, 2) with weights and bias requiring gradients. The forward pass is y_pred = x @ W + b, and the loss is loss = mse(y_pred, target) with a scalar target. We use the values from the canonical PyTorch leaf-tensor example:

• x = [1, 1, 1], shape (1, 3), requires_grad=False
• W = [[1, 1], [1, 1], [1, 1]], shape (3, 2), requires_grad=True (a leaf)
• b = [1, 1], shape (1, 2), requires_grad=True (a leaf)
• target = [1, 1], shape (1, 2), requires_grad=False (a leaf)

Trace (forward, with arithmetic).

1. z = x @ W — z[0, 0] = 1·1 + 1·1 + 1·1 = 3; z[0, 1] = 3. Shape (1, 2). grad_fn = <MmBackward0>, requires_grad=True.
2. z_biased = z + b — z_biased[0, 0] = 3 + 1 = 4; z_biased[0, 1] = 4. Shape (1, 2). grad_fn = <AddBackward0>, requires_grad=True.
3. y_pred = relu(z_biased) — relu(4) = 4; y_pred = [4, 4]. Shape (1, 2). grad_fn = <ReluBackward0>, requires_grad=True.
4. diff = y_pred - target — [4-1, 4-1] = [3, 3]. Shape (1, 2). grad_fn = <SubBackward0>, requires_grad=True.
5. sq = diff * diff — [9, 9]. Shape (1, 2). grad_fn = <MulBackward0>, requires_grad=True.
6. loss = sq.mean() — (9 + 9) / 2 = 9. Shape (). grad_fn = <MeanBackward0>, requires_grad=True.

Trace (backward, by hand, with arithmetic).

7. ∂loss/∂loss = 1.
8. MeanBackward: ∂loss/∂sq = [1/2, 1/2] (the mean’s derivative divides each element by the number of elements).
9. MulBackward: ∂loss/∂diff = 2·diff·(∂loss/∂sq) = [2·3·(1/2), 2·3·(1/2)] = [3, 3].
10. SubBackward: ∂loss/∂y_pred = [3, 3], and ∂loss/∂target = [-3, -3] but target does not require grad, so this gradient is not computed.
11. ReluBackward: ∂loss/∂z_biased = ∂loss/∂y_pred · 1[z_biased > 0] = [3·1, 3·1] = [3, 3].
12. AddBackward: ∂loss/∂z = [3, 3] and ∂loss/∂b = [3, 3].
13. MmBackward: ∂loss/∂W = x^T · ∂loss/∂z = [1, 1, 1]^T · [3, 3] = [[3, 3], [3, 3], [3, 3]]. ∂loss/∂x = ∂loss/∂z · W^T = [6, 6, 6] but x does not require grad, so this gradient is not computed.

Output and interpretation. After .backward(), W.grad = [[3, 3], [3, 3], [3, 3]], b.grad = [3, 3], and no other tensor has a .grad. The intermediate tensors z, z_biased, y_pred, diff, sq, and loss have grad_fn set but grad is None — their gradients were computed transiently and discarded. The learner who expected y_pred.grad to be [3, 3] has assumed the M1 model (every tensor gets a gradient); the correct model is that only leaves accumulate.

Self-explanation 1. Before reading on: in step 13, why is ∂loss/∂W a 3×2 matrix rather than a 2×3 matrix? What does the shape of a gradient tell you?

Answer. The gradient of a scalar with respect to a tensor has the same shape as the tensor — a fundamental convention of matrix calculus in the numerator layout that PyTorch uses. W has shape (3, 2), so W.grad has shape (3, 2). The learner who wrote [[3, 3, 3], [3, 3, 3]] computed the transpose, which is the shape of ∂loss/∂W^T, not ∂loss/∂W; the transpose convention is a common source of shape errors in manual gradient derivations.

Self-explanation 2. Before reading on: in step 9, why is the derivative of the mean 1/2 rather than 1? What would the gradient be if the loss were sum instead of mean?

Answer. The mean of two elements is (a + b) / 2, so ∂/∂a = 1/2 and ∂/∂b = 1/2; the constant is 1/N where N is the number of elements. If the loss were sum, the derivative would be 1 for each element, and every subsequent gradient in the chain would be doubled — ∂loss/∂diff = [6, 6] rather than [3, 3], and W.grad = [[6, 6], [6, 6], [6, 6]]. This is the observation WE-L.2.c uses as its “one parameter changed” exercise.

WE-L.1.b — Completion problem

Input. Same x = [2, 0, 1] (shape (1, 3)), W = [[1, 0], [0, 1], [1, 1]] (shape (3, 2)), b = [0, 0], target = [1, 1]. Same loss: mean of squared error.

Trace (forward and backward, final 25% as prompts).

1. z = x @ W — z[0, 0] = 2·1 + 0·0 + 1·1 = 3; z[0, 1] = 2·0 + 0·1 + 1·1 = 1. z = [3, 1].
2. z_biased = z + b = [3, 1].
3. y_pred = relu(z_biased) = [3, 1].
4. diff = y_pred - target = [2, 0].
5. sq = diff * diff = [4, 0].
6. loss = sq.mean() = (4 + 0) / 2 = 2.
7. ∂loss/∂sq = [1/2, 1/2].
8. ∂loss/∂diff = [2·2·(1/2), 2·0·(1/2)] = [2, 0].
9. ∂loss/∂y_pred = [2, 0] and ∂loss/∂target not computed.
10. ∂loss/∂z_biased = [2·1, 0·1] = [2, 0] (both z_biased values are positive, so the ReLU mask is [1, 1]).
11. ∂loss/∂z = [2, 0]; ∂loss/∂b = [2, 0].
12. ∂loss/∂W = ⟨your turn: compute x^T · ∂loss/∂z and give the shape⟩.
13. ∂loss/∂x = ⟨your turn: compute ∂loss/∂z · W^T, and state whether .grad is populated on x⟩.

Answers.

12. x^T = [2, 0, 1]^T (shape (3, 1)); ∂loss/∂z = [2, 0] (shape (1, 2)); the product is [[2·2, 2·0], [0·2, 0·0], [1·2, 1·0]] = [[4, 0], [0, 0], [2, 0]] — shape (3, 2), matching W.
13. ∂loss/∂z · W^T = [2, 0] · [[1, 0, 1], [0, 1, 1]] = [2·1 + 0·0, 2·0 + 0·1, 2·1 + 0·1] = [2, 0, 2]. But x has requires_grad=False, so x.grad remains None. The gradient is computed during the backward pass (the MmBackward computes it) but is not stored because x is not a leaf that requires grad.

The lesson. The gradient with respect to a non-required input is computed and discarded; the framework computes it because the generic backward node does not know in advance which operands require grad, and the accumulation step filters. This is why a learner who wants to inspect the gradient with respect to an input must either set requires_grad=True on it or call .retain_grad() on the appropriate intermediate.

WE-L.1.c — Faded: the running case with one parameter changed

Setup. Same network as WE-L.1.a, but the loss is sum rather than mean. Everything else is identical.

Question. Compute W.grad and b.grad by hand, showing each intermediate derivative. Then state the ratio between this gradient and the WE-L.1.a gradient, and explain where the factor comes from.

You must produce the intermediate steps. The setup and the answer are given; the intermediate computation is the learner’s.

Answer. With loss = sum(sq), step 7 (∂loss/∂sq) is [1, 1] instead of [1/2, 1/2]. Every subsequent gradient is therefore doubled: ∂loss/∂diff = [6, 6], ∂loss/∂z = [6, 6], ∂loss/∂b = [6, 6], ∂loss/∂W = [[6, 6], [6, 6], [6, 6]]. The ratio is 2, and it comes from the MeanBackward node: the mean divides by N = 2 in the forward pass, and its backward multiplies by 1/N; the sum does not divide, so its backward multiplies by 1. The learner who changed mean to sum and saw the loss increase by a factor of 2 has correctly traced the loss; the learner who also saw the gradients increase by a factor of 2 has correctly traced the gradient.

The lesson. The gradient’s scale is a product of every backward node’s scale factor, and the loss’s normalization is one of those factors. A change from mean to sum is a 2× change in the learning rate for every parameter, and the learner who makes this change without adjusting the learning rate will diverge the training; the learner who makes it and then adjusts the learning rate by 1/2 has made the training equivalent.

WE-L.1.d — Contrastive near-miss: retain_graph=True to silence the double-backward error

Correct solution. The learner calls .backward() once, reads W.grad and b.grad, and steps the optimizer. The graph is freed; the gradients are correct.

Near-miss. The learner calls .backward() once, then calls .backward() again to compute a second loss. The second call raises RuntimeError: Trying to backward through the graph a second time. The learner adds retain_graph=True to the second call to silence the error, and the second call succeeds. The gradients in .grad are now the sum of the two backward passes, not the second pass alone.

Why the near-miss produces a plausible but incorrect number. The second .backward() adds its gradient to whatever was in .grad from the first call, and the learner who intended to compute a fresh gradient for the second loss has instead computed grad(loss1) + grad(loss2). The optimizer.step() that follows applies the sum, not the second loss’s gradient. The training loss decreases — because the direction is still a descent direction for at least one of the two losses — but the parameters are being updated by a gradient that corresponds to neither of the intended objectives. The bug is silent in the sense that no exception is raised, but the resulting gradient is wrong in a way that is extremely difficult to diagnose from the loss curve.

The discrepancy, stated. Without retain_graph=True, the second call raises; the learner sees the error, which is the framework’s signal that the graph is gone. With retain_graph=True, the second call succeeds silently, and the success is the bug. The learner who reasons “the error says the graph is gone, so I will keep it” has confused making the error go away with making the computation correct — and the retain_graph flag was designed for the legitimate use case of a second backward on a graph that is genuinely needed twice, not as a silencer for a training loop that should be calling zero_grad between iterations.

Self-explanation 3. Before reading on: in the near-miss, what is the minimal change that makes the second backward pass compute the second loss’s gradient alone, without the retain_graph flag?

Answer. optimizer.zero_grad() (or manual W.grad = None; b.grad = None) between the two backward calls. The accumulation is in .grad, not in the graph; zeroing the leaf gradients before the second backward makes the second pass’s contribution the only one in .grad. The retain_graph flag is orthogonal — it keeps the graph alive, which is a different resource from the gradient buffers — and the learner who reaches for retain_graph when the actual fix is zero_grad has misdiagnosed which resource they needed to reset.

§L.1.3 — Comparative table: node kinds and their attributes

Design principle (P2.1). Know which tensor you are looking at before you reach for .grad. The .grad attribute is populated only on leaves that require grad; a learner who inspects y_pred.grad and finds None has not found a bug — they have found the framework working as designed. If the intermediate gradient is genuinely needed (for attribution, for a KL penalty, for a per-layer analysis), call .retain_grad() on that tensor before the backward pass; the call is O(1) and the gradient is the same value that would otherwise be discarded.

§L.1.4 — Concept check

Concept check (displaces M1). A learner calls .backward() and then inspects y_pred.grad for the intermediate activation y_pred, finding it None. They conclude that the backward pass has failed. What has actually happened, and what would they have needed to do to inspect the intermediate gradient?

Answer. The backward pass succeeded; y_pred is a non-leaf, so its gradient was computed transiently and discarded rather than stored in .grad. The learner would need to call y_pred.retain_grad() before .backward(), which allocates a gradient buffer for the tensor and causes the backward pass to populate it. The learner’s conclusion — that the backward pass failed — is wrong; the backward pass did exactly what it is designed to do, and the learner’s model of which tensors accumulate gradients is what needs updating.

§L.1.5 — Consolidation checkpoint 1

(a) You should now be able to: (i) decide, for any tensor in a forward pass, whether it is a leaf, and state whether .grad will be populated after .backward(); (ii) trace the chain of grad_fn objects from the loss back to the leaves, naming each operation; (iii) compute a small network’s leaf gradients by hand and match them against the framework’s output.

(b) Transfer probe. A learner adds a nn.Linear(6, 6) layer between two existing layers of a policy network and reports that the new layer’s parameters are not learning — their .grad is None after .backward(). They have confirmed that the forward pass runs without error and that the loss decreases. What is the most likely cause? Answer sketch: the new layer’s parameters are not nn.Parameter (they were assigned as a raw tensor), so they are not in model.parameters(), and either the optimizer was not constructed with them or the module’s registration is missing. The .grad is None because the parameter is a leaf with requires_grad=False (a raw tensor) — the layer is not being trained. The test is any(p is new_layer.weight for p in model.parameters()); if it is False, the registration is the bug.

(c) Remediation rule. If you cannot do (b), return to §L.1.2 and redo WE-L.1.b, paying attention to which operand’s .grad was or was not populated and why; then re-attempt (b) by asking which leaf’s requires_grad is True.

Part B — .backward() and Gradient Accumulation

§L.2 — Objects: the backward pass, accumulation, and graph freeing

.backward() is the method that triggers the backward pass. It takes an optional gradient argument (for non-scalar outputs; we will see this in the lab’s break-it step) and two optional flags: retain_graph and create_graph. Its behavior:

1. It traverses the graph in reverse topological order, from the tensor on which it was called to every leaf that requires grad and is reachable.
2. For each node, it calls that node’s backward function, which computes the gradient of the output with respect to the inputs and adds it to the .grad of the leaf tensors (and of any non-leaf with retain_grad() set).
3. After the traversal, by default, it frees the graph — the saved tensors’ references are dropped, and the intermediate nodes are eligible for garbage collection.

The three behaviors interact to produce four operational rules the learner must internalize:

Rule 1: Only leaves accumulate. Already established in Part A.

Rule 2: Gradients accumulate; they do not overwrite. Every .backward() call adds to whatever is in .grad. The learner who calls .backward() twice without clearing gets the sum. The PyTorch docs state this explicitly; it is by design, because it is what makes gradient accumulation (simulating a larger batch by summing over mini-batches) and multi-task losses (summing the gradients of several losses) possible.

Rule 3: The graph is freed after the backward pass. A second .backward() on the same loss raises unless retain_graph=True. The flag keeps the saved tensors alive, but it does not change the accumulation behavior — the second backward still adds, so a learner who uses retain_graph to fix an error and does not also clear .grad has introduced the accumulation bug described in §L.2.2 (WE-L.2.a).

Rule 4: create_graph=True builds a graph of the backward pass. This enables higher-order gradients, which are required for meta-learning, for differentiating through a learned dynamics model, and for some physics-informed losses. It is not needed in the vast majority of the curriculum; the learner should know that the flag exists and what it is for.

§L.2.1 — Result: .backward() is a sum, not an assignment, and the sum is the mechanism behind both gradient accumulation and the zero-grad requirement

Claim. Let L₁ and L₂ be two scalar losses computed from the same parameters, with .backward() called on each in sequence without clearing .grad in between. Then after both calls, p.grad = ∂L₁/∂p + ∂L₂/∂p for every leaf p. The identity holds regardless of whether L₁ and L₂ share a graph or are computed from separate forward passes.

Proof sketch. Each backward pass traverses the graph reachable from its root and adds the corresponding gradient to .grad. The addition is in-place and commutative; the order of the two backward passes does not matter. The only requirement is that both graphs are alive at the time of their respective backward passes, which is guaranteed if the second forward pass was performed after the first backward pass (the first graph has been freed but the second is fresh), and if the losses are not backward-ed twice. The result is the sum, and the sum is the mechanism behind gradient accumulation: if L₁ is the loss on mini-batch 1 and L₂ is the loss on mini-batch 2, the sum ∂L₁/∂p + ∂L₂/∂p is proportional to the gradient of the average loss on the combined batch, and the learner who wants a larger effective batch simply accumulates gradients over several mini-batches and steps once.

Corollary. The learner who wants a single loss’s gradient must clear .grad before the backward pass, because the framework will otherwise add to whatever is there. This is what optimizer.zero_grad() does: it iterates the optimizer’s parameter groups and sets each p.grad to None (or to zero, depending on the version and the set_to_none flag; None is the default and the more memory-efficient).

§L.2.2 — Worked-example ladder

WE-L.2.a — Fully worked: accumulation with real numbers

Input. The same linear layer from WE-L.1.a: W = [[1, 1], [1, 1], [1, 1]] with requires_grad=True, b = [1, 1] with requires_grad=True. Two losses: L₁ = mse(W·[1,1,1]^T + b, [1, 1]) = 9, and L₂ = mse(W·[2,0,1]^T + b, [2, 4]) = 2 — note the different target: L₂’s predictions are [4, 4], so with target [1, 1] it would equal 9 as well, and the accumulation would read 3+9 rather than 3+4.

Trace.

1. L₁.backward() — computes W.grad = [[3, 3], [3, 3], [3, 3]] and b.grad = [3, 3]; frees the first graph.
2. L₂.backward() — computes W.grad = [[3+4, 3+0], [3+0, 3+0], [3+2, 3+0]] = [[7, 3], [3, 3], [5, 3]] and b.grad = [3+2, 3+0] = [5, 3]; frees the second graph.

Output and interpretation. The final W.grad is the sum of the two individual gradients, not the second one alone. The learner who intended L₂’s gradient (and wrote the training loop to compute it) has instead the sum, and the optimizer step that follows applies ∂(L₁+L₂)/∂W. In a real training loop this is the gradient accumulation pattern — if L₁ and L₂ are losses on two halves of a batch, the sum is the gradient of the batch’s total loss — but in a loop that was written without the accumulation pattern in mind, it is a bug.

Self-explanation 4. Before reading on: in step 2, why does W.grad[0, 1] receive 3 + 0 = 3 and not 3 + 0 = 3 recomputed from scratch?

Answer. W.grad[0, 1] was 3 after L₁.backward(). L₂’s gradient with respect to W[0, 1] is 0 (because x₁ = 0 in L₂’s input, so W[0, 1] does not affect L₂’s output). The accumulation adds 0 to 3, leaving 3. The learner who expected W.grad[0, 1] to be 0 (the second loss’s contribution alone) has forgotten that .grad was not cleared; the value is the sum, and the sum’s second term happens to be zero.

WE-L.2.b — Completion problem

Input. The same W and b. L₁ = mse(W·[1,1,1]^T + b, [1,1]) = 9 and L₂ = mse(W·[2,0,1]^T + b, [2,4]) = 2. The learner writes:

L₁.backward()
optimizer.zero_grad()
L₂.backward()

Trace (final 25% as prompts).

1. L₁.backward() — W.grad = [[3,3],[3,3],[3,3]], b.grad = [3,3].
2. optimizer.zero_grad() — ⟨your turn: what are W.grad and b.grad now?⟩
3. L₂.backward() — ⟨your turn: what are W.grad and b.grad now?⟩
4. Compare with WE-L.2.a’s final gradient. ⟨your turn: what is the difference, and what accounts for it?⟩

Answers.

1. After zero_grad(), W.grad and b.grad are None (the default set_to_none=True), or zero tensors if the optimizer was configured with set_to_none=False. Either way, the buffers are cleared.
2. After L₂.backward(), W.grad = [[4, 0], [0, 0], [2, 0]] and b.grad = [2, 0] — exactly L₂’s gradient, unaccumulated.
3. The difference from WE-L.2.a is that the zero_grad between the two backward passes cleared the accumulated L₁ gradient, so the final gradient is L₂’s alone rather than the sum.

The lesson. The placement of zero_grad is the difference between “the gradient of the last loss” and “the gradient of the sum of the losses,” and the framework will accept both without complaint. The learner who wants one or the other must state which, and the placement is the statement.

WE-L.2.c — Faded: the running case with one parameter changed

Setup. The running case’s 8-environment policy is trained on a batch of 256 transitions. The learner wants an effective batch of 512 and has a machine that fits only 256. They use gradient accumulation: two mini-batches of 128, each with its own forward pass and .backward(), and one optimizer.step() after both.

Question. Write the loop in pseudocode, state where zero_grad and step are placed, and state the effective learning rate relative to a single 512-batch training run. Hint: what does the sum of two losses’ gradients correspond to, and how does that compare to the gradient of the mean loss over the combined batch?

You must produce the intermediate steps. The setup and the answer are given.

Answer.

optimizer.zero_grad()                         # once per effective batch
for micro_batch in [batch_1, batch_2]:        # two micro-batches of 128
    loss = compute_loss(model, micro_batch)   # loss is the mean over 128
    loss.backward()                           # accumulates into .grad
optimizer.step()                              # once per effective batch

The gradients accumulate over the two micro-batches, so after the loop p.grad = ∂L₁/∂p + ∂L₂/∂p where L₁ and L₂ are the per-micro-batch means. The mean over the combined 256 is (L₁ + L₂) / 2, so the accumulated gradient is twice the gradient of the combined mean. To match the single 512-batch run (whose loss is the mean over 512), the learner must either divide the loss by 2 before backward (i.e., use loss = compute_loss(...) / num_micro_batches) or halve the learning rate. The learner who accumulates without dividing has effectively doubled the learning rate, which often looks like faster initial learning and then a divergence or an instability that is hard to attribute to the accumulation.

The lesson. Gradient accumulation sums gradients, and a sum of means is not a mean of means — the factor of num_micro_batches is a learning-rate change in disguise. The learner who accumulates without adjusting has changed the optimization problem without changing the code that looks like it changes the optimization problem.

WE-L.2.d — Contrastive near-miss: retain_graph to reuse a forward pass

Correct solution. The learner wants two losses from a single forward pass — a policy loss and a value loss, sharing a trunk. They compute both losses from the same forward pass and call .backward() on the sum: (policy_loss + value_loss).backward(). The graph is traversed once, the gradients are the correct sum, and no flags are needed.

Near-miss. The learner calls policy_loss.backward(retain_graph=True) and then value_loss.backward(). Both calls succeed; the graph is kept alive for the second call. The gradients in .grad are ∂(policy_loss + value_loss)/∂p, which happens to equal the correct solution’s gradient — if zero_grad was called before the first backward and not between the two. But the learner who also calls optimizer.zero_grad() between the two backward calls (out of habit, or because the training loop’s structure suggests it) clears the policy loss’s gradient before the value loss’s backward, and the resulting gradient is ∂value_loss/∂p alone. The policy loss’s gradient is silently lost.

Why the near-miss produces a plausible but incorrect number. The retain_graph=True call succeeds, the zero_grad call succeeds, the second backward call succeeds, and the optimizer step succeeds. The training loss decreases — because the value loss is a legitimate objective and the parameters are still being updated — but the policy is not being trained at all, or is being trained only by whatever gradient survives the zeroing. The learner who sees a decreasing value loss and a stagnant policy has a signature that is consistent with many causes (a bad reward, a saturated policy, a data issue), and the actual cause is a zero_grad placed one line too early.

The discrepancy, stated. The correct solution sums the losses and calls .backward() once; the near-miss calls .backward() twice with a retain_graph flag and a zero_grad between them. The two are numerically identical if and only if the zero_grad is before the first backward and not between; the near-miss’s failure mode is that the zero_grad placement is a line that “looks right” by the training-loop template and is wrong for the retained-graph pattern. The learner who has not internalized that zero_grad clears the accumulated gradient (not the graph) cannot see the bug, because both the graph and the gradients look alive.

Self-explanation 5. Before reading on: in the near-miss, the learner’s zero_grad between the two backward calls clears p.grad. What would the gradient be if the learner had used retain_graph=True and moved the zero_grad to before the first backward?

Answer. The gradient would be ∂(policy_loss + value_loss)/∂p — the correct sum — because the first backward adds the policy loss’s gradient to the (cleared) .grad, and the second backward adds the value loss’s gradient to it. The retain_graph flag is doing its job (keeping the graph alive for the second backward), and the zero_grad placement is correct. The two changes together — move zero_grad to before the first backward, keep retain_graph=True — reproduce the correct solution’s gradient, but at the cost of a less clear loop; the cleaner solution is the single .backward() on the sum.

§L.2.3 — Comparative table: ways to handle two losses

Design principle (P2.2). Sum the losses, then backward once. If two losses share a forward pass, their sum’s gradient is the sum of their gradients, and one backward pass computes it. Reach for retain_graph only when the loss computation genuinely cannot be combined, and when you do, place zero_grad once, before the first backward, never between.

§L.2.4 — Concept check

Concept check (displaces M3). A learner says: “Calling .backward() twice just recomputes the same gradients.” Give the two facts that make this false, and name the flag that is sometimes used to silence the first fact without addressing the second.

Answer. First, the graph is freed after the first backward, so a second call raises unless retain_graph=True was passed. Second, the gradients accumulate rather than overwrite, so even with the graph retained, the second call’s gradients are added to the first call’s. The flag is retain_graph=True, and it addresses only the first fact; the learner who uses it without also clearing .grad has introduced the accumulation bug described in §L.2.2 (WE-L.2.a).

§L.2.5 — Consolidation checkpoint 2

(a) You should now be able to: (i) predict the value of .grad after two backward calls with and without zero_grad in between; (ii) choose between the sum-and-single-backward pattern, the sequential pattern with retain_graph, and the accumulation pattern, and state the trade-off; (iii) compute the effective learning rate of a gradient-accumulated loop relative to a single-batch loop.

(b) Transfer probe. A learner trains a policy with PPO and reports that the policy loss oscillates violently between iterations, with some iterations showing a 10× increase in loss. Their loop does loss.backward(); optimizer.step() for each of 4 epochs over the same rollout, with zero_grad only once per iteration (before the 4 epochs). What is the most likely cause? Answer sketch: the PPO inner loop performs 4 epochs over the same data, and with zero_grad only once before the epochs, the gradients from the 4 epochs accumulate, so the effective update after 4 epochs is 4× the intended update. The fix is zero_grad inside the inner loop (once per epoch), which is the standard PPO pattern. The learner who zeroes once per iteration has written a 4× learning rate increase without changing the learning rate.

(c) Remediation rule. If you cannot do (b), return to §L.2.2 and redo WE-L.2.b, paying attention to which gradients are in .grad after each operation; then re-attempt (b) by writing the loop’s gradient state at each line.

Part C — .detach(): The Surgical Cut

§L.3 — Objects: detachment, shared storage, and clone().detach()

.detach() returns a new tensor that shares the same data buffer as the original but is disconnected from the computational graph. It has requires_grad=False and grad_fn=None. The operation is O(1) and does not copy memory.

The two properties — shared storage and disconnected graph — are the source of everything that follows.

Shared storage means that .detach() is a view in the sense of Lesson 0.1.1: an in-place modification to the detached tensor modifies the original, and vice versa. The learner who wants an independent tensor must call .clone() before or after the detach — x.detach().clone() and x.clone().detach() both produce an independent, graph-free tensor, and they are equivalent in their effect (the difference is whether the copy happens before or after the detach, which is irrelevant because the detach is free).

Disconnected graph means that operations on the detached tensor are not tracked, and no gradient flows back through it to the original. The original tensor’s graph is unaffected; the detach is a property of the new tensor, not a modification of the source.

The distinction between .detach() and .clone() is important and easy to miss:

• .detach() shares the buffer, disconnects the graph.
• .clone() copies the buffer, preserves the graph connection (if the source requires grad, the clone requires grad and has a CloneBackward grad_fn).
• .detach().clone() or .clone().detach() does both: independent buffer, no graph.

§L.3.1 — Result: .detach() shares storage and removes the graph; neither property is visible in the tensor’s repr

Claim. For a tensor x with requires_grad=True, y = x.detach() satisfies y.data_ptr() == x.data_ptr() and y.grad_fn is None. The first property is a memory-sharing relation, the second is a graph relation, and neither is shown in the printed repr of y (which shows shape, dtype, and values).

Proof sketch. The detach operation in PyTorch constructs a new Tensor object that points to the same Storage as x, with requires_grad=False and no grad_fn. The Storage is the reference-counted buffer; both tensors hold a reference to it, so the buffer is not freed while either is alive. The repr of y prints the values and the dtype, both of which are identical to x’s; the repr does not print the storage pointer or the grad_fn, so a learner comparing the two reprs sees no difference.

Consequence. The learner cannot determine whether a tensor is a detached view or an independent copy from its repr. They must either call .data_ptr() and compare, or know the operation that produced it. The lab’s Step 8 makes this comparison, and the exercise E 0.1.2.3 asks the learner to reason about it from the operation alone.

§L.3.2 — Worked-example ladder

WE-L.3.a — Fully worked: detach, modify, observe the source

Input. The running case’s policy output action of shape (8, 6) (8 environments, 6 joints), with requires_grad=True because it came from a differentiable layer.

Trace.

1. action = policy(obs) — shape (8, 6), requires_grad=True, grad_fn=<AddmmBackward0>.
2. action_detached = action.detach() — shape (8, 6), requires_grad=False, grad_fn=None, action_detached.data_ptr() == action.data_ptr() is True.
3. action_detached[0, 0] = 99.0 — modifies the shared buffer, so action[0, 0] is also 99.0.
4. action_copy = action.detach().clone() — shape (8, 6), requires_grad=False, grad_fn=None, action_copy.data_ptr() != action.data_ptr().
5. action_copy[0, 0] = -1.0 — action[0, 0] remains 99.0.

Output and interpretation. The detached view and the original share storage; the cloned detached tensor does not. The learner who wrote only action.detach() and then modified it in place has silently modified the original, which is the aliasing bug that Lesson 0.1.1’s view/copy material introduced; the .clone().detach() pattern is the fix.

Self-explanation 6. Before reading on: in step 2, does action_detached have a grad_fn? Why or why not, and what does the answer tell you about what detach does to the graph?

Answer. No — detach produces a tensor with grad_fn=None and requires_grad=False, because it is disconnected from the graph. The answer tells the learner that detach is not a “copy of the graph but with no gradient”; it is a tensor that is not part of the graph at all, and operations on it are not tracked. The original action still has its grad_fn and is still connected; the detach is a property of the new tensor, not a modification of the source.

WE-L.3.b — Completion problem

Input. The same action. The learner wants to compute a loss that depends on action and on a fixed target action_target, where action_target is the detach of action (a common pattern for a self-supervised consistency loss). The learner writes loss = mse(action, action.detach()).

Trace (final 25% as prompts).

1. action — requires_grad=True, in the graph.
2. action.detach() — requires_grad=False, out of the graph, shares storage.
3. mse(action, action.detach()) — ⟨your turn: what is the value of this loss, and why?⟩
4. loss.backward() — ⟨your turn: what is the gradient of loss with respect to action, and why?⟩

Answers.

1. The value is 0, because the two arguments are the same tensor’s values — the detach shares storage, so the two arguments are numerically identical at every element. mse(x, x) = 0.
2. The gradient is zero for every element, because the target is a constant (detached) and the difference action - action.detach() is identically zero, so the derivative of the square of zero is zero. The loss is degenerate: it always evaluates to zero, and its gradient is always zero.

The lesson. Detaching a tensor and then comparing it to itself produces a degenerate loss. The pattern is legitimate only when the detached tensor is not the same as the tensor being trained — for example, when the target comes from a slowly-updated copy of the network (the target-network pattern of §L.5), or from a reference policy that is not being updated. The learner who uses action.detach() as a target for action has written a loss that trains nothing, and the code looks plausible.

WE-L.3.c — Faded: the running case with one parameter changed

Setup. The running case’s policy outputs action of shape (8, 6). The learner wants a loss that penalizes action for being far from a reference action action_ref that comes from a frozen copy of the policy. The frozen copy is stored in ref_policy and its parameters have requires_grad=False.

Question. Write the loss expression and state whether action_ref = ref_policy(obs) needs an explicit .detach(), given that ref_policy’s parameters do not require grad. Then state what would change if ref_policy’s parameters did require grad (for example, if the reference is a slowly-updated copy that is currently in the optimizer’s parameter list by mistake).

You must produce the intermediate steps. The setup and the answer are given.

Answer. loss = mse(action, ref_policy(obs)); if ref_policy’s parameters have requires_grad=False, then ref_policy(obs) is already a tensor with requires_grad=False (because no operand required grad), and the explicit .detach() is redundant. If ref_policy’s parameters did require grad — a plausible mistake if the reference policy was constructed from the policy’s weights without copying and freezing — then ref_policy(obs) has requires_grad=True, and the loss’s gradient would flow into both action and ref_policy’s parameters. The optimizer would update the reference, the reference would not be a fixed target, and the loss would become a moving objective that both networks chase toward each other. The explicit .detach() on the reference output would prevent this, regardless of whether ref_policy’s parameters were frozen.

The lesson. The reference tensor needs to be detached when its parameters might require grad — and in a codebase where a reference policy is constructed by copy.deepcopy(policy), the deep copy’s parameters do require grad by default, because copy.deepcopy preserves requires_grad. The detach is not redundant; it is the safety net that catches the case where the freeze was forgotten.

WE-L.3.d — Contrastive near-miss: .detach() vs. .detach().clone() in a target buffer

Correct solution. The learner maintains a target network by copying the online network’s parameters into the target’s parameters every C steps: target.weight.data.copy_(online.weight.data). This is a copy — the data buffer of target.weight is overwritten with the values of online.weight, and the two remain separate buffers.

Near-miss. The learner instead writes target.weight = online.weight.detach(). On a real nn.Module this raises — TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected) — because nn.Module.__setattr__ requires an nn.Parameter for a registered parameter name. The aliasing described below is therefore not reachable through this exact line; it is reachable through the two spellings that do run: target.weight.data = online.weight.detach() (which rebinds the .data attribute and leaves the module’s parameter bookkeeping inconsistent) or target.weight.data = online.weight.data (which shares the buffer without any error). Both produce the same defect: the two modules share one buffer. The next optimizer step on the online network updates that buffer in place, and the target network’s weight changes at the same time — not every C steps, but every step. The target is no longer a delayed copy; it is a live view of the online network.

Why the near-miss produces a plausible but incorrect number. The detach() call is the right idea — it removes the gradient connection — but the shared storage means the values still track the online network in real time. The target-network update, which the learner thought was happening every C steps, is happening every step, because the “copy” was an alias. The Q-learning update then has a target that moves with the online network at every step, which is the same instability as not having a target network at all: the objective is non-stationary, the Q-values chase themselves, and the training either diverges or plateaus at a poor policy. The learner who diagnoses this by inspecting the target’s weights between steps will see them change, which is the smoking gun, but the diagnosis requires knowing to look. The TypeError on the = spelling is, in a sense, the lucky outcome: the framework refused the aliasing that the .data spelling accepts.

The discrepancy, stated. x.data.copy_(y.data) copies values into an existing buffer; x = y.detach() rebinds the name x to a new tensor that shares y’s buffer. The two are both “make x be y’s values,” and they differ in exactly one property — whether the buffers are separate — which is the property that determines whether a future mutation of y is visible in x. The target-network pattern requires the copy, because the point of the target is that it does not track the online network between updates.

Self-explanation 7. Before reading on: in the near-miss, the learner used .detach() because they wanted to prevent gradient flow from the target to the online network. Was the .detach() necessary, and was it sufficient?

Answer. It was necessary only if the assignment was going to be used in a differentiable expression — but the assignment is to a parameter attribute, not to a loss term, and a successful parameter assignment does not build a graph through the assigned value. In fact nn.Module.__setattr__ refuses a plain Tensor outright, so the .detach() here buys nothing the framework was not already going to enforce. It was not sufficient for the target-network pattern, because the target-network requirement is not only “no gradient flow” but also “separate buffer, updated only every C steps.” The .detach() addressed the first requirement and left the second unaddressed; the .data.copy_() pattern addresses both.

§L.3.3 — Comparative table: detachment variants

Design principle (P2.3). Use .detach().clone() for targets that must be independent. Whenever a tensor is used as a target in a loss — a Q-learning target, a reference policy’s action, a frozen embedding, an EMA teacher’s output — use .detach().clone() unless the target is known to be either not modified downstream or genuinely shared. The .clone() is O(N) and the target is typically a single batch, so the cost is negligible; the bug it prevents is the aliasing bug that is invisible in the repr and deadly in the training.

§L.3.4 — Concept check

Concept check (displaces M2, part 1). A learner says: “I use .detach() when I want to stop gradients, and torch.no_grad() when I want to stop gradients faster.” Name the actual difference between the two, and state the operation that makes .detach() safe against in-place aliasing.

Answer. The difference is not speed but scope: .detach() stops gradient flow through one tensor, and the tensor still shares its storage with the source; torch.no_grad() stops graph recording for an entire block of operations, so no graph is built at all for anything executed inside it. The operation that makes .detach() safe is .clone(), because the clone’s independent buffer means an in-place modification to the detached tensor does not modify the source.

§L.3.5 — Consolidation checkpoint 3

(a) You should now be able to: (i) state the two properties of .detach() (shared storage, disconnected graph) and the operation that adds the third (independent values); (ii) distinguish x.detach() from x.clone() from x.detach().clone() by their effect on buffer sharing and graph connectivity; (iii) diagnose a target-network alias bug from the symptom that the target’s weights change every step.

(b) Transfer probe. A learner implements an EMA (exponential moving average) teacher for a self-supervised robot learning objective. They write teacher.weight = student.weight.detach() * (1 - m) + teacher.weight * m. The run crashes on the first assignment: TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected). Why does the framework reject this, and what is the correct implementation? Answer sketch: nn.Module.__setattr__ intercepts names registered as parameters and requires an nn.Parameter; the right-hand side is a plain Tensor, so the assignment is refused before any training happens. The learner who “fixes” the error by wrapping it — teacher.weight = nn.Parameter(...) — trades a crash for a quieter defect: the module now holds a different Parameter object than the optimizer was constructed with, so the teacher’s weights are absent from the optimizer’s parameter groups, and any weight decay or later update silently skips them. The correct implementation updates the existing buffer in place, under torch.no_grad(): teacher.weight.data.mul_(m).add_(student.weight.detach().data, alpha=1 - m). The bug is the alias-vs-copy confusion of the near-miss, but the framework catches this spelling loudly — which is exactly why the .data spelling, which it does not catch, is the dangerous one.

(c) Remediation rule. If you cannot do (b), return to §L.3.2 and redo WE-L.3.d, paying attention to which operation copies and which operation aliases; then re-attempt (b) by asking, for each line, whether the left-hand side’s buffer is overwritten or the name is rebound.

Part D — torch.no_grad(): The Contextual Cut

§L.4 — Objects: the context manager, its scope, and its interaction with the graph

torch.no_grad() is a context manager that disables graph recording for the duration of its block. Operations executed inside the block do not build a graph, regardless of the requires_grad of their inputs, and the outputs have requires_grad=False and grad_fn=None.

The canonical uses are:

1. Inference and evaluation. The model’s parameters have requires_grad=True, but the evaluation does not need gradients. Wrapping the evaluation forward pass in torch.no_grad() avoids building and immediately discarding a graph on every batch, which is both a memory saving (no saved activations) and a latency saving (no graph construction).
2. Target computation. In a Q-learning update, the target is a constant with respect to the online network; the target computation is a forward pass through the target network, and torch.no_grad() makes the constancy explicit and enforced.
3. Model construction and parameter initialization. When a model is being built or a parameter is being set from a computation, the computation is not a training step and does not need a graph.
4. Rollout and data collection. In an on-policy RL loop, the rollouts are collected by the current policy, and the policy’s actions during collection are not training steps; collecting them inside torch.no_grad() avoids building a graph that will never be backpropagated.

§L.4.1 — Result: torch.no_grad() disables graph construction for a block; .detach() removes a tensor from an existing graph

Claim. Inside a torch.no_grad() block, the forward pass builds no graph, the output tensors have grad_fn=None, and the memory cost of the forward pass is the size of the output tensors, not the size of the saved activations. Outside the block, a tensor produced by .detach() has grad_fn=None but the rest of the graph is unaffected, and operations on other tensors outside the block build graphs as usual.

Proof sketch. torch.no_grad() sets a thread-local flag that the autograd dispatch checks before building any node. Every operation inside the block sees the flag and skips the grad_fn construction, including the saving of intermediate activations for the backward pass. The flag is restored when the block exits. .detach(), by contrast, is an operation on a single tensor: it produces a new tensor with requires_grad=False and grad_fn=None, but it does not affect any other tensor, and operations outside the detach build graphs as usual.

Consequence. The two are not substitutes. A learner who wants “no graph for this forward pass” uses torch.no_grad(); a learner who wants “no gradient flow through this one tensor, but the rest of the expression is differentiable” uses .detach(). The first is the right tool for inference and target computation; the second is the right tool for a mixed expression, such as a loss that combines a learnable term with a fixed coefficient or a fixed target.

§L.4.2 — Worked-example ladder

WE-L.4.a — Fully worked: the same expression inside and outside no_grad

Input. The running case’s policy has parameters W with requires_grad=True, and an observation obs of shape (8, 6). The forward pass is action = policy(obs).

Trace.

1. action = policy(obs) outside no_grad — action.requires_grad=True, action.grad_fn=<AddmmBackward0>. The graph is built.
2. action = policy(obs) inside with torch.no_grad(): — action.requires_grad=False, action.grad_fn=None. No graph is built; the intermediate activations are not saved.
3. action_detached = policy(obs).detach() outside no_grad — the forward pass builds a graph, then .detach() produces a tensor disconnected from it. The graph is built and immediately discarded (unless another reference keeps it alive). This is the inefficient version of (2).
4. loss = mse(action, target) outside no_grad — if action is from (1), the loss has requires_grad=True and loss.backward() will compute gradients; if action is from (2), loss.requires_grad=False and loss.backward() will raise (element 0 of tensors does not require grad).

Output and interpretation. The comparison between (2) and (3) is the lesson: no_grad prevents the graph from being built, and detach disconnects a tensor after the graph was built. The memory cost of (3) includes the graph’s activations, which are discarded after the detach; the memory cost of (2) does not. The learner who uses .detach() in an inference loop (where the entire forward pass is gradient-free) has chosen the wrong tool: the graph was built anyway, and the memory and latency costs were paid.

Self-explanation 8. Before reading on: in step 4, why does loss.backward() raise when action is from step 2, and what does the error message tell the learner about the connection between the loss and the model’s parameters?

Answer. The loss’s requires_grad is False, because every operand was produced under no_grad and has requires_grad=False; a backward pass on a tensor that does not require grad raises element 0 of tensors does not require grad and does not have a grad_fn. The error tells the learner that the loss is not connected to the model’s parameters — the graph that would have connected them was never built — and the fix is to compute the loss outside the no_grad block. The error is the good case: the framework has caught the mistake. The bad case is when a part of the expression is inside no_grad and the loss is a valid scalar that requires grad but is missing the contribution of the inside-block terms; the loss trains a subset of the intended objective, and nothing raises.

WE-L.4.b — Completion problem

Input. The same policy. The learner wants to compute a frozen baseline v_frozen = value_net(obs).detach() and use it in a loss that also depends on a learnable v_pred = value_net(obs). The forward pass is inside no_grad, and the learner writes:

with torch.no_grad():
    v_frozen = value_net(obs)
    v_pred = value_net(obs)
loss = mse(v_pred, v_frozen)

Trace (final 25% as prompts).

1. v_frozen — shape (8, 1), requires_grad=False, grad_fn=None.
2. v_pred — shape (8, 1), requires_grad=False, grad_fn=None (same reason as v_frozen: it is inside the block).
3. loss = mse(v_pred, v_frozen) — ⟨your turn: what is loss.requires_grad, and what is the value of the loss?⟩
4. loss.backward() — ⟨your turn: what happens, and what is the fix?⟩

Answers.

1. loss.requires_grad is False — both operands are graph-free, so the loss is graph-free. The value is mse(v, v) where both are the same forward pass’s output, so the value is 0.
2. loss.backward() raises, because the loss does not require grad. The fix is to compute v_pred outside the no_grad block (v_pred = value_net(obs)) and only v_frozen inside it (with torch.no_grad(): v_frozen = value_net(obs)), and to compute the loss outside both. The corrected loss has requires_grad=True, and its gradient flows into value_net’s parameters through v_pred, while v_frozen is a constant.

The lesson. The no_grad block’s scope is what determines which tensors are in the graph, and the learner who wraps both branches of a self-supervised objective in the same block has made the loss a constant. The detach-after-the-fact version (v_frozen = value_net(obs).detach()) achieves the same result with a graph built and discarded; the no_grad version is more efficient but requires the scope to be exactly the frozen branch, not the whole computation.

WE-L.4.c — Faded: the running case with one parameter changed

Setup. The running case’s training loop has a rollout collection phase (the policy acts in the simulator) and a training phase (the policy’s parameters are updated). The rollout collection is currently outside any no_grad block, so each step’s forward pass builds a graph that is discarded when the next step’s forward pass begins.

Question. Estimate the memory and latency cost of the missing no_grad in the rollout, and state the one-line change that removes it. Then state what would break if the no_grad were placed around the entire training iteration, including the loss and the backward pass.

You must produce the intermediate steps. The setup and the answer are given.

Answer. The rollout collection runs num_envs × rollout_len forward passes per iteration (8 × 50 = 400 in the running case), and each forward pass’s graph is freed when the next begins. The peak memory is the size of a single step’s graph, not the sum, but the graph construction and destruction cost is paid 400 times per iteration, and the latency includes the grad_fn construction and the saved-activation bookkeeping on every step. The one-line change is with torch.no_grad(): around the rollout’s inner loop. Placing no_grad around the entire training iteration (including the loss and the backward) would make the loss graph-free and the backward would raise, or — if the backward were somehow reached — it would have no graph to traverse; the training would stop learning, and the failure would be the “loss is a constant” case of WE-L.4.b.

The lesson. The no_grad block’s boundaries are the boundaries of “no graph needed.” The rollout and the training step have different boundaries; the learner who uses one block for both has either paid the rollout’s graph cost (too wide a scope for the rollout) or broken the training step (too wide a scope for the training). The correct pattern is two blocks with different scopes, and the scope is a design decision the learner must state.

WE-L.4.d — Contrastive near-miss: no_grad in the inference loop, but only around the model call

Correct solution. The learner wraps the entire evaluation loop in torch.no_grad(), including the data loading, the model call, the loss computation, and the metric accumulation.

Near-miss. The learner wraps only the model call: with torch.no_grad(): pred = model(x), and computes loss = loss_fn(pred, y) outside the block. The loss computation is differentiable with respect to pred (which is graph-free) and with respect to y (which is a target, graph-free); the loss itself is graph-free, so the backward is not attempted. But the metric accumulation is outside the block, and if the metric involves an operation on pred that builds a graph (it does not, because pred is graph-free) — the near-miss is benign in this case. The near-miss becomes a bug when the data loading is outside the block and the loader’s tensors require grad (which they do not, in a standard loader) — again benign. The near-miss is genuinely benign in most cases, and the lesson is that the safe version — wrap the whole loop — costs nothing and removes the learner’s need to reason about which sub-expression is graph-free.

Why the near-miss produces a plausible but incorrect number. In the common case it does not; it produces the correct number with a slightly larger memory footprint than the safe version. The case in which it matters is when the evaluation computes a metric that does build a graph — for example, an evaluation that computes a differentiable quantity (a gradient norm, a Fisher information estimate, a Lipschitz constant) and then discards the graph. If the metric’s construction is outside the no_grad, the graph is built; if the metric’s construction is inside, the graph is not. The safe version puts everything inside, and the learner who needs the graph for a differentiable metric must deliberately take it out, which is the right default.

The discrepancy, stated. The two versions produce the same numbers in the common case and different memory profiles in the rare case. The difference is a default: the safe version’s default is “no graph unless I ask for one,” and the near-miss’s default is “graph unless I remember to suppress it.” For a learner who is new to autograd, the safe default is the one that does not require remembering, and the cost of the safe default is zero.

Self-explanation 9. Before reading on: in the near-miss, the learner’s pred is graph-free but the loss is outside the block. Is the loss graph-free? Why, and what would it mean if it were not?

Answer. The loss is graph-free because both operands (pred, y) are graph-free, so the arithmetic operations on them do not build a graph (no operand requires grad). If the loss were not graph-free — for example, if y were a tensor with requires_grad=True for some reason — the loss would build a graph with respect to y, and a backward pass would compute y.grad (which the learner does not want and probably does not notice). The y.grad accumulation is a silent device-retention-like bug: it consumes memory for a gradient that is never used, and it is invisible unless the learner inspects y.grad.

§L.4.3 — Comparative table: .detach() vs. torch.no_grad()

Design principle (P2.4). No graph unless you ask for one. In any code path whose purpose is to produce values for use — not to produce gradients — wrap the whole path in torch.no_grad(). In any code path whose purpose is to produce a gradient, do not wrap it, and use .detach() on the specific tensors that must be constants. The two paths are different in their default, and the default should be stated by the path’s structure, not by remembering to add a flag.

§L.4.4 — Concept check

Concept check (displaces M2, part 2). A learner wraps an entire training step — forward, loss, backward — in torch.no_grad() and reports that the loss is 0.0 and the model does not learn. What did the block do, and what is the correct scope for the no_grad in a training loop that also collects rollouts?

Answer. The no_grad block disabled graph construction for the entire step, so the loss is a graph-free constant, the backward pass has no graph to traverse, and the optimizer’s update is based on zeros (or the backward raises, depending on whether the loss’s requires_grad is checked). The correct scope is the rollout collection only: the rollout’s forward passes are wrapped in no_grad, and the training step’s forward, loss, and backward are outside the block, so the graph is built for the parameters that are being updated.

§L.4.5 — Consolidation checkpoint 4

(a) You should now be able to: (i) state the effect of torch.no_grad() on graph construction, memory, and the requires_grad of the outputs; (ii) choose between .detach() and torch.no_grad() for a given gradient-stopping requirement; (iii) place the no_grad boundary correctly in a loop that has both a rollout phase and a training phase.

(b) Transfer probe. A learner’s on-policy RL loop collects rollouts and trains on them. The rollout collection is inside no_grad, the training step is outside, and the loop runs correctly. The learner then adds a value function’s bootstrapped target that is computed from the next observation through the same value network, and the target is used in the value loss. Does the bootstrap target need no_grad, .detach(), or both, and where in the loop?

Answer sketch: the bootstrap target is r + γ · V(s') where V is the value network being trained; the target must be a constant with respect to the value network’s parameters for this update, so V(s') must be detached. Because V(s') is computed inside the training step (the next observation is available from the rollout), the no_grad block does not cover it — the training step is outside no_grad by design. The correct operation is .detach() on V(s'), not a no_grad block, because the forward pass for V(s') must happen inside the training step and the target must be a constant. The no_grad-around-rollout pattern handles the rollout’s forward passes; the .detach()-on-target pattern handles the training step’s target. The two are different scopes and different tools.

(c) Remediation rule. If you cannot do (b), return to §L.4.2 and redo WE-L.4.b, then re-attempt (b) by asking, for each forward pass, whether its output needs gradients and whether its output is used as a target.

Part E — The Target-Network Pattern, the Failure Taxonomy, and the Lab

§L.5 — The target-network pattern and its silent failure modes

The target-network pattern is the canonical place in embodied AI where gradient stopping is an algorithmic correctness requirement, not a performance optimization. The pattern appears in every deep Q-learning variant, in soft actor-critic’s value target, in TD3’s twin-critic targets, in offline RL’s conservative value penalty, in behavior cloning’s KL-regularized reference policy, and in the world-model literature’s latent-prediction targets.

The simplified Q-learning update is:

where θ are the online network’s parameters and θ̄ are the target network’s parameters. The target network’s parameters are updated periodically (every C steps) or softly (θ̄ ← τθ + (1-τ)θ̄), and are not updated by the optimizer’s step on L.

The gradient-stopping requirement is on the target tensor y = r + γ max_{a'} Q_{\barθ}(s', a'). For the update to be the TD update, y must be a constant with respect to θ. If y is computed through the online network — because the target network is a shared module, because the target computation was not detached, or because the online network was called for both the prediction and the target — then the optimizer’s step changes y as well, and the update is not a TD update but a self-consistent minimization of a moving objective.

The three silent failure modes in the pattern are:

1. Missing .detach() on the target. The target is computed by target_net(s'), and if target_net’s parameters require grad (which they do, by construction, because the target network is a copy of the online network), the target tensor has requires_grad=True, the loss’s gradient flows into both the online and the target networks, and the optimizer’s step changes the target network’s parameters. The target is no longer delayed; it moves with the online network at every step. The fix is target = target_net(s').detach(), or wrapping the target computation in torch.no_grad().
2. The target network’s parameters are in the optimizer’s parameter list. A learner who constructs the optimizer as Adam(model.parameters(), ...) where model is a wrapper that includes both the online and the target networks will update both, and the target will track the online at every step. The fix is to construct the optimizer with only the online network’s parameters.
3. The target buffer alias. The learner updates the target network by target_param = online_param.detach() (the near-miss of WE-L.3.d), which shares the online parameter’s buffer, and the target tracks the online in real time despite the .detach(). The fix is .data.copy_() on an existing buffer, or an explicit clone.

§L.5.1 — The failure taxonomy

The following table is the lesson’s debugging map. It covers every break-it step in the lab plus two failures not exercised there (the CPU-tensor case and the non-contiguous-view case from Lesson 0.1.1, both of which are gradient-adjacent and recur in training loops).

§L.6 — The lab

§L.6.1 — Task

Build a small training loop for the running case — a 6-DOF arm, 8 environments, 50-step rollout — that computes a linear policy’s loss, backpropagates, and steps the optimizer; and use the loop to reproduce, predict, and fix three gradient bugs. The loop is the first half of the pipeline that Lesson 0.1.3 will complete; the learner will extend this loop in 0.1.3 with DataLoader, nn.Module, and the full training/eval structure.

§L.6.2 — Conditions

A machine with Python 3.10+ and PyTorch 2.2+. The lab runs on CPU in under 10 minutes. All runs use torch.manual_seed(42). The policy is a single nn.Linear(6, 6) (no hidden layer) so that the gradients can be computed by hand and compared to the framework’s output. The loss is mse(policy(obs), action_target) where action_target is a fixed random tensor.

§L.6.3 — Controls held fixed

The seed is 42. The number of environments (8), the observation dimension (6), the action dimension (6), and the learning rate (0.01) are fixed. The loss is the mean of squared error (not the sum), so the hand-computed gradients in the exercises match the lab’s. The number of training steps is 50 for all runs.

§L.6.4 — Metrics table

§L.6.5 — Pre-registered hypotheses

State each of the following and predict the number before running any code.

• H1. grad_norm_step1 on the first batch will be ≈ 0.764, not 3√6. Two different quantities are easy to conflate here and the lesson separates them: the toy hand case (x0 of shape (1, 3), W0 of shape (3, 2), all-ones, relu saturated) has W0.grad = [[3,3],[3,3],[3,3]] and norm 3√6 ≈ 7.348; the lab’s own W is (6, 6) with 48 output elements averaged by mse_loss, so its first-batch gradient norm is an order of magnitude smaller. The prediction for the lab’s W.grad is 0.7642, falsified if it differs by more than 1e-3. The analytic value is ‖(2/(N·D))·(pred − target)ᵀ @ obs‖ with N = 8, D = 6, which is a check the learner can compute by hand without the framework.
• H2. grad_norm_step2_zeroed will be smaller than grad_norm_step2_no_zero. If the two batches were identical the ratio would be exactly 2; if the gradients are independent the ratio is √2 ≈ 1.414 in expectation, with variance because each is a norm of a random vector. The prediction is the range 1.2 to 1.8; measured here, 1.7511 — inside the range, and its position inside the range carries no information, because it is one draw.
• H3. target_weight_drift is 0.0 with the detach and > 0.01 without it — but only when the target’s parameters require grad and are in the optimizer. If the target is frozen (requires_grad_(False) and excluded from the optimizer) the missing detach changes nothing observable: drift is 0.0 in both conditions. The prediction is therefore conditional, and the frozen-target run is the control that fails to discriminate — a learner who runs only that version concludes, wrongly, that the detach is optional. Measured: frozen 0.000000 both ways; unfrozen 0.000000 with detach, 0.423201 without.
• H4. double_backward_grad_norm with retain_graph=True and no zero will be exactly twice the single-backward norm for the same batch — ratio 2.0000, because the second backward adds an identical gradient to an accumulator that was never cleared. The prediction is the ratio, not an absolute number: measured 1.5284 / 0.7642 = 2.0000.

§L.6.6 — The break-it sequence

Three deliberate corruptions, ordered by subtlety. The learner writes down the predicted symptom before running each, then records the actual.

1. The non-scalar loss (loud). Change loss = mse(pred, target) (which returns a scalar) to loss = (pred - target).pow(2) (which returns a (8, 6) tensor), and call .backward() on it. Predicted symptom: a RuntimeError with the text “grad can be implicitly created only for scalar outputs,” because the framework refuses to backpropagate from a non-scalar without an explicit gradient argument. Actual symptom: the RuntimeError — this is the loud case.
2. The missing zero_grad (silent). Run the training loop for 50 steps with optimizer.zero_grad() removed. Predicted symptom: no exception; the gradients accumulate across steps; grad_norm_step2_no_zero is larger than grad_norm_step2_zeroed; the loss trajectory differs from the zeroed run’s, and the final loss is different. Actual symptom: as predicted — silent.
3. The missing target detach (silent). Implement a one-step Q-learning update with a target network whose parameters require grad and are in the optimizer, and run 50 steps with target = target_net(s_next) (no detach) and then with .detach(). Predicted symptom: no exception in either run; in the no-detach run, target_net.weight.grad is not None after the backward and the target’s weights drift by 0.42 in norm, so the “target” is being trained alongside the online network. Actual symptom: as predicted — silent. The control matters: with the target frozen (requires_grad_(False), absent from the optimizer) the drift is 0.0 both ways, so that version of the experiment cannot see the bug and a learner who runs only it draws the wrong conclusion.

Two of the three — steps 2 and 3 — are silent, satisfying the requirement that at least two break-it steps raise no exception.

§L.6.7 — Acceptance criteria

§L.6.8 — What good looks like

A strong submission has five properties:

1. The hand computation is shown, not just the framework’s output. The notebook contains the chain-rule derivation for the first backward pass with the arithmetic shown, and a numerical comparison to W.grad. The learner who reports only “the gradients match” has not demonstrated the mechanism.
2. The predictions are numerical and timestamped before the runs. The four hypotheses are predictions with numbers or ranges, not “it will be larger.” The predictions are in a separate file or a timestamped cell, so the learner cannot revise them after observing.
3. The silent failures are identified as silent, with the mechanism named. The learner writes, for each of steps 2 and 3, why the failure does not raise, and what the observable difference is (a larger gradient, a nonzero target drift). The learner who writes “the bug was that the gradient was wrong” without naming the accumulation has not demonstrated the mechanism.
4. The target-network experiment distinguishes drift from gradient. The learner reports both target_net.weight.grad (which should be None with detach) and target_weight_drift (which should be 0.0 with detach between scheduled updates), and explains that the first is a graph property and the second is a value property. The two are different bugs with different fixes.
5. The fixes are one line each, and the learner states what the line does. “Add .detach()” is not a fix unless the learner can say that .detach() removes the target tensor from the graph so the optimizer’s step does not reach the target network’s parameters. The demonstration is the fix’s rationale, not the diff.

§L.6.9 — Runnable lab code

# lab_0_1_2.py — Gradients, accumulation, detach, and no_grad on the running case
# Requires: Python 3.10+, PyTorch 2.2+

import copy
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(42)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"device = {device}")

# --- Step 1: the small policy and the first batch -------------------------

policy = nn.Linear(6, 6).to(device)
W, b = policy.weight, policy.bias                    # leaves with requires_grad=True
print(f"W.is_leaf = {W.is_leaf}, W.grad_fn = {W.grad_fn}")
print(f"W.shape = {W.shape}, W.requires_grad = {W.requires_grad}")

obs = torch.randn(8, 6, device=device)              # 8 environments, 6 joints
target = torch.randn(8, 6, device=device)           # a fixed action target

# --- Step 2: the forward pass and the graph ------------------------------

pred = policy(obs)
print(f"pred.is_leaf = {pred.is_leaf}, pred.grad_fn = {pred.grad_fn}")
print(f"pred.requires_grad = {pred.requires_grad}")

loss = F.mse_loss(pred, target)
print(f"loss.shape = {loss.shape}, loss.grad_fn = {loss.grad_fn}")

# --- Step 3: retain_grad on a non-leaf, before the backward --------------
pred.retain_grad()
loss.backward()
print(f"pred.grad is None? {pred.grad is None}")     # Expected: False (retained)
print(f"W.grad is None? {W.grad is None}")           # Expected: False (leaf)
print(f"loss.grad is None? {loss.grad is None}")     # Expected: True (non-leaf, not retained)

# --- Step 4: hand-computed gradient comparison --------------------------

# For the first batch, we recompute the gradient with a closed-form expression
# that mirrors the arithmetic in WE-L.1.a, using the same input structure.
x0 = torch.tensor([[1.0, 1.0, 1.0]], device=device)
W0 = torch.ones(3, 2, device=device, requires_grad=True)
b0 = torch.ones(1, 2, device=device, requires_grad=True)
t0 = torch.ones(1, 2, device=device)
y0 = F.mse_loss(torch.relu(x0 @ W0 + b0), t0)
y0.backward()
print(f"hand case loss = {y0.item():.6f}")           # Expected: 9.0
print(f"W0.grad = {W0.grad}")                        # Expected: [[3,3],[3,3],[3,3]]
print(f"b0.grad = {b0.grad}")                        # Expected: [3,3]
print(f"W0.grad.norm() = {W0.grad.norm().item():.4f}")   # Expected: 7.3485 = 3*sqrt(6)

# The toy case above is NOT the lab's W. The lab's W is (6, 6) with 48 outputs
# averaged by mse_loss, so its gradient is much smaller. Compare the framework's
# real W.grad against the closed form dL/dW = (2/(N*D)) * (pred - target)^T @ obs.
analytic_W = (2.0 / (obs.shape[0] * obs.shape[1])) * ((pred - target).detach().t() @ obs)
analytic_b = (2.0 / (obs.shape[0] * obs.shape[1])) * (pred - target).detach().sum(0)
hand_diff = (analytic_W - W.grad).abs().max().item()   # captured now: W.grad is overwritten later
print(f"W.grad.norm() = {W.grad.norm().item():.6f}")            # Expected: 0.764189
print(f"analytic W max abs diff = {hand_diff:.2e}")             # Expected: < 1e-6
print(f"analytic b max abs diff = {(analytic_b - b.grad).abs().max().item():.2e}")  # < 1e-6

# --- Step 5: gradient accumulation, with and without zero_grad ----------

def run_two_steps(zero_between: bool):
    W.grad = None; b.grad = None
    obs1 = torch.randn(8, 6, device=device)
    tgt1 = torch.randn(8, 6, device=device)
    obs2 = torch.randn(8, 6, device=device)
    tgt2 = torch.randn(8, 6, device=device)

    loss1 = F.mse_loss(policy(obs1), tgt1)
    loss1.backward()
    if zero_between:
        W.grad = None; b.grad = None
    loss2 = F.mse_loss(policy(obs2), tgt2)
    loss2.backward()
    return W.grad.norm().item()

torch.manual_seed(42)
norm_zeroed = run_two_steps(zero_between=True)
torch.manual_seed(42)
norm_no_zero = run_two_steps(zero_between=False)
print(f"norm_zeroed = {norm_zeroed:.4f}, norm_no_zero = {norm_no_zero:.4f}")
print(f"ratio = {norm_no_zero / norm_zeroed:.4f}")   # Expected: > 1, typically ~1.2-1.8

# --- Step 6: H4 — double backward with retain_graph ----------------------

torch.manual_seed(42)
W.grad = None; b.grad = None
single_loss = F.mse_loss(policy(obs), target)
single_loss.backward()
single_norm = W.grad.norm().item()

torch.manual_seed(42)
W.grad = None; b.grad = None
single_loss = F.mse_loss(policy(obs), target)
single_loss.backward(retain_graph=True)
single_loss.backward()
double_norm = W.grad.norm().item()
print(f"single_norm = {single_norm:.4f}, double_norm = {double_norm:.4f}")
print(f"double / single = {double_norm / single_norm:.4f}")   # Expected: 2.0

# --- Step 7: detach vs. no_grad on the same computation -----------------

torch.manual_seed(42)
pred_detach = policy(obs).detach()
print(f"detach: requires_grad = {pred_detach.requires_grad}, grad_fn = {pred_detach.grad_fn}")

with torch.no_grad():
    pred_no_grad = policy(obs)
print(f"no_grad: requires_grad = {pred_no_grad.requires_grad}, grad_fn = {pred_no_grad.grad_fn}")
# Both should be False / None; the difference is that no_grad did not build a graph

# --- Step 8: detach shares storage, clone does not -----------------------

torch.manual_seed(42)
a = policy(obs)
a_detached = a.detach()
a_cloned = a.detach().clone()
print(f"detached shares buffer: {a_detached.data_ptr() == a.data_ptr()}")  # Expected: True
print(f"cloned shares buffer:   {a_cloned.data_ptr() == a.data_ptr()}")     # Expected: False

a_detached[0, 0] = 99.0
print(f"a[0,0] after writing to detached: {a[0, 0].item()}")                 # Expected: 99.0
a_cloned[0, 0] = -1.0
print(f"a[0,0] after writing to cloned:   {a[0, 0].item()}")                 # Expected: 99.0 (unchanged)

# --- Step 9: break-it #1 — the non-scalar loss (loud) -------------------

torch.manual_seed(42)
pred = policy(obs)
non_scalar = (pred - target).pow(2)                 # shape (8, 6), not a scalar
try:
    non_scalar.backward()
except RuntimeError as e:
    print(f"break-it #1 raised as expected: {e}")

# --- Step 10: break-it #2 — the missing zero_grad (silent) -------------

torch.manual_seed(42)
policy2 = nn.Linear(6, 6).to(device)
opt2 = torch.optim.SGD(policy2.parameters(), lr=0.01)
losses_no_zero = []
for step in range(50):
    batch_obs = torch.randn(8, 6, device=device)
    batch_tgt = torch.randn(8, 6, device=device)
    loss = F.mse_loss(policy2(batch_obs), batch_tgt)
    loss.backward()                                  # no zero_grad — silent accumulation
    opt2.step()
    losses_no_zero.append(loss.item())
print(f"break-it #2 final loss (no zero): {losses_no_zero[-1]:.4f}")

torch.manual_seed(42)
policy3 = nn.Linear(6, 6).to(device)
opt3 = torch.optim.SGD(policy3.parameters(), lr=0.01)
losses_zero = []
for step in range(50):
    batch_obs = torch.randn(8, 6, device=device)
    batch_tgt = torch.randn(8, 6, device=device)
    loss = F.mse_loss(policy3(batch_obs), batch_tgt)
    opt3.zero_grad()
    loss.backward()
    opt3.step()
    losses_zero.append(loss.item())
print(f"break-it #2 final loss (zeroed): {losses_zero[-1]:.4f}")
# The two finals differ; neither run raised; the difference is the accumulation

# --- Step 11: break-it #3 — the missing target detach (silent) ---------

# The target network's parameters must require grad AND be handed to the
# optimizer for the bug to be observable at all. In the frozen-target form
# (requires_grad_(False), no optimizer entry) the missing detach changes
# nothing: the target contributes no gradient and cannot drift. That is the
# trap -- a learner whose target is frozen sees this "fix" pass and concludes
# the detach is unnecessary.
def q_learning_run(detach_target: bool, freeze_target: bool, steps: int = 50):
    torch.manual_seed(0)
    online = nn.Linear(4, 2, device=device)
    target_net = copy.deepcopy(online).to(device)
    if freeze_target:
        for p in target_net.parameters():
            p.requires_grad_(False)
    params = list(online.parameters())
    if not freeze_target:
        params += list(target_net.parameters())
    opt = torch.optim.Adam(params, lr=1e-2)
    w0 = target_net.weight.detach().clone()
    for step in range(steps):
        s = torch.randn(16, 4, device=device)
        s_next = torch.randn(16, 4, device=device)
        r = torch.randn(16, 1, device=device)
        a = torch.randint(0, 2, (16, 1), device=device)
        q = online(s).gather(1, a)
        tgt = target_net(s_next)                     # built with grad
        if detach_target:
            tgt = tgt.detach()                       # the fix
        y = r + 0.99 * tgt.max(dim=1, keepdim=True).values
        loss = F.mse_loss(q, y)
        opt.zero_grad()
        loss.backward()
        opt.step()
    drift = (target_net.weight.detach() - w0).norm().item()
    return drift, target_net.weight.grad is not None

for frozen in (True, False):
    for detached in (True, False):
        drift, has_grad = q_learning_run(detach_target=detached, freeze_target=frozen)
        print(f"frozen={frozen!s:5s} detach={detached!s:5s} -> "
              f"drift={drift:.6f}, target.weight.grad is not None={has_grad}")
# Expected (verified):
#   frozen=True  detach=True  -> drift=0.000000, grad=False   <- the trap: both look "fixed"
#   frozen=True  detach=False -> drift=0.000000, grad=False   <- no signal either
#   frozen=False detach=True  -> drift=0.000000, grad=False   <- the fix, now observable
#   frozen=False detach=False -> drift=0.423201, grad=True    <- the bug, now observable

# --- Step 12: the summary ----------------------------------------------

print("\n--- lab 0.1.2 summary ---")
print(f"device: {device}")
print(f"W is a leaf: {W.is_leaf}, pred is a leaf: {pred.is_leaf}")
print(f"analytic-vs-framework max abs diff: {hand_diff:.2e}")
print(f"accumulation ratio (no-zero / zeroed): {norm_no_zero / norm_zeroed:.4f}")
print(f"double-backward ratio: {double_norm / single_norm:.4f}")
print(f"break-it silent steps: #2 (accumulation), #3 (target detach)")

Run the file end to end. The output should match the expected comments. If any step does not match, the failure taxonomy in §L.5.1 names the likely causes.

§L.7 — The cheat sheet

§L.8 — Exercises with solutions

E 0.1.2.1 — The hand-computed gradient (numerical). A linear layer has W = [[1, 2], [3, 4]] (shape (2, 2)), b = [0, 0], and input x = [1, 1] (shape (1, 2)). The loss is mse(x @ W + b, [1, 1]). Compute W.grad and b.grad by hand, showing each intermediate.

S 0.1.2.1. Forward: z = x @ W = [1·1 + 1·3, 1·2 + 1·4] = [4, 6]. z + b = [4, 6]. diff = [4-1, 6-1] = [3, 5]. sq = [9, 25]. loss = (9+25)/2 = 17. Backward: ∂loss/∂sq = [1/2, 1/2]. ∂loss/∂diff = [2·3·(1/2), 2·5·(1/2)] = [3, 5]. ∂loss/∂z = [3, 5]. ∂loss/∂W = x^T · [3, 5] = [[3, 5], [3, 5]]. ∂loss/∂b = [3, 5]. The lesson: the gradient’s rows are the input’s contributions, and the gradient’s columns are the output’s contributions; the W.grad[i, j] entry is x[i] · grad_z[j], which is the outer product of the input and the output gradient.

E 0.1.2.2 — The diagnostic exercise (diagnostic). A learner’s training loop runs without error, but the loss decreases to a value that is half the value they expected from a reference implementation. Their loss is F.mse_loss(pred, target) where pred and target are (8, 6). The reference implementation uses (pred - target).pow(2).sum(). What is the discrepancy, and what is the ratio between the gradients?

S 0.1.2.2. F.mse_loss with the default reduction='mean' divides by the number of elements (8·6 = 48); the reference implementation’s sum() does not divide. The ratio between the two losses is 1/48, not 1/2 — the learner’s stated “half” is wrong, and the discrepancy is a factor of 48. The gradient ratio is also 1/48: the reference implementation’s gradient is 48× larger, and a training loop using it with the same learning rate will take 48× larger steps and diverge. The fix is to use the same reduction on both sides, or to scale the learning rate by the reciprocal of the reduction factor. The lesson: the reduction’s normalization is a learning-rate multiplier in disguise; a change in the reduction is a change in the optimization, and the two implementations that “compute the same loss” do not, because the reduction is part of the loss.

E 0.1.2.3 — The construct-the-target exercise (transfer probe). A learner implements a behavior cloning objective with a KL penalty to a reference policy: loss = mse(policy(obs), action) + β · KL(policy(obs), ref_policy(obs)). The ref_policy is a frozen copy of the policy at the start of training. Write the pseudocode for the loss and state which quantities must be detached, and why.

S 0.1.2.3.

ref_logits = ref_policy(obs).detach()          # must be detached: the reference is fixed
policy_logits = policy(obs)                    # must NOT be detached: the policy is being trained
bc_loss = mse(policy_logits, action)
kl_loss = KL(policy_logits.log_softmax(-1), ref_logits.softmax(-1))
loss = bc_loss + beta * kl_loss

The ref_logits must be detached, because the KL is a target quantity: the reference policy is fixed at the start of training, and the gradient must flow only through policy_logits. If ref_logits is not detached, the gradient flows into ref_policy’s parameters, and — if ref_policy’s parameters are in the optimizer’s list, or if they share storage with policy (the deep-copy-aliasing hazard of WE-L.3.d) — the reference drifts and the KL penalty becomes a moving objective. The policy_logits must not be detached, because it is the quantity being trained. The lesson: in a loss with a learnable term and a fixed term, the fixed term must be detached, and the detachment is a statement about the term’s role, not about the term’s parameters.

E 0.1.2.4 — The diagnostic exercise (diagnostic). A learner’s Q-learning agent’s TD loss decreases, but the agent’s performance is at chance, and the Q-values grow without bound. They show you their target computation: target = reward + gamma * target_net(next_obs).max(dim=1).values. They show you that target_net’s parameters are not in the optimizer’s parameter list. What is the most likely cause?

S 0.1.2.4. The target tensor is computed from target_net(next_obs) without .detach(), and target_net’s parameters, while not in the optimizer’s list, do require grad (they are an nn.Parameter created by copy.deepcopy of the online network, and deepcopy preserves requires_grad=True). The loss’s gradient therefore flows into target_net’s parameters, and the optimizer’s step on online’s parameters does not reach them — but the target tensor is a function of the target network, and the target network is not updated, so the gradient accumulates into target_net’s .grad and is never zeroed or applied. The more important consequence is that the target is not a constant with respect to online through the shared graph if target_net is the same module as online (a common implementation shortcut), in which case the online network’s update changes the target, and the TD update is not a TD update. The fix is target = reward + gamma * target_net(next_obs).detach().max(dim=1).values (or wrap the target computation in torch.no_grad()), and — if the target network shares parameters with the online network — separate them. The lesson: “the target’s parameters are not in the optimizer’s list” is not the same as “the target is constant”; the constancy is a property of the graph, and the .detach() is what makes it so.

E 0.1.2.5 — The pseudocode exercise (constructive). Write pseudocode for a PPO inner loop that performs 4 epochs over a rollout, with the correct zero_grad placement, and state the effective learning rate relative to a single-epoch loop.

S 0.1.2.5.

for epoch in range(4):                  # 4 epochs over the same rollout
    for minibatch in rollout.split(batch_size):
        optimizer.zero_grad()            # once per minibatch — inside the epoch loop
        loss = ppo_loss(policy, minibatch)
        loss.backward()
        optimizer.step()

The zero_grad is inside the epoch loop, once per minibatch, because each minibatch’s gradient is a separate update and must not accumulate with the previous one. The effective learning rate is the same as a single-epoch loop’s, because the four epochs apply four independent updates, not one accumulated update. The near-miss — zero_grad outside the epoch loop — accumulates four epochs of gradients and applies a 4× update once per rollout, which is a 4× learning rate increase. The lesson: the zero_grad placement defines the unit of accumulation, and the unit is a design decision, not a convention; the learner who copies a loop template without understanding the unit has a loop whose effective learning rate is not the one in the code.

E 0.1.2.6 — The transfer probe (transfer). A learner wants to compute a gradient penalty for a Wasserstein GAN, which requires the gradient of the critic’s output with respect to its input. They write critic_out = critic(x); grad = torch.autograd.grad(critic_out.sum(), x, create_graph=True)[0]. What does create_graph=True do, and why is it necessary here? What would happen without it?

S 0.1.2.6. create_graph=True builds a graph of the backward pass, so that the gradient grad is itself a differentiable function of the critic’s parameters, and the subsequent grad.norm().backward() can compute the gradient of the penalty with respect to those parameters. Without it, grad is a constant tensor (graph-free), and the penalty’s .backward() would raise (no graph to traverse) or would silently produce zeros for the parameters — the create_graph flag is what makes the second-order gradient possible. The penalty’s gradient is a second derivative of the critic, and second derivatives require differentiating through the first derivative’s computation. The lesson: create_graph is the flag for higher-order gradients, and it is necessary whenever a gradient is used as a value in a differentiable expression — not merely inspected. The flag is expensive (it doubles the graph) and should be used only when the second derivative is genuinely needed.

E 0.1.2.7 — The transfer probe (transfer). A learner uses .detach() in an inference loop and reports that the loop is slow and uses more memory than expected. They are correct that .detach() removes the graph connection. Why is it the wrong tool for an inference loop, and what is the right one?

S 0.1.2.7. .detach() removes a tensor from the graph after the graph was built — the forward pass still constructed the grad_fn nodes and saved the intermediate activations for the backward pass, and those activations are only freed when the graph is garbage-collected. In an inference loop, the graph is built and discarded on every batch, which costs memory (the saved activations) and latency (the graph construction and destruction) for no benefit. The right tool is torch.no_grad(), which prevents the graph from being built, so the saved activations are never allocated and the grad_fn nodes are never created. The memory saving is the size of the forward pass’s activations; the latency saving is the graph bookkeeping, which in a small model is a small fraction of the forward time but in a large VLA is measurable. The lesson: .detach() is for one tensor in an otherwise-differentiable expression; no_grad is for a block in which no gradient is needed. The learner who uses .detach() in an inference loop has paid the graph’s cost and then discarded it.

E 0.1.2.8 — The diagnostic exercise (diagnostic). A learner reports that their policy’s gradient is exactly twice the value they compute by hand, on every step, with no other anomaly. They call .backward() once per step. What is the most likely cause?

S 0.1.2.8. The loss contains a term that is computed twice — a double-counted term, a loss that is added to itself (loss = bc_loss + bc_loss), or a .backward() that is called on a tensor whose graph includes the same parameter through two paths (a residual connection whose skip path and main path both reach the same parameter). The “exactly twice” signature is diagnostic: an accumulation bug would give a growing multiple (2×, then 3×, then 4×, as steps accumulate), not a constant 2×; a reduction bug would give a ratio equal to the reduction factor (a constant, but the learner would need to know the reference’s reduction to see it). A constant 2× on every step is a doubled computation in the loss or in the graph. The fix is to trace the loss expression’s terms and the graph’s paths to the parameter, and to confirm whether either reaches the parameter twice. The lesson: the time signature of a gradient bug — whether the multiple is constant, growing, or scaling with a known factor — is the discriminator between causes; the learner who reports only “the gradient is wrong” has discarded the most informative diagnostic.

§L.9 — Exit test

You are ready for Lesson 0.1.3 when you can, without notes:

1. State the difference between a leaf and a non-leaf, and decide which one a given tensor is from the code that created it.
2. Trace a three-layer forward pass and name the grad_fn of each intermediate tensor.
3. Compute by hand the gradients of a small linear layer’s loss with respect to its weight and bias, and match the framework’s .grad to within 1e-6.
4. State the two reasons a second .backward() on the same loss is not a “recomputation,” and the flag that addresses only one of them.
5. State the placement of zero_grad that produces “the gradient of the sum” and the placement that produces “the gradient of the last loss,” and give a scenario for each.
6. State the two properties of .detach() and the operation that adds the third, and give a code snippet that demonstrates the aliasing hazard.
7. Distinguish .detach() from torch.no_grad() by scope, graph construction, and memory cost.
8. State the target-network pattern, which tensor must be detached, and what happens to the Q-values if it is not.
9. State the difference between “the target’s parameters are not in the optimizer’s list” and “the target is constant,” and give a scenario where the first is true and the second is false.
10. Explain why .detach() is the wrong tool for an inference loop, and what the right tool is.
11. Diagnose a constant 2× gradient and a growing multiple, and name the cause each signature suggests.
12. State the create_graph=True flag’s purpose and one algorithmic context in which it is required.

If any of items 1–12 is not answerable without notes, return to the section that introduced it: item 1 to §L.1, item 2 to §L.1.2, item 3 to §L.1.2, item 4 to §L.2, item 5 to §L.2.2, item 6 to §L.3, item 7 to §L.4.3, item 8 to §L.5, item 9 to E 0.1.2.4, item 10 to E 0.1.2.7, item 11 to E 0.1.2.8, and item 12 to E 0.1.2.6.

§L.10 — Provenance and evidence tiers

What this lesson reproduces. Every hypothesis in §L.6.5 is executed by the lab in §L.6.9, and the numbers in the text are the ones the lab prints. The lab is the evidence; tier A, status R.

What this lesson does not reproduce. The claim that the target-network pattern is an algorithmic correctness requirement — that a missing detach degrades Q-learning, SAC, TD3, or a KL-regularized reference policy in practice — is asserted from the literature and is not measured here. It is tier B, status S: well established elsewhere, not reproduced in this lesson. What the lesson measures is narrower and sufficient for its argument: that omitting .detach() lets gradient flow through the target and moves the target’s parameters by a measurable amount. The step from that mechanism to “this breaks the algorithm” is the literature’s, not the lab’s.

The H3 condition is the load-bearing one. The drift is > 0.01 only when the target’s parameters require grad and are in the optimizer. A frozen target shows no drift, and a learner who tests the detach against a frozen target concludes that detach is unnecessary. The lab states this condition in the hypothesis rather than in a footnote, because the near-miss — detach looks pointless — is the failure mode.

No external sources are cited, and the argument does not need them. Every number in the text comes from the lab, and the one literature-derived claim is marked above rather than woven into a derivation.