Lesson 3.2.2 — Video Prediction Models as World Models

Week 21

Prerequisite lessons: 3.2.1 (latent world models, RSSM, Dreamer), 2.2.2 (model-based RL and MBPO), 2.1.1 (camera models and image formation), 3.1.3 (data collection and splits), 3.1.4 (evaluation discipline and forgetting).

Prior results used in this lesson. From 3.2.1: the latent-state formalism, the prior–posterior split, the coverage-gap ratio, and the horizon discipline that imagined returns are only trustworthy over short rollouts. From 2.1.1: the projection model $\tilde{\mathbf{p}} = \frac{1}{Z}\mathbf{K}\mathbf{X}_C$ and the fact that a pixel is a projection of state, not the state. From 2.2.2: the short-rollout result that model error compounds with length. From 3.1.4: the requirement that evaluation be measured on a held-out split that the model did not see. Each is restated at the point of use.

What this lesson hands forward. The pixel-versus-latent comparison, the action-conditioning requirement, the FVD and control-success evaluation pair, and the decision rule for when a video model is the right forward model. Lesson 3.2.3 (planning with world models) uses the forward model selected here. Module 3.4 (sim-to-real) contrasts a learned video simulator with a physics simulator.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, and divider are unchanged; this lesson asks whether a model that predicts the camera’s pixels is a better forward model for the running case than the RSSM of 3.2.1, which predicts only a latent.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) state the three roles a video model can play and which role each architecture serves; (2) compute the dimensionality and storage of a pixel-space prediction and compare it to a latent prediction; (3) distinguish deterministic from stochastic video models and state what blur means in each; (4) explain why an action-conditional model is required for control and how Finn et al. 2016 established it; (5) name the four architecture families and their inference costs; (6) state the objective–control gap and give two examples of a perceptually good model that is a poor controller; (7) write the FVD definition at the level of its inputs and outputs and state what it does not measure; (8) design a control-relevant evaluation with a rollout test and a compounding-error metric; (9) apply the head-to-head decision rule to the running case; (10) state the composition in which a video model and an RSSM coexist.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 3.2.1, the RSSM’s latent is a control statistic, not a compressed image. What does the observation model $p(o_t|h_t,z_t)$ exist for, if the latent is not meant to reconstruct?

Answer. It exists as a training signal: the reconstruction term of the ELBO forces the latent to carry enough information to explain the observation, which is the pressure that prevents the latent from collapsing to a trivial code. Used in §L.A.2 to distinguish a latent model’s decoder from a video model’s output.

Q2. In 3.2.1, the horizon discipline says imagined returns are only trustworthy over 15–50 steps. What is the mechanism, and does it depend on whether the model predicts pixels or latents?

Answer. The mechanism is compounding error: each step’s prediction error feeds the next step’s input. It applies to both pixel and latent models, but the error’s magnitude per step differs, and Part A computes how. Used in §L.A.2 and §L.D.3.

Q3. In 2.1.1, a pixel is a projection of a 3D point. What information does the projection discard, and why does that matter for a video model asked to plan a grasp?

Answer. It discards depth and the point’s 3D position; two different 3D configurations can produce the same pixel. A video model that predicts pixels can be consistent with two different grasps, and its prediction cannot distinguish them. Used in §L.C.1 to explain the objective–control gap.

Q4. In 3.1.4, the held-out split is by episode because adjacent frames leak. What is the analogous leakage in evaluating a video model?

Answer. Evaluating on frames or clips from the same episode as training leaks the dynamics and inflates the metric; the split must be by episode, and the metric must be measured on rollouts that the model did not see. Used in §L.D.2.

Part A — What a video model buys and what it costs

§L.A.1 — The appeal: no state estimator

The RSSM of 3.2.1 requires an encoder that maps the observation to a latent, a dynamics model over the latent, and a decoder that is trained but not used for control. A video model removes the encoder and the latent: it predicts the next observation directly, $p_\theta(o_{t+1} | o_{1:t}, a_{1:t})$.

The appeal is architectural simplicity. There is no latent to design, no encoder to train, no prior–posterior split, and no KL balancing. The model’s output is the same space as its input, so the training signal is a direct comparison. For a deployment that needs to show a human what the model expects — a visualization, a human-in-the-loop check, an operator’s preview — the video model is the natural output.

The second appeal is generality of the observation model. A latent model’s encoder is trained on a specific camera and a specific scene; a video model that predicts pixels is trained on the same camera but does not need to commit to what the latent should contain. In a setting where the task varies and the relevant state is not known in advance, this is a real advantage.

The third appeal is that modern generative video models are extremely good at the distribution of futures. Diffusion video models (Ho et al. 2022, tier A, S; Blattmann et al. 2023, tier B, S) produce samples that are visually indistinguishable from real video over several seconds. That quality is a genuine capability, and it is the reason the family is worth studying even when the latency is prohibitive for control.

§L.A.2 — The cost: dimensionality, blur, compounding

The arithmetic is unforgiving. A single frame at the eye-in-hand camera’s native resolution is

A 30-step rollout at 10 Hz predicts $30 \times 921{,}600 = 27{,}648{,}000$ numbers. At one byte each that is 27.6 MB per imagined trajectory, versus 30 latent vectors of 64 floats, or about 7.7 KB, in the RSSM. The ratio is roughly 3,600:1, the same ratio computed in 3.2.1 §L.A.1.

The cost is not only storage. It is what the loss weights. A reconstruction or diffusion loss weights every pixel. The table’s texture, the lighting gradient, and the bin’s rim are large, high-contrast, and easy to predict; the cube’s edge is a few hundred pixels and the contact patch is invisible. The model therefore spends its capacity where the loss is largest, which is not where the control is.

The second cost is blur. A deterministic video model trained with an MSE loss learns the conditional mean of the future. When the future is multimodal — the cube can slip left or right in the gripper — the mean is a blend of the two, which is a blurry image that corresponds to neither. This is not a training failure; it is what the MSE objective asks for. Stochastic models address it by learning a distribution, but the distribution’s samples are not calibrated for planning.

The third cost is compounding error, and 2.2.2’s short-rollout result applies with more force. In latent space, a prediction error of 0.02 in normalized units is small and the model can recover. In pixel space, a small pose error becomes a visible displacement, the next step conditions on the displaced image, and the displacement grows. The horizon at which a pixel model’s rollouts are useful is shorter than the latent model’s, for the same dynamics.

Self-explanation 1. Why does an MSE-trained deterministic video model blur rather than pick one future?

Answer. Because the MSE is minimized by the conditional mean, and the mean of a bimodal distribution lies between the modes, which is a blend.

§L.A.3 — The three roles, and which one is being asked for

A video model can play three roles, and confusing them is the source of most deployment disappointment.

Role 1: a learned simulator for planning. The planner queries the model for the consequences of candidate action sequences. This role demands action conditioning, calibration over the planning horizon, and a latency compatible with the planner’s loop. It does not demand visual fidelity; a model that predicts a schematic future accurately is better than one that predicts a beautiful future inaccurately.

Role 2: a representation for policy learning. The model’s internal features, or its decoder’s reconstruction, supply a training signal for a policy. This role demands that the features be task-relevant, which is the same requirement as 3.2.1’s sufficient statistic. Visual fidelity is irrelevant here; a contrastive objective can supply the features without a decoder.

Role 3: a human-in-the-loop preview. The model shows an operator what it expects. This role does demand fidelity, because the operator’s judgment is visual. It does not demand accuracy over long horizons, because the operator is watching the first second.

The three roles have different evaluation criteria, and §L.D separates them. The running case’s question in Part E is which role, if any, the cell needs.

Concept check. A team trains a video model and reports that its samples look indistinguishable from real video. Which role does that result support, and which does it not?

Answer. It supports role 3, the preview, and is consistent with role 2 if the features are probed. It says nothing about role 1, because visual fidelity is measured on the sample distribution and planning requires accuracy on the specific action-conditioned future. [displaces M1]

Worked-example ladder.

WE-L.A.a (fully worked). A video model predicts at $256 \times 256 \times 3$ and is evaluated over a 20-step rollout at 5 Hz. Compute the number of predicted numbers per rollout and the storage at one byte each, and compare with an RSSM with a 64-float latent at the same rate.

Solution. Per frame: $256 \times 256 \times 3 = 196{,}608$. Per rollout: $20 \times 196{,}608 = 3{,}932{,}160$ numbers, 3.9 MB. The RSSM: $20 \times 64 \times 4 = 5{,}120$ bytes. The ratio is 768:1 at the reduced resolution, down from 3,600:1 at native resolution.

WE-L.A.b (completion). The video model is used only to visualize the rollout for an operator, and the planner uses the RSSM. Which parts of the rollout need to be accurate, and which need only to look plausible?

Completion. The first few frames of the visualization need to be plausible but need not be accurate, because the operator is judging the intent, not the outcome. The planner’s latent rollout needs to be accurate over the planning horizon, which is a different requirement on a different object. The completion shows that the two models serve different consumers and are not substitutes.

WE-L.A.c (faded). Derive the conditional-mean argument for blur: given a bimodal future with modes at $x = -1$ and $x = +1$ with equal probability and an MSE loss, compute the optimal prediction and the loss at that prediction.

Faded step. The MSE is minimized at the mean, which is 0, and the loss there is $0.5(1)^2 + 0.5(1)^2 = 1$, versus 0 at either mode. Completion. The optimal prediction has a loss of 1 and corresponds to neither mode; this is why the deterministic model blurs and why the loss value alone does not diagnose it.

WE-L.A.d (contrastive near-miss). A learner trains a deterministic video model on the running case’s data and observes that the predicted cube slowly fades to a smear over ten steps. The learner attributes this to insufficient model capacity and doubles the parameters. The smear persists. Why it fails: the smear is the conditional mean of a multimodal future — the cube’s position is uncertain and the mean is a blend — and capacity does not change the objective. The fix: a stochastic model (Part B) or a latent model that represents the distribution explicitly. The near-miss is the second misconception in operational form.

Comparative table: three model targets.

Design principle P40.1. Decide which of the three roles you are asking for before choosing a model; the roles have different requirements and the same model rarely serves two.

Consolidation checkpoint 1

(a) Three capabilities. You can compute the dimensionality and storage of a pixel rollout at any resolution. You can state the conditional-mean argument for blur. You can name the three roles and their requirements.

(b) Transfer probe. A deployment needs a forward model for a 10 Hz planner and a preview for an operator. Should it use one model or two? Answer: two, because the planner needs a calibrated, action-conditioned latent and the preview needs fidelity; a single model would be over-built for one role and under-built for the other.

(c) Remediation rule. If you cannot derive the blur from the MSE objective, redo WE-L.A.c before Part B.

Part B — The four architecture families

§L.B.1 — Deterministic recurrent video models

The first family predicts the next frame with a recurrent network. ConvLSTM (Shi et al. 2015, tier A, S) replaces the fully connected gates of an LSTM with convolutions, so the hidden state is a spatial feature map and the recurrence preserves spatial structure. Action-conditional video prediction (Finn et al. 2016, tier A, S) adds the action to the recurrence and trains the model to predict the next frame given the current frame and the action.

Finn et al.’s contribution is not architectural but conceptual: they showed that a video model trained on robot pushing data without action conditioning learns to predict the average future, while the action-conditioned model learns to predict the consequence of the action. The action-conditioned model can then be used for control — the planner queries it for the consequence of a candidate action — and the unconditioned model cannot. This is the first lesson of the family: action conditioning is not a feature, it is the requirement.

The architecture’s inference cost is one recurrent step per frame, which is cheap. Its failure mode is the blur of §L.A.2, and it is severe on multimodal futures.

§L.B.2 — Stochastic video models

The second family learns a distribution over futures. SVG-LP (Denton & Fergus 2018, tier A, S) factorizes the prediction into a deterministic next-frame estimate and a stochastic latent that captures what the deterministic part cannot. SAVP (Babaeizadeh et al. 2018, tier A, S) adds an adversarial loss so that samples are sharp rather than blurry. Latent dynamics models (Villegas et al. 2019, tier A, S) predict in a learned latent and decode, which is the bridge to the RSSM of 3.2.1.

The stochastic family solves the blur by sampling, but it introduces a new problem: the samples are not calibrated. A sample from the model is a plausible future, not the expected future, and a planner that averages over samples or trusts a single sample is using a distribution whose coverage has not been measured. The distinction matters because the calibration is exactly what a planner needs and exactly what the training objective does not supply.

The RSSM’s prior–posterior structure is the cleanest solution to this problem, which is why 3.2.1 presented it as the reference latent model: the prior is trained to predict what the posterior will see, which is a calibration objective in disguise.

§L.B.3 — Autoregressive token models

The third family discretizes the video into tokens and predicts them with a transformer. VideoGPT (Yan et al. 2021, tier A, S) trains a VQ-VAE to tokenize frames and an autoregressive transformer over the token sequence. The approach inherits the language-model training recipe and its scaling behaviour, and it produces samples that are sharper than the recurrent family’s.

The cost is inference: autoregressive decoding is sequential over the token sequence, and a frame is hundreds of tokens. At the running case’s resolution that is prohibitive for a control loop; at reduced resolution it is feasible for the preview role. The same sequential-latency floor appears here as it did for action tokenization in 3.1.2 §L.A.2.

§L.B.4 — Diffusion video models

The fourth family generates video by iterative denoising. Video Diffusion Models (Ho et al. 2022, tier A, S) extend image diffusion to the temporal dimension; Stable Video Diffusion (Blattmann et al. 2023, tier B, S) is a large-scale open model; Flexible Diffusion Modeling (Harvey et al. 2022, tier A, S) addresses long-horizon generation by predicting at a coarse temporal scale and refining.

The family’s strength is sample quality and multimodality: a diffusion model represents the distribution and produces diverse, sharp samples. Its weakness for control is inference cost: each sample requires tens to hundreds of denoising steps, and each step is a full network pass. At the running case’s budget, a diffusion video model is a preview tool, not a planner.

The family has produced the most striking recent results in interactive generation. Genie (Bruce et al. 2024, tier B, S) learns a generative environment from unlabelled video with a latent action model that infers the action from frame pairs, and produces a playable environment at a few frames per second. UniSim (Yang et al. 2023, tier B, S) trains a generative simulator on real and synthetic data and uses it for policy training. These are existence proofs that a video model can be action-conditioned at scale; they are not yet a substitute for a physics simulator in a control loop.

§L.B.5 — What the families share

Four properties are common to all four families, and they are the properties a deployment must evaluate.

Action conditioning is required for role 1, and the model must be trained on action–observation pairs, not observations alone. The action space must match the deployment’s.

The observation model is the output. Whatever the internal representation, the output is in observation space, so the objective weights pixels and the blur/calibration trade applies.

The horizon is finite. All four families degrade with rollout length; the degradation’s rate differs, and the evaluation of Part D measures it.

Inference cost dominates deployment. The recurrent family is cheap and blurry; the token and diffusion families are expensive and sharp. The choice is a latency decision before it is an accuracy decision.

Self-explanation 2. Why does action conditioning change the model’s kind rather than only its accuracy?

Answer. Because an unconditioned model averages over actions and learns the marginal future, while a conditioned model learns the conditional; the conditional is what a planner queries, and the marginal cannot answer it.

Concept check. A team trains an unconditioned video model on the running case’s data and finds it predicts the cube’s motion well on average. Can it be used to plan a grasp?

Answer. No. It predicts the average future over all actions taken in the data, which does not distinguish “close the gripper now” from “close it later”; the planner needs the conditional, and the unconditioned model cannot supply it. [displaces M1]

Worked-example ladder.

WE-L.B.a (fully worked). A ConvLSTM predicts a $128 \times 128 \times 3$ frame per step at 10 Hz. Compute the per-step output size and the total for a 30-step rollout, and compare with a diffusion model that runs 50 denoising steps per frame.

Solution. Per frame $128 \times 128 \times 3 = 49{,}152$ numbers. Per rollout $1{,}474{,}560$. The diffusion model’s cost is $50 \times$ the per-frame network passes, so its inference cost is 50 times the ConvLSTM’s for the same output, before counting the larger network.

WE-L.B.b (completion). SVG-LP samples $K = 10$ futures per query. Compute the planner’s cost if it evaluates 100 candidate action sequences with 10 samples each. Completion. $100 \times 10 = 1{,}000$ rollouts, each 30 steps, so $30{,}000$ model steps per planning query. At 1 ms per step that is 30 s per query — far beyond a control loop. The planner must use fewer candidates, fewer samples, or a latent model.

WE-L.B.c (faded). Derive the sequential-latency floor for VideoGPT at 64 tokens per frame and 5 ms per token pass. Faded step. Per frame $64 \times 5 = 320$ ms; a 10-frame rollout is 3.2 s. Completion. The floor is set by the token count, not the network size, and it is the same lesson as 3.1.2’s discrete action decoder.

WE-L.B.d (contrastive near-miss). A learner uses a diffusion video model as the planner’s forward model and finds the planner too slow to run at 10 Hz. The learner reduces the denoising steps from 50 to 5. The samples become blurry and the planner’s decisions degrade. Why it fails: the denoising steps are what sharpen the sample, and reducing them trades quality for speed without changing the architecture. The fix: use a latent model for the planner and reserve the diffusion model for the preview, which is the composition of §L.E.3.

Comparative table: the four families.

Design principle P40.2. Action conditioning is what makes a video model a forward model; without it the model predicts the marginal future, which no planner can use.

Consolidation checkpoint 2

(a) Three capabilities. You can place each family in the cost–fidelity plane. You can state the action-conditioning requirement and its origin. You can compute a planner’s inference cost from candidates, samples, and steps.

(b) Transfer probe. A planner evaluates 50 candidates with one sample each at 10 Hz. Which family fits? Answer: the recurrent deterministic family if the futures are unimodal, the stochastic family with one sample if not; the token and diffusion families do not fit at that budget.

(c) Remediation rule. If you cannot state why an unconditioned model cannot answer a conditional query, re-read §L.B.1 before Part C.

Part C — Objectives and the objective–control gap

§L.C.1 — The four objectives

Each family is trained by one or more of four objectives, and each optimizes something different.

Reconstruction (MSE or L1). Minimizes the per-pixel difference. It is the objective behind the blur of §L.A.2 and the simplest to train. Its gradient is dominated by large, high-contrast regions, which is why the cube’s small edge receives little of the signal.

Perceptual loss. Compares features from a pretrained network rather than raw pixels. It produces sharper images because the feature space weights structure over brightness. The choice of the pretrained network determines what “perceptual” means, and the network is usually trained on natural images, not on robot scenes.

Adversarial loss. A discriminator judges whether a sample is real. It produces the sharpest deterministic-family samples and is what SAVP adds to SVG. Its failure mode is mode collapse and unstable training, and its evaluation is famously difficult.

Diffusion or flow loss. Regresses a noise or velocity field, as in 3.1.2 §L.D.2. It is the most stable high-fidelity objective and the most expensive at inference.

The four objectives trade off against each other, and none of them is the control objective. That gap is the lesson’s central concern.

§L.C.2 — The objective–control gap

The objective–control gap is the difference between what the training objective measures and what the planner needs. Three examples make it concrete.

Example 1: texture dominance. A model trained with MSE predicts the table’s wood grain with high accuracy and the cube’s edge with moderate accuracy, because the grain occupies more pixels. The planner needs the edge. The model is good by its objective and poor for control.

Example 2: depth ambiguity. A pixel prediction is consistent with a range of 3D configurations, by the projection of 2.1.1. A model that predicts the correct pixels can be wrong about the depth, and the grasp depends on depth. This is not a training failure; it is an information-theoretic property of the output space.

Example 3: contact invisibility. The contact between the finger and the cube is a force event with a visual signature of a few pixels at most. The model’s prediction of the contact region is dominated by the surrounding surfaces, and its error there is not what the loss measures. The 2.3.3 momentum observer sees the contact; the video model does not.

The gap is not closed by a better video model. It is closed by adding the task-relevant signal: an auxiliary loss on pose, a contrastive objective on task-relevant features, or a composition with a latent model that carries the task statistic. The 3.2.1 RSSM’s latent is trained with the reward and continuation terms of the ELBO, which is a task-relevant signal; a video model trained only on reconstruction has none.

Self-explanation 3. Why is adding an auxiliary pose loss a legitimate fix to the objective–control gap, while adding more capacity is not?

Answer. Because the auxiliary loss changes what the model is asked to be accurate about, while capacity only lets it be more accurate about what it was already asked.

§L.C.3 — Action conditioning and the counterfactual requirement

A planner does not ask “what happens next?”; it asks “what happens next if I do this?” The query is a counterfactual, and answering it requires the model to have been trained on a distribution that includes variation in the action.

The requirement is stronger than it looks. The model must be accurate on the difference between two nearby actions, not only on each action’s consequence. If the training data’s actions are highly correlated with the state — as they are in a closed-loop dataset, where the policy chooses the action based on the state — the model can learn a shortcut that predicts the future from the state alone and ignores the action. The shortcut is invisible in the training loss because the action was redundant given the state, and it fails at planning time when the planner queries off-policy actions.

Finn et al. 2016’s insight is precisely this: the model must be trained on data where the action is not determined by the state. Their robot pushing data included random actions, which made the action informative. The running case’s 2.2.4 dataset was collected by a closed-loop policy, so the action is correlated with the state, and the shortcut risk is real.

The mitigations are three. Randomize the action during data collection, which the running case did not do. Add off-policy actions by collecting a small dataset with a random or exploratory policy. Use a model that enforces action dependence architecturally, for example by making the action an explicit input to every recurrent step and checking that the model’s output changes when the action does.

The third is a test, not a training method, and it is the diagnostic: perturb the action by a small amount and measure whether the prediction changes by more than the noise floor. A model that ignores the action passes the training loss and fails this test.

Concept check. A video model trained on the running case’s closed-loop dataset has low training error. Perturbing the action leaves its predictions unchanged. Diagnose and propose two fixes.

Answer. The model has learned the shortcut and ignores the action. Fix by collecting off-policy data with randomized actions, or by architecturally forcing the action into every step and retraining with an action-dependence regularizer. [displaces M1]

Worked-example ladder.

WE-L.C.a (fully worked). A model’s prediction changes by 0.1 pixels when the action is perturbed by one standard deviation, and the pixel noise floor is 0.5 pixels. Compute the signal-to-noise ratio and decide whether the model uses the action.

Solution. $\mathrm{SNR} = 0.1/0.5 = 0.2$, below the 1.0 threshold, so the model’s action dependence is indistinguishable from noise. The model is not usable for planning.

WE-L.C.b (completion). After collecting 5,000 off-policy transitions with randomized actions and retraining, the change is 2.0 pixels. Recompute and decide. Completion. $\mathrm{SNR} = 4.0$, well above threshold; the model is action-dependent. The completion shows that the fix is data, not architecture, for this failure.

WE-L.C.c (faded). Derive the auxiliary pose loss that closes the objective–control gap, using the 2.1.2 estimator’s pose as a target. Faded step. Add $\lambda_{pose}|\hat{p}_t - p_t|^2$ to the reconstruction loss, with the pose target from the estimator on the training frames. Completion. The auxiliary weight is chosen so that the pose loss’s gradient norm is comparable to the reconstruction loss’s, following the 3.1.2 gradient-norm rule.

WE-L.C.d (contrastive near-miss). A learner adds an adversarial loss to sharpen the model and finds the planner’s performance degrades, even though the samples look better. Why it fails: the adversarial loss sharpens texture and edges, which are not what the planner uses, and it destabilizes the model’s predictions in low-contrast regions like the contact area. The fix: keep the reconstruction loss for the planner and use the adversarial model for the preview, which is the composition of §L.E.3.

Comparative table: objectives and what they optimize.

Design principle P40.3. The training loss and the planning query are different questions; close the gap with a task-relevant auxiliary signal, not with capacity.

Consolidation checkpoint 3

(a) Three capabilities. You can name the four objectives and their trade-offs. You can state the three examples of the objective–control gap. You can run the action-dependence test and interpret its SNR.

(b) Transfer probe. A model passes the action-dependence test at SNR 5 and fails the planner at contact. What is the remaining gap? Answer: the contact signal is invisible in pixel space; the fix is a composition with the 2.3.3 observer or an auxiliary contact loss, not more video data.

(c) Remediation rule. If you cannot state why closed-loop data creates the shortcut, re-read §L.C.3 before Part D.

Part D — Evaluation

§L.D.1 — FVD and what it measures

The standard video-generation metric is the Fréchet Video Distance (Unterthiner et al. 2018, tier A, S). It compares the distribution of generated videos with the distribution of real videos in the feature space of a pretrained video classifier, usually I3D. Concretely, it fits a multivariate Gaussian to the features of each set and computes

where $r$ and $g$ denote real and generated. Lower is better. The metric is well-defined, reproducible, and the community’s standard.

What it measures is distributional similarity in a feature space trained for action recognition. What it does not measure is: whether a specific rollout is accurate, whether the model is action-conditional, whether the model is calibrated, and whether the model is useful for control. A model can achieve a low FVD while being action-blind, because the feature space’s statistics are dominated by appearance.

For the planner role, FVD is therefore necessary at most and sufficient at never. It should be reported, and it should not be the acceptance criterion.

§L.D.2 — Control-relevant metrics

Three metrics measure what the planner needs, and each is computed on held-out episodes per 3.1.4’s split discipline.

Action-dependence SNR, as in §L.C.3. It is the first gate: a model below threshold is not a forward model.

Rollout accuracy over the planning horizon. For each held-out episode, roll the model forward for $H$ steps under the recorded actions from the recorded start, and compute the prediction error against the recorded observations. Report the error at $H = 1, 5, 15$, because the curve is the compounding-error signature. The threshold is set by the planner’s tolerance, which §L.E.1 computes for the running case.

Decision quality. The ultimate test is whether a policy trained or planned with the model succeeds in the real environment. This is the most expensive metric and the only one that measures the whole system. It should be reported with a confidence interval, using the standard-error formula of 3.1.4 §L.D.1.

The three metrics form a ladder: SNR gates the model, rollout accuracy gates the planner, decision quality gates the deployment. A model that fails SNR is not tested further.

§L.D.3 — Long-horizon drift

The compounding-error curve deserves its own paragraph because it is the metric most often hidden by aggregation. Plot the rollout error against the horizon for both the video model and the RSSM on the same episodes. Three shapes are possible.

Flat. The error does not grow, which is unusual and suggests the task’s dynamics are nearly deterministic and the model has captured them.

Linear. The error grows linearly, which is the expected shape for independent per-step errors.

Super-linear. The error grows faster than linearly, which indicates that the model’s errors feed back into its input and amplify. This is the shape that limits the horizon most severely, and it is the shape that pixel models produce more often than latent models, because the pixel input carries the error into the next step.

The horizon at which the error crosses the planner’s tolerance is the model’s usable horizon, and it must be measured, not assumed. 3.2.1’s horizon of 15–50 is a default; the running case’s number is derived in §L.E.1.

Self-explanation 1. Why does a super-linear error curve indicate feedback rather than accumulation?

Answer. Because independent errors accumulate linearly; super-linear growth requires the error to be an input to the next step, which is exactly what conditioning on the predicted frame does.

Concept check. Model A has FVD 40 and a usable horizon of 3 steps; model B has FVD 200 and a usable horizon of 15 steps. Which is the better forward model for a planner with a 10-step horizon?

Answer. Model B, because FVD measures distributional similarity and the usable horizon measures the planner’s requirement; a beautiful model that is accurate for three steps cannot plan ten. [displaces M4]

Worked-example ladder.

WE-L.D.a (fully worked). Rollout errors at $H = 1, 5, 15$ are 0.01, 0.05, 0.30. Fit a power law $\epsilon(H) = a H^b$ and compute $b$.

Solution. From $H=1$ to $H=15$, the ratio is $0.30/0.01 = 30$ over a factor of 15 in horizon, so $b = \log(30)/\log(15) \approx 3.40/2.71 = 1.26$. The exponent above 1 indicates super-linear growth, consistent with feedback.

WE-L.D.b (completion). The planner’s tolerance is 0.05. Compute the usable horizon from the fit. Completion. $0.05 = 0.01 H^{1.26}$, so $H^{1.26} = 5$ and $H = 5^{1/1.26} \approx 3.7$, so the usable horizon is 3 steps. The completion shows that the tolerance, not the metric, sets the horizon.

WE-L.D.c (faded). Compute the FVD between two sets of 100 clips each with 4-dimensional features and identity covariance. Faded step. With $\Sigma_r = \Sigma_g = I$, the trace term is $\mathrm{Tr}(I + I - 2I) = 0$, and the FVD reduces to $|\mu_r - \mu_g|^2$. Completion. The completion shows that with equal covariances FVD reduces to a mean difference, which is why a model that gets the mean right but the variance wrong can score well.

WE-L.D.d (contrastive near-miss). A team reports FVD 35 on the running case and declares the model ready for planning. The planner fails on the first contact. Why it fails: FVD measured appearance, not the contact region, and the model was never tested on the planner’s horizon or its tolerance. The fix: report the action-dependence SNR and the rollout accuracy curve before the decision, and use FVD only as a sanity check.

Comparative table: evaluation metrics.

Design principle P40.4. Gate on action dependence, evaluate on rollout error versus horizon, accept on decision quality. Never accept on FVD.

Consolidation checkpoint 4

(a) Three capabilities. You can write the FVD definition and state its limits. You can compute a power-law exponent from three error points. You can derive the usable horizon from the tolerance.

(b) Transfer probe. Two models have the same FVD and different power-law exponents, 0.9 and 1.4. Which is preferable for a 10-step planner? Answer: the one with exponent 0.9, because its error grows sub-linearly and its usable horizon is longer for the same tolerance.

(c) Remediation rule. If you cannot state why FVD is not a planner metric, re-read §L.D.1 before Part E.

Part E — The running case

§L.E.1 — Deriving the planner’s tolerance

The running case’s planner (3.2.3 will formalize it) needs to evaluate candidate action sequences for the grasp. The tolerance is set by the 2.3.3 controller’s grasp window: the gripper must be within about 1 cm of the cube’s centre when it closes, and the cube is 5 cm across, so the tolerance on the predicted cube pose is 1 cm.

Translating that into pixel error at the eye-in-hand camera requires the depth. At a working distance of $Z = 0.3$ m and focal length $f = 500$ px, a 1 cm displacement projects to $f \cdot \Delta X / Z = 500 \times 0.01 / 0.3 = 16.7$ px. The planner’s tolerance is therefore a 16-pixel error on the cube’s projected position, and the rollout error metric must be computed on the cube’s region rather than the whole image.

This is the arithmetic that makes the running case’s decision. A model whose whole-image rollout error looks small can still exceed 16 pixels on the cube, and a model whose whole-image error looks large can be accurate on the cube. The metric must be computed on the task-relevant region, which is the objective–control gap of §L.C.2 in evaluation form.

§L.E.2 — The head-to-head

The comparison is between three candidates on the same held-out episodes.

Candidate A: the 3.2.1 RSSM. Latent dimension 64, trained on the 100,000 transitions, with the reaching coverage gap of 3.2.1 §L.E.2 and a proposed 3,000-transition grasping extension. Its usable horizon is computed from its own rollout error curve.

Candidate B: a recurrent deterministic video model at $128 \times 128$. Action-conditioned, trained on the same data. Its inference is cheap and its predictions blur at multimodal futures.

Candidate C: a small stochastic video model (SVG-LP-style) with 5 samples per query.

The head-to-head is decided by three measurements, in the order of the ladder.

SNR. A and B and C are all tested. The 2.2.4 data’s closed-loop collection makes the shortcut risk real, so the SNR test is not a formality. If any candidate fails, it is eliminated before the expensive metrics.

Rollout accuracy on the cube region. Each candidate’s error at $H = 1, 5, 15$, computed on the cube’s bounding box from the 2.1.2 estimator rather than the whole image. This is the number that maps to the 16-pixel tolerance.

Decision quality. A grasp-planner trained in each model, evaluated on the real cell’s held-out episodes. This is the expensive metric and the one that matters.

§L.E.3 — The expected outcome and the composition

The expected outcome, given the curriculum’s prior results, is that A wins on the planner role and B or C wins on the preview role. The RSSM’s latent carries the task statistic and its horizon is longer for the same tolerance; the video model’s output is what a human can inspect. The composition is:

• The RSSM is the planner’s forward model, queried by 3.2.3 for candidate evaluation.
• The video model is the preview, rendered for the operator at a lower rate and at reduced resolution, showing the first few frames of the planned trajectory.
• The 2.1.2 estimator supplies the pose metric on which the RSSM’s rollout error is computed and on which the tolerance is defined.

The composition is the lesson’s answer to the framing question. A video model is not a better or worse forward model than a latent model; it is a different forward model with a different output space, and the running case needs both because it has both a planner and an operator.

Self-explanation 2. Why does the preview role not need the video model to be action-conditional?

Answer. Because the operator is watching the model’s expectation under the policy’s chosen actions, not querying counterfactuals; the conditional and the marginal coincide when the action is the one being executed.

Worked-example ladder.

WE-L.E.a (fully worked). The RSSM’s cube-region error is 3 pixels at $H = 5$ and 12 at $H = 15$, with a power-law exponent of 1.2. The video model’s is 5 at $H = 5$ and 40 at $H = 15$, exponent 1.7. Compute each usable horizon against the 16-pixel tolerance.

Solution. RSSM: $16 = 3 (H/5)^{1.2}$ gives $(H/5)^{1.2} = 5.33$, $H/5 = 4.0$, $H = 20$. Video model: $16 = 5 (H/5)^{1.7}$ gives $(H/5)^{1.7} = 3.2$, $H/5 = 1.96$, $H = 9.8$. The RSSM’s usable horizon is 20 steps, the video model’s is about 10. Both exceed the planner’s 10-step horizon, but the RSSM has twice the margin.

WE-L.E.b (completion). The cube’s grasp tolerance tightens to 5 mm, or 8.3 pixels. Recompute both horizons. Completion. RSSM: $8.3 = 3(H/5)^{1.2}$, $(H/5)^{1.2} = 2.77$, $H/5 = 2.38$, $H = 11.9$. Video: $8.3 = 5(H/5)^{1.7}$, $(H/5)^{1.7} = 1.66$, $H/5 = 1.36$, $H = 6.8$. The video model’s horizon falls below the planner’s 10 steps; the RSSM’s is marginal. The tighter tolerance eliminates the video model from the planner role.

WE-L.E.c (faded). Design the preview’s resolution and rate so that its cost is under 10% of the control loop’s budget. Faded step. At 5 Hz and $128 \times 128$, the preview costs 5 frames per second of a 0.5 MFLOP network, which is a few milliseconds; the budget is met with room to spare. Completion. The preview can run at a lower rate than the planner because the operator does not need frame-rate fidelity; the composition is cheap.

WE-L.E.d (contrastive near-miss). A team uses the video model for both roles to simplify the stack. The planner’s horizon is exactly at the tolerance boundary, and the operator’s preview is at native resolution. The planner occasionally executes a grasp that the preview showed failing, and the operator cannot reconcile the two. Why it fails: the two roles have different tolerances, and the single model is at its limit for the planner and over-built for the preview; the divergence between what the planner used and what the operator saw is a communication failure. The fix: two models, as in the composition, with the planner’s model named explicitly in the operator’s display.

Comparative table: the running case’s candidates.

Design principle P40.5. Compose the forward model from the role it serves; a planner needs calibration on the task region, a preview needs fidelity on the operator’s screen, and the two are different objects.

Consolidation checkpoint 5

(a) Three capabilities. You can derive the planner’s tolerance from the task’s grasp window and the camera’s projection. You can compute usable horizons from a power-law fit. You can state the composition and the role of each component.

(b) Transfer probe. The camera moves to a 0.5 m working distance. How does the tolerance in pixels change, and what does that do to the two horizons? Answer: the projection scale falls by $0.3/0.5 = 0.6$, so the tolerance in pixels falls from 16.7 to 10, and both horizons shorten; the RSSM’s margin over the video model is preserved.

(c) Remediation rule. If you cannot compute the pixel tolerance from the grasp window and the projection, redo WE-L.E.a before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: architectural shortcut, planner latency, and discriminator instability.

The lab

Task. Train two forward models on the same synthetic partially observed dataset — a deterministic action-conditioned recurrent predictor and a stochastic predictor with a sampling head — and evaluate them by FVD-style distributional similarity, action-dependence SNR, and rollout error versus horizon on the task-relevant region.

Conditions. Python 3.10+, PyTorch 2.x, CPU sufficient. The synthetic task is a 2-D point mass observed through a 16×16 image with the mass rendered as a 3-pixel disc. Actions are 2-D forces. Episode length 40, 400 episodes. Seed all RNGs.

Controls. (i) Deterministic predictor; (ii) stochastic predictor with 5 samples; (iii) deterministic predictor trained with the action zeroed in half the batches (a shortcut-inducing control); (iv) evaluation of each on the disc region versus the whole image.

Metrics.

Pre-registered hypotheses.

H1. Control (iii) has a low whole-image error and a low SNR, because the shortcut is invisible in the reconstruction loss.

H2. The disc-region error is larger than the whole-image error for every control, because the disc occupies a small fraction of the pixels and the loss weights the rest.

H3. The stochastic predictor’s error exponent is smaller than the deterministic predictor’s, because sampling reduces the mean-blur feedback.

H4. The distributional metric does not separate controls (i) and (iii), because both produce plausible-looking futures.

Break-it sequence.

1. Zero the action in half the batches silently. The SNR collapses while the whole-image loss stays low. Silent: the loss curve looks normal.
2. Evaluate on the whole image only silently. The disc-region failure is hidden. Silent: the metric is plausible.
3. Use a deterministic model for a multimodal future silently. The rollout blurs and the planner’s decisions degrade. Silent: the loss is small because the mean is the optimum.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The SNR is reported before any other metric, because it gates. (2) The disc-region and whole-image errors are reported side by side, so the gap is visible. (3) The error-versus-horizon plot has the fitted exponent annotated. (4) The distributional metric is reported with the caveat that it cannot separate the shortcut control. (5) The break-it results are a table.

A runnable core, seeded and annotated:

# lab_3_2_2.py — video predictor, SNR test, region error
import numpy as np, torch, torch.nn as nn
torch.manual_seed(0); np.random.seed(0)

H = W = 16                                                 # image size
$D_ACT$ = 2

def rollout_episode():                                     # synthetic episode
    p = np.random.randn(2) * 0.3
    v = np.zeros(2)
    frames, acts = [], []
    for t in range(40):
        img = np.zeros((3, H, W), dtype=np.float32)
        ix, iy = int(p[0] * 10 + H / 2), int(p[1] * 10 + W / 2)
        if 0 <= ix < H and 0 <= iy < W:                    # render 3-pixel disc
            img[0, ix, iy] = 1.0
            img[0, max(ix-1,0), iy] = 1.0
            img[0, min(ix+1,H-1), iy] = 1.0
        a = np.random.randn(2) * 0.5                       # random action
        v = 0.9 * v + 0.1 * a
        p = p + v * 0.1
        frames.append(img); acts.append(a)
    return (torch.tensor(np.stack(frames)),
            torch.tensor(np.stack(acts), dtype=torch.float32))

class VideoModel(nn.Module):
    def __init__(self, hidden=128):
        super().__init__()
        self.enc = nn.Sequential(nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(),
                                 nn.Conv2d(16, 32, 3, padding=1), nn.ReLU())
        self.gru = nn.GRUCell(32 * H * W // 16 + $D_ACT$, hidden)
        self.out = nn.Linear(hidden, 3 * H * W)
    def forward(self, frame, act, h):
        f = self.enc(frame)
        h = self.gru(torch.cat([f.flatten(1), act], dim=1), h)
        return self.out(h).view(-1, 3, H, W), h

data = [rollout_episode() for _ in range(400)]
model = VideoModel(); opt = torch.optim.Adam(model.parameters(), lr=1e-3)

for step in range(2000):                                   # training
    frames, acts = data[np.random.randint(len(data))]
    h = torch.zeros(1, 128); loss = 0.0
    for t in range(39):
        pred, h = model(frames[t:t+1], acts[t:t+1], h)
        loss = loss + nn.functional.mse_loss(pred, frames[t+1:t+2])
    opt.zero_grad(); loss.backward(); opt.step()

def snr_test(model, n=20):                                 # §L.C.3 diagnostic
    diffs, noises = [], []
    for _ in range(n):
        frames, acts = data[np.random.randint(len(data))]
        h = torch.zeros(1, 128)
        for t in range(10):
            _, h = model(frames[t:t+1], acts[t:t+1], h)
        base, _ = model(frames[10:11], acts[10:11], h.clone())
        pert, _ = model(frames[10:11], acts[10:11] + 0.5, h.clone())
        diffs.append((base - pert).abs().mean().item())
    return np.mean(diffs) / 0.05                           # noise floor 0.05

print("action-dependence SNR:", round(snr_test(model), 2))

# Region error vs whole-image error over horizons
def region_error(model, horizon=5):
    disc, whole = [], []
    for _ in range(20):
        frames, acts = data[np.random.randint(len(data))]
        h = torch.zeros(1, 128); preds = []
        for t in range(horizon):
            p, h = model(frames[t:t+1], acts[t:t+1], h)
            preds.append((p, frames[t+1:t+2]))
        for p, gt in preds:
            disc.append(((p - gt) ** 2)[:, :, 5:11, 5:11].mean().item())
            whole.append(((p - gt) ** 2).mean().item())
    return np.mean(disc), np.mean(whole)

d, w = region_error(model)
print(f"disc-region MSE {d:.4f}  whole-image MSE {w:.4f}")

Line 9 renders the mass as a 3-pixel disc, which is the small, task-relevant region the whole-image loss under-weights. Lines 22–30 are the action-conditioned model: the encoder produces features, the GRU consumes them with the action, the output layer reconstructs the frame. Lines 32–39 train with reconstruction. Lines 41–51 are the SNR test: the model’s prediction is compared with the same model’s prediction under a perturbed action, divided by the noise floor. Lines 53–65 compute the disc-region and whole-image errors separately, which is the evaluation the lesson argues for.

Cheat sheet

Exercises with solutions

E 3.2.2.1. Compute the number of predicted numbers in a 25-step rollout at $320 \times 240 \times 3$ and the storage at one byte each.

S 3.2.2.1. Per frame $320 \times 240 \times 3 = 230{,}400$; per rollout $25 \times 230{,}400 = 5{,}760{,}000$ numbers, 5.76 MB. The lesson: the pixel space is large even at reduced resolution, and the planner’s cost is dominated by it.

E 3.2.2.2 (transfer). A future is bimodal at $x = 0$ and $x = 2$ with equal probability. Compute the MSE-optimal prediction and the loss there.

S 3.2.2.2. The mean is 1; the loss is $0.5(1)^2 + 0.5(1)^2 = 1$. Either mode has loss 0. The lesson: the MSE optimum is between the modes and is not a valid future; this is the blur.

E 3.2.2.3. State why action conditioning is required for the planner role and what Finn et al. 2016 showed.

S 3.2.2.3. The planner queries counterfactuals, which require the conditional $p(o'|o,a)$; an unconditioned model learns the marginal, which cannot distinguish actions. Finn et al. showed that an action-conditioned model trained on robot pushing learns the consequence of the action and can be used for control, while the unconditioned model cannot. The lesson: conditioning changes the model’s kind, not its accuracy.

E 3.2.2.4 (diagnostic). A model has low training loss and an SNR of 0.3. Diagnose and propose two fixes.

S 3.2.2.4. The model has learned the shortcut from closed-loop data and ignores the action. Fixes: collect off-policy data with randomized actions; architecturally force the action into every step and retrain with an action-dependence regularizer. The lesson: the SNR is the gate that reconstruction loss cannot provide.

E 3.2.2.5. Compute the FVD between two 4-dimensional Gaussian feature sets with means $(0,0,0,0)$ and $(1,0,0,0)$ and identity covariances.

S 3.2.2.5. The trace term is $\mathrm{Tr}(I + I - 2I) = 0$ and the mean term is 1, so FVD = 1. The lesson: with equal covariances FVD reduces to a mean difference and says nothing about the model’s use of the action.

E 3.2.2.6 (transfer). Rollout errors at $H = 1, 5, 20$ are 0.02, 0.08, 0.64. Compute the power-law exponent.

S 3.2.2.6. From $H=1$ to $H=20$, the ratio is 32 over a factor of 20, so $b = \log(32)/\log(20) = 3.466/2.996 = 1.16$. The lesson: an exponent above 1 is the super-linear signature of feedback, and it sets the usable horizon more than the absolute error does.

E 3.2.2.7. A camera has $f = 500$ px and working distance 0.4 m. Compute the pixel displacement of a 1 cm object displacement.

S 3.2.2.7. $f \Delta X / Z = 500 \times 0.01 / 0.4 = 12.5$ px. The lesson: the tolerance in pixels is a projection of the task’s tolerance in metres, and it moves with the working distance.

E 3.2.2.8 (diagnostic). A model has FVD 30 and a usable horizon of 4 steps; another has FVD 180 and 18 steps. Which is the better planner model, and what does the comparison show?

S 3.2.2.8. The second, because the planner needs horizon, not distributional similarity. The comparison shows that FVD and planning utility are not monotonically related. The lesson: the acceptance metric must match the role.

E 3.2.2.9 (transfer). Design the composition for a cell that needs both a planner and an operator preview, naming the model for each role and the metric that gates it.

S 3.2.2.9. RSSM for the planner, gated by cube-region rollout error against the pixel tolerance; a video model at reduced resolution for the preview, gated by subjective plausibility and by the operator’s judgment; the 2.1.2 estimator supplies the pose metric for both. The lesson: the composition is determined by the roles, not by a single ranking of models.

Exit test

You are ready for Lesson 3.2.3 when you can, without notes:

1. State the three roles a video model can play and their requirements.
2. Compute the dimensionality and storage of a pixel rollout.
3. Derive the MSE optimum under a bimodal future and explain the blur.
4. State the action-conditioning requirement and its origin in Finn et al. 2016.
5. Place the four architecture families in the cost–fidelity plane.
6. Name the four objectives and their trade-offs.
7. Give three examples of the objective–control gap.
8. Run the action-dependence SNR test and interpret its value.
9. Write the FVD definition and state what it does not measure.
10. Compute a power-law error exponent from three points.
11. Derive a usable horizon from a tolerance and a fitted curve.
12. Compute a pixel tolerance from a task tolerance and the camera’s projection.
13. Design the control-relevant evaluation ladder: SNR, rollout, decision.
14. State the running case’s head-to-head and its expected outcome.
15. Design the composition of a latent planner model and a video preview.

Cross-references. 3.2.1 supplies the RSSM, the coverage-gap ratio, and the horizon discipline restated in §L.A.1 and §L.E.2. 2.1.1 supplies the projection model used to convert the task tolerance to pixels in §L.E.1. 2.2.2 supplies the short-rollout result restated in §L.A.2. 3.1.4 supplies the held-out evaluation discipline restated in §L.D.2. Lesson 3.2.3 uses the forward-model selection of §L.E.3 as the planner’s model. Module 3.4 contrasts the learned video simulator with a physics simulator for sim-to-real.

Ready. The next lesson to write is 3.2.3 — Planning with World Models. Say “next” to generate it.