Lesson 3.1.2 — VLA Architectures: RT-1, RT-2, OpenVLA, and the π-family

Week 17

Prerequisite lessons: 3.1.1 (introduction to VLAs), 2.3.3 (contact-rich manipulation), 2.3.4 (in-hand and dexterous manipulation), 2.2.4 (offline-to-online fine-tuning), 0.1.3 (nn.Module, optim, DataLoader), 2.1.2 (object detection and 6D pose estimation).

Prior results used in this lesson. From 3.1.1: the VLA interface $\pi_\theta:\mathcal{O}\times\mathcal{L}\to\mathcal{A}$, the three grounding problems, the four eras, and the resolution–range trade of action tokenization. From 2.3.3: the 3 ms contact-response requirement that sets the latency floor. From 2.3.4: teacher–student distillation as the privileged-information pattern. From 2.2.4: the seven-phase deployment pipeline with the runtime Q-shield. Each is restated at the point of use.

What this lesson hands forward. The block-level anatomy of a VLA, the parameter/token/latency accounting that lets you size a deployment, the three action-decoder families (discrete tokens, continuous regression, flow matching), and the adaptation ladder from full fine-tuning to LoRA to frozen-backbone heads. Lesson 3.1.3 uses the anatomy to decide what must be annotated; Lesson 3.1.4 uses the adaptation ladder to define what fine-tuning is allowed to change.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, and divider are unchanged; this lesson asks which of four published architectures you would put behind the instruction channel for that cell, and what each choice costs in parameters, tokens, and milliseconds.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) decompose any VLA into observation encoder, language encoder, fusion block, and action decoder, and say what each published system chooses; (2) compute the token budget of an image history and the resulting attention cost; (3) explain RT-1’s token-learner bottleneck and why it was necessary at 3 Hz; (4) state precisely what RT-2’s co-fine-tuning adds and what the web-data ablation showed; (5) compute the discretization resolution of an action tokenizer and state the resolution–range trade in numbers; (6) describe OpenVLA’s visual encoder fusion and its LoRA adaptation budget; (7) derive the flow-matching training objective and the Euler integration used at inference; (8) compare action-chunk lengths by their open-loop blind window; (9) state the knowledge-insulation principle and the failure it prevents; (10) select an architecture for the running case from a stated cost model and defend the choice with latency, accuracy, and data arguments.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 3.1.1, the VLA interface is $\pi_\theta:\mathcal{O}\times\mathcal{L}\to\mathcal{A}$. Which of the three spaces carries the task identity, and what happens to the other two when the task changes?

Answer. The language space carries task identity; the observation and action spaces stay fixed in shape but their distributions shift with the task. Used in §L.A.1 to argue that only the language path must be task-general.

Q2. In 3.1.1, the resolution–range trade is resolution $=$ range divided by bin count. With 256 bins over 4 cm, what is the resolution, and what does doubling the bin count buy?

Answer. $4/256 = 0.0156$ cm, or 0.156 mm. Doubling the bins halves the resolution but doubles the action-token count, which raises the decoder’s sequence length and latency. Used in §L.B.1 and §L.C.2.

Q3. In 2.3.3, the contact-detection requirement is 3 ms and the impedance controller runs at 1 kHz. Why does that number matter for a VLA whose inference is 100–200 ms?

Answer. Because the VLA cannot be in the contact loop; it must be composed with a fast controller that handles force and slip. Used in §L.E.2 to justify the composition architecture.

Q4. In 2.3.4, the student is trained on the teacher’s state distribution rather than a uniform one. For a VLA trained on teleoperation, what plays the role of the teacher’s state distribution?

Answer. The distribution of states that the teleoperators actually visited under their own closed-loop behaviour, which is the dataset’s state distribution. Used in §L.B.3 to explain why co-fine-tuning on web data cannot fix a state-distribution gap.

Part A — The four-block skeleton

§L.A.1 — Anatomy of every VLA

Every published VLA, regardless of size, is a composition of four blocks. Naming them explicitly is what makes the four architectures comparable.

Block 1: the observation encoder. It converts images (and sometimes depth and proprioception) into a sequence of embedding vectors. RT-1 uses an EfficientNet-B3 (Tan & Le 2019, tier A, S) followed by a TokenLearner (Ryoo et al. 2021, tier A, S). RT-2 uses the vision encoder of a pretrained vision-language model, PaLI-X or PaLM-E. OpenVLA uses a fused pair of encoders, DINOv2 (Oquab et al. 2023, tier A, S) and SigLIP (Zhai et al. 2023, tier A, S). π0 uses the PaliGemma vision tower.

Block 2: the language encoder. It converts the instruction into embeddings. RT-1 uses a USE (Universal Sentence Encoder) embedding and injects it by FiLM (Perez et al. 2018, tier A, S). RT-2 and OpenVLA use the language model’s own tokenizer and embedding table, so the instruction is simply part of the input sequence. π0 uses PaliGemma’s language path.

Block 3: the fusion block. It is where image and language embeddings interact. There are two families. Late fusion keeps the language as a conditioning vector applied to the vision features, as in RT-1’s FiLM layers: the language does not attend to the image; it modulates the vision features elementwise. Early fusion concatenates image and language tokens into one sequence and lets self-attention mix them, as in RT-2 and OpenVLA. The distinction matters: late fusion cannot resolve compositional references that require attending to a specific image region, because the language never sees the image.

Block 4: the action decoder. It converts the fused representation into actions. Three families: discrete tokens (RT-1, RT-2, OpenVLA), continuous regression or diffusion (Octo; Ghosh et al. 2024, tier A, S; Diffusion Policy; Chi et al. 2023, tier A, S), and flow matching over continuous action chunks (π0; Black et al. 2024, tier B, S).

The four blocks map onto the three grounding problems of 3.1.1 as follows. Visual grounding is solved by blocks 1 and 3. Linguistic grounding is solved by block 2 and, for compositional instructions, by block 3. Motor grounding is solved by block 4. This is not decoration: it tells you where to look when a deployment fails.

Self-explanation 1. Why does late fusion fail on “the cube on the left of the bin” while early fusion can succeed?

Answer. Because resolving “on the left of” requires comparing two image regions, which requires the language representation to attend over the image; a conditioning vector applied elementwise cannot make a spatial comparison.

Concept check. A VLA follows “pick up the cube” perfectly but fails on “pick up the cube that is not in the bin”. Which block is the most likely locus, and which test isolates it?

Answer. Block 3, fusion, because negation plus spatial reference requires attending over the image with the language. The isolating test is to present the same instruction with only one cube in the scene: if the failure disappears, the deficit is compositional fusion rather than object recognition. [displaces M2]

§L.A.2 — The three budgets: parameters, tokens, latency

Sizing a VLA deployment means computing three budgets, and each is a small piece of arithmetic.

Parameter budget. The total is the sum over blocks. The vision encoder dominates at small scales and the language backbone dominates at large scales. The table below gives the published figures; treat the numbers as the order of magnitude and the ratios as the durable fact.

Token budget. For an image history of $H$ frames, each encoded to $T$ tokens, plus $L$ instruction tokens, the transformer’s self-attention cost per layer scales as

where $d$ is the model width. The quadratic term in $H \cdot T$ is why RT-1 inserted a TokenLearner: reducing $T$ from 81 to 8 per image cuts the attention cost by roughly a factor of ten at $H=6$.

Latency budget. Inference latency has three additive parts: encoder time, autoregressive decode time, and communication. For a discretized action of $n_a$ dimensions, the decode is $n_a$ sequential steps, each a full forward pass unless the implementation caches the key/value state. A flow head replaces the $n_a$ sequential steps with a fixed number $K$ of integration steps over the whole action vector simultaneously, which changes the latency profile rather than simply reducing it.

Self-explanation 2. Why does action tokenization create a sequential latency floor that a regression head does not have?

Answer. Because autoregressive decoding produces one token per forward pass, so seven action dimensions need seven passes; a regression head produces all dimensions in one pass.

Worked-example ladder.

WE-L.A.a (fully worked). An image history of 6 frames, each encoded to 81 tokens, plus a 16-token instruction, with model width $d = 512$. Compute the attention cost exponent and compare it with the same system using 8 tokens per image.

Solution. With 81 tokens: sequence length is $6 \cdot 81 + 16 = 502$. Quadratic term $502^2 = 252{,}004$; times $d = 512$ gives $1.29 \times 10^8$ in arbitrary units per layer. With 8 tokens: sequence length is $48 + 16 = 64$; $64^2 = 4096$; times 512 gives $2.10 \times 10^6$. The ratio is $61\times$, not the $10\times$ that the token ratio suggests, because the instruction tokens are a fixed offset that matters much more at short sequence lengths.

WE-L.A.b (completion). With a 12-token instruction and 8 tokens per image, how many frames can be added before the sequence length doubles relative to 6 frames?

Completion. At 6 frames the length is $48 + 12 = 60$. Doubling to 120 requires $8H + 12 = 120$, so $H = 13.5$, i.e., 13 frames. The completion shows that at short token counts, history length is cheap; the cost is dominated by the fixed instruction offset until the history grows.

WE-L.A.c (faded). A discrete decoder emits 7 action dimensions at 5 Hz. Estimate its latency floor if a single forward pass takes 25 ms, and compare with a flow head that runs $K=10$ integration steps sharing one forward pass per step.

Faded step. Discrete: $7 \times 25 = 175$ ms plus key/value caching overhead. Flow: $10 \times 25 = 250$ ms if each step is a full pass, or less if the vision encoding is shared across steps. Completion. The flow head is not automatically faster; its advantage is that it produces the whole action vector per step and can output a chunk of 50 actions, so the per-action cost is far lower.

WE-L.A.d (contrastive near-miss). An engineer replaces RT-1’s TokenLearner with raw patch tokens to “give the model more information”, keeping the same backbone and training budget. The model trains but the inference rate falls from 3 Hz to 0.6 Hz and the success rate does not improve. Why it fails: the extra tokens are redundant at the model’s capacity and the attention cost rises quadratically; the information was not the bottleneck. The fix: keep the bottleneck, or increase the model width and data together. The near-miss shows that the token budget is a capacity decision, not only a compute decision.

Comparative table: the four architectures at a glance.

The table’s role is selection, not ranking: the systems target different points on the accuracy, latency, and data-cost trade-off, and Part E turns the table into a decision.

Design principle P36.1. Decompose a VLA into its four blocks before comparing it to another; comparisons at the level of “the model” hide the choices that determine deployment behaviour.

Consolidation checkpoint 1

(a) Three capabilities. You can name the four blocks and place each published system’s choice in them. You can compute attention cost from history, tokens per image, and instruction length. You can state why tokenization creates a sequential latency floor.

(b) Transfer probe. A deployment needs a 500 ms end-to-end latency ceiling and an 8-frame history at 64 tokens per frame. Does a discrete decoder with 7 dimensions fit? Answer: the sequence length is $8 \cdot 64 + 16 = 528$; the decode alone at 25 ms per pass is 175 ms, leaving 325 ms for encoding and transport, so it fits only if the encoder is fast and local.

(c) Remediation rule. If you cannot compute the attention-cost ratio of WE-L.A.a, re-derive Eq. (A.1) before Part B.

Part B — RT-1 and RT-2

§L.B.1 — RT-1: the architecture that made data scale work

RT-1 (Brohan et al. 2022, tier A, S) is the system that demonstrated a single transformer could follow hundreds of instructions from a real robot. The architecture is deliberately small, and every size choice is a latency choice.

The observation encoder is an EfficientNet-B3 applied to each of six history frames. Its output is 81 spatial tokens per frame at the final feature map. Those 486 tokens are reduced by a TokenLearner to 8 tokens per frame, 48 in total. The instruction is embedded with the Universal Sentence Encoder to a 512-dimensional vector and injected into the transformer through FiLM: at each layer, the language embedding produces per-channel scale and shift parameters that modulate the vision features.

The action decoder is discrete: an 11-dimensional action vector (6 arm, 2 base, 3 mode) is quantized into 256 bins per dimension and emitted autoregressively. The control rate is about 3 Hz.

Two design choices deserve attention because they recur in every later system.

The token bottleneck. The TokenLearner is not a convenience; at 3 Hz with six frames, the raw token count would have made the attention cost dominate the latency budget. The bottleneck forces the model to compress the history into a small set of learned spatial summaries, which is a form of attention pooling. The cost is spatial resolution; the benefit is the control rate.

The action quantization. Discretizing 11 dimensions into 256 bins each is a classification problem with 11 heads. The output is not a distribution over trajectories; it is 11 independent argmaxes. This means RT-1’s actions are not multimodal — it cannot represent two valid ways to grasp a cube — but it is also why the model is stable to train: cross-entropy with 256 classes is well-conditioned.

Self-explanation 3. Why does independent per-dimension discretization destroy multimodality?

Answer. Because the argmax per dimension picks the mode of each marginal independently, which can produce a combination that no demonstrated trajectory contained; the joint distribution is not represented.

§L.B.2 — RT-2: actions as a language

RT-2 (Brohan et al. 2023, tier A, S) makes one structural change with large consequences: the action is expressed as text tokens in the model’s vocabulary, and the policy is trained as a next-token predictor over the concatenation of image tokens, instruction tokens, and action tokens.

Concretely, each action dimension’s 256 bins is mapped to 256 tokens drawn from a reserved region of the vocabulary. The action is then a short string of these tokens. Training uses the standard cross-entropy loss over the whole sequence, and the model is initialized from a pretrained vision-language model — PaLI-X (Chen et al. 2023, tier A, S) at 55B or PaLM-E (Driess et al. 2023, tier A, S) at 12B.

The change from RT-1 is not only scale. It is that the loss function is the same one the model was pretrained with. That is what makes co-fine-tuning possible: the model is trained jointly on robot trajectories (where the “text” is the action string) and on web-scale vision-language data (where the text is ordinary language). The web data keeps the semantic prior alive; the robot data teaches the model to emit action strings.

The reported consequence is emergent semantic generalization: RT-2 can follow instructions involving objects and concepts it never saw in robot data, because the vision-language pretraining already encoded them. Brohan et al. 2023 report that removing the web co-fine-tuning data degrades performance on the generalization benchmarks while leaving seen-task accuracy largely intact (tier A, S). That ablation is the single most important empirical fact in this lesson, and it is misread often. It does not say web data makes the robot better at the tasks it was trained on. It says web data makes the robot better at the tasks it was not trained on.

§L.B.3 — What co-fine-tuning buys and what it does not

The ablation’s pattern is worth stating as a rule, because it determines how you budget a fine-tuning run.

Co-fine-tuning buys semantic breadth. The model can map a new noun to a visual referent it has never manipulated, because the noun and the referent already share a region of the pretrained embedding space. This is the capability that makes a VLA different from a multi-task specialist.

Co-fine-tuning does not buy motor breadth. The action distribution is entirely determined by the robot data. If the demonstrations never include a 45°-rotated grasp, the model’s action tokens for that configuration are untrained, and the web data does not help. This is the state-distribution gap of 2.3.4 restated: the teacher’s distribution is the teleoperators’ distribution, and no amount of web text changes it.

Co-fine-tuning can hurt seen-task accuracy if the loss is mis-weighted. The combined loss is

and the ratio $\lambda_{robot}/\lambda_{web}$ is a real hyperparameter. Brohan et al. 2023 report that the mixing ratio matters, and the practical guidance is to weight so that the robot loss’s gradient norm is comparable to the web loss’s, not so that the sample counts are equal (tier A, S).

Concept check. A team co-fine-tunes on 1M web image-text pairs and 50k robot episodes, with equal sample weighting. The model’s seen-task accuracy drops. Name the mechanism and the fix.

Answer. The web gradient dominates because there are 20 times more web samples; the model drifts toward the pretraining distribution and away from the action tokens. The fix is to weight by gradient norm or to oversample the robot data. [displaces M3]

Worked-example ladder.

WE-L.B.a (fully worked). RT-2 emits a 7-dimensional action, each quantized into 256 bins. Compute the length of the action string, the number of possible strings, and the fraction that correspond to the demonstrated action distribution if the demonstrations cover 10,000 distinct actions.

Solution. The string has 7 tokens. The number of possible strings is $256^7 \approx 7.2 \times 10^{16}$. The demonstrated set of 10,000 actions is a fraction $1.4 \times 10^{-13}$ of the space. The lesson: the token space is enormous and almost entirely off-distribution, which is why a Q-shield (or any out-of-distribution rejection) is not optional.

WE-L.B.b (completion). Compute the same for 64 bins per dimension and argue which is preferable given a 4 cm range.

Completion. $64^7 \approx 4.4 \times 10^{12}$, a 16,000-fold reduction in the space. Resolution becomes $4/64 = 0.0625$ cm $= 0.625$ mm, still adequate for the running case’s 2 cm placement tolerance. Fewer bins reduce off-distribution surface area at the cost of resolution; the choice should be made against the task’s precision requirement, not by default.

WE-L.B.c (faded). Write the gradient-norm balancing rule for Eq. (B.1): choose $\lambda_{robot}$ so that $|\lambda_{robot} \nabla L_{robot}| \approx |\lambda_{web}\nabla L_{web}|$ on a pilot batch. Faded step: compute both gradient norms on a small batch, set $\lambda_{robot} = \lambda_{web}|\nabla L_{web}|/|\nabla L_{robot}|$, and re-check periodically because the norms change during training. Completion. The rule is a running estimate, not a constant, because the robot loss falls faster than the web loss early in training.

WE-L.B.d (contrastive near-miss). A learner freezes the vision-language backbone and trains only the action head of RT-2 on robot data. The model performs well on seen tasks and fails on “move the cube next to the green object”, an instruction whose words are all in the pretraining vocabulary. Why it fails: freezing prevents the fusion from learning to attend the action tokens to the new referent; the head can only reweight the frozen features, and the grounding path is not in the head. The fix: unfreeze the top transformer layers, or use LoRA across the fusion block (§L.C.2). The near-miss is the first misunderstanding of what fine-tuning actually adapts.

Comparative table: RT-1 versus RT-2.

Design principle P36.2. Co-fine-tuning buys breadth, not precision. Budget web data for generalization benchmarks and robot data for motor competence, and never expect one to substitute for the other.

Consolidation checkpoint 2

(a) Three capabilities. You can state RT-1’s four blocks and its token budget. You can state RT-2’s structural change and what the web-data ablation showed. You can set a co-fine-tuning loss weight by gradient norm.

(b) Transfer probe. A deployment needs unseen-object generalization but must run on a 10 W onboard computer. Which of RT-1 or RT-2, and what is the compromise? Answer: neither fits as published; the compromise is a small model with early fusion, or an RT-1-scale model distilled from RT-2, or server-side inference with a local safety controller.

(c) Remediation rule. If you cannot state what the web-data ablation did and did not show, re-read §L.B.2 before Part C.

Part C — OpenVLA

§L.C.1 — The open recipe

OpenVLA (Kim et al. 2024, tier A, S) is a reproduction of the RT-2 recipe on open weights, trained on 970k robot episodes aggregated from Open X-Embodiment (O’Connell et al. 2023, tier A, S). Its value in this curriculum is that every component is inspectable, so it is the architecture you should reason with.

The blocks. Observation encoder: a fused pair of DINOv2 (self-supervised, strong on spatial structure and object boundaries) and SigLIP (contrastive, strong on semantic alignment) patch features. The two are concatenated along the channel dimension and projected to the language model’s width. The fusion is a deliberate hedge: DINOv2 supplies the geometric precision that a purely contrastive encoder lacks, and SigLIP supplies the semantic grounding that a purely self-supervised encoder lacks.

Language encoder and fusion: Llama-2 (Touvron et al. 2023, tier A, S) tokenizer and embedding, with image and language tokens concatenated into one sequence and processed by the transformer. Early fusion, as in RT-2.

Action decoder: 7-dimensional discretized actions, 256 bins, autoregressive. Same as RT-2.

The reported result is that OpenVLA at 7B outperforms RT-2-X, a 55B model, by a substantial margin on a broad manipulation benchmark (Kim et al. 2024, tier A, S). The lesson is not that size does not matter; it is that data curation and architecture choices can dominate a factor of eight in parameters at fixed robot-data scale. The 55B model’s advantage is concentrated in language-heavy generalization, where OpenVLA’s Llama-2 backbone is weaker.

§L.C.2 — Parameter-efficient adaptation: the LoRA budget

Fine-tuning a 7B model on a single arm’s data is a recipe for catastrophic forgetting: the 100k transitions of the running case’s 2.2.4 dataset would overwrite far more than they teach. LoRA (Hu et al. 2021, tier A, S) addresses this by constraining the weight update to a low-rank subspace:

with $r \ll \min(d,k)$. The pretrained $W$ is frozen; only $A$ and $B$ are trained. The parameter count per adapted matrix is $r(d+k)$, and the total is that times the number of adapted matrices.

The budget computation is the practical skill. For a transformer with $L$ layers, each with four attention projections of shape $d \times d$ and two feedforward matrices of shapes $d \times 4d$ and $4d \times d$, adapting only the attention projections gives

For Llama-2-7B, $L=32$ and $d=4096$. At $r=32$, $N_{LoRA} = 32 \cdot 4 \cdot 32 \cdot 8192 \approx 3.36 \times 10^7$, about 0.5% of the 6.7B backbone. Memory during training falls accordingly because the frozen weights need no gradients or optimizer state.

The limitation follows from the rank. LoRA adapts style and calibration — the mapping from features to outputs within the subspace the pretrained model already spans — well. It adapts new motor skills poorly, because a genuinely new behaviour requires changing the feature representation, not only the readout. The practical rule is: LoRA for new objects, new instructions, and new calibration; full fine-tuning (or a new action head trained from scratch) for new embodiments and new control modalities.

Self-explanation 1. Why does LoRA’s rank bound the kind of change it can make, not just the amount?

Answer. Because the update is confined to the span of $r$ directions in weight space; a change that requires a direction outside that span cannot be represented at any magnitude.

§L.C.3 — What reproduces and what does not

Three findings from the open-reproduction literature are durable.

Data curation dominates scale. Open X-Embodiment’s aggregation and filtering mattered more than any single architectural change (O’Connell et al. 2023, tier A, S). The same architecture trained on unfiltered data underperforms.

Evaluation is the weak link. General manipulation benchmarks are heterogeneous, and a model’s ranking changes with the benchmark’s task mix. Kim et al. 2024 report a large margin over RT-2-X on their benchmark; that margin should be read as “on this task distribution”, not as a universal ordering (tier A, S).

Smaller models are competitive on narrow distributions. Octo (Ghosh et al. 2024, tier A, S) at 27M and 93M parameters is competitive with much larger generalists when fine-tuned to a single embodiment. This is the specialist argument of 3.1.1 reappearing inside the VLA family.

Worked-example ladder.

WE-L.C.a (fully worked). Compute the LoRA parameter count for OpenVLA’s Llama-2-7B backbone with $L=32$, $d=4096$, $r=16$, adapting only the four attention projections.

Solution. $N = 32 \cdot 4 \cdot 16 \cdot 2 \cdot 4096 = 32 \cdot 4 \cdot 16 \cdot 8192 = 1.68 \times 10^7$, about 0.25% of 6.7B. Training memory for the adapters is negligible; the frozen weights still occupy their inference footprint.

WE-L.C.b (completion). Raise $r$ to 128 and add the feedforward matrices. Compute the new count and the fraction of the backbone.

Completion. Attention only: $32 \cdot 4 \cdot 128 \cdot 8192 = 1.34 \times 10^8$. Adding the two feedforward matrices per layer, each of rank $r$ with shapes $d \times 4d$ and $4d \times d$, adds $32 \cdot 2 \cdot 128 \cdot (4096 + 16384) = 1.68 \times 10^8$. Total about $3.0 \times 10^8$, roughly 4.5% of the backbone. At this rank the adapter is no longer “parameter-efficient” in the sense that motivated LoRA, and full fine-tuning becomes competitive.

WE-L.C.c (faded). Derive the rule for choosing $r$ from the required change: measure the rank of the difference between the fine-tuned and pretrained weight matrices on a small held-out task, and set $r$ above that rank. Faded step: compute $W_{ft} - W_{pre}$ for a fully fine-tuned model on a proxy task, take its singular values, and choose $r$ at the knee of the spectrum. Completion. In practice the spectrum is heavy-tailed and the knee is not sharp; the operational rule is to sweep $r \in {8, 16, 32, 64}$ and pick the smallest that matches full fine-tuning on the target task.

WE-L.C.d (contrastive near-miss). A team applies LoRA with $r=8$ to teach a 7B VLA a new embodiment with a 7-DOF arm whose joint ordering differs from the training set. The model learns the new camera and the new instructions but cannot reach the object. Why it fails: the new embodiment requires a different mapping from image space to action space, which is a representation change, not a readout change; rank 8 cannot express it. The fix: retrain the action head from scratch on the new embodiment’s data, and use LoRA only on the vision-language path for semantic adaptation.

Comparative table: adaptation strategies.

Design principle P36.3. Match the adaptation mechanism to the kind of change required: readout changes take a head, calibration changes take low-rank adapters, representation changes take full fine-tuning, and no adapter can substitute for data.

Consolidation checkpoint 3

(a) Three capabilities. You can name OpenVLA’s encoder fusion and say why both encoders are present. You can compute a LoRA parameter budget. You can state when LoRA is insufficient.

(b) Transfer probe. A team must adapt a 7B VLA to a new gripper with a different opening range. Which strategy? Answer: the action-space change is a readout change if the rest of the embodiment is unchanged, so a new action head plus low-rank adapters on the fusion layers; if the gripper’s kinematics differ, the mapping changes and full fine-tuning is required.

(c) Remediation rule. If you cannot derive Eq. (C.2), re-derive it from the shapes before Part D.

Part D — The π-family and continuous actions

§L.D.1 — Why discrete tokens are not enough

Three properties of discretized autoregressive actions cause trouble at deployment.

Resolution coupling. As established in 3.1.1 and §L.B.1, resolution equals range divided by bins. Widening the range for faster motion coarsens the action.

No multimodality. Per-dimension argmax cannot represent two valid action modes, as §L.B.1 showed. In contact-rich manipulation, multiple valid grasps are common.

Sequential latency. Autoregressive decoding costs one forward pass per dimension.

A fourth property is subtler and matters most for smoothness: the tokenizer’s bin boundaries create discontinuities in the action function. A small change in the observation can flip a token, producing a step change in the commanded action — audible as jerk and visible as chatter against a stiff contact (the failure diagnosed in 2.3.3).

§L.D.2 — Flow matching

π0 (Black et al. 2024, tier B, S) replaces the discrete head with a flow-matching head over continuous action chunks. The idea is to learn a velocity field that transports a simple distribution (Gaussian noise) to the action distribution.

Let $a \in \mathbb{R}^{d_a}$ be an action chunk and $a^0 \sim \mathcal{N}(0, I)$. Define a probability path $a^t = (1-t)a^0 + t a^1$ for $t \in [0,1]$, where $a^1$ is a demonstrated chunk. The conditional velocity along the path is $a^1 - a^0$, and the training loss regresses the network’s predicted velocity onto it:

At inference, start at $a^0 \sim \mathcal{N}$ and integrate

with Euler steps for $K$ steps, then execute the resulting chunk. The head produces the entire chunk in one shot per integration step, so the latency profile is $K$ passes rather than $d_a$ passes, and the output is continuous and multimodal.

The system-level choice that accompanies the flow head is knowledge insulation: the gradients from the action loss are not backpropagated through the vision-language path. The vision-language path is trained only on its own objectives, so the semantic knowledge that makes RT-2 generalize is preserved rather than overwritten by motor gradients. This is the architectural descendant of the co-fine-tuning insight of §L.B.3, implemented as a gradient mask instead of a loss weight.

π0-FAST replaces the flow head with a frequency-domain tokenizer (FAST) that compresses action sequences into discrete tokens efficiently, giving an autoregressive alternative that keeps the language-model interface while reducing sequence length. The design space between autoregressive tokens and continuous flow is therefore not binary; the tokenizer’s compression is a third axis.

Self-explanation 2. Why does flow matching represent multimodality that per-dimension argmax cannot?

Answer. Because the velocity field is a function over the whole action vector, so the transport can split probability mass between two separated modes; per-dimension argmax collapses each marginal independently and cannot.

Concept check. A flow head is trained with $K = 10$ Euler steps. What happens at $K = 1$, and what happens at $K = 100$?

Answer. At $K=1$ the integration is a single Euler step from noise, which is a coarse transport and produces blurry, averaged actions; at $K=100$ the trajectory is accurate but the latency is ten times higher. $K$ trades action fidelity against inference cost. [displaces M4]

§L.D.3 — Action chunks and the blind window

Every modern VLA outputs an action chunk: a sequence of $k$ future actions produced by one inference. The chunk length $k$ and the execution rate $f$ define the open-loop blind window

During the blind window, the policy is not observing; if the environment changes, the policy cannot react until the next inference. The trade is direct: a longer chunk amortizes inference cost but lengthens the blind window. ACT (Zhao et al. 2023, tier A, S) introduced chunking for this reason and reported that it addresses the non-stationarity of human demonstrations — the policy commits to a short trajectory rather than chasing a noisy per-step target.

The running case’s constraint is set by 2.3.3: contact must be detected within 3 ms and responded to within a few control cycles. A chunk that commands the arm for 800 ms blinds it for that long. The composition answer of 3.1.1 reappears: the chunk drives the arm during free-space motion, and a fast controller takes over when contact is detected, preempting the remainder of the chunk.

Worked-example ladder.

WE-L.D.a (fully worked). Derive the flow-matching loss for a one-dimensional action with two demonstrated modes, $a^1 = +1$ and $a^1 = -1$, each with probability one half, and a Gaussian $a^0$. Compute the optimal velocity at $a^t = 0$, $t = 0.5$.

Solution. The conditional velocity is $a^1 - a^0$. At $a^t = 0.5 a^1 + 0.5 a^0 = 0$, for the $+1$ mode $a^0 = -1$ and the velocity is $1 - (-1) = 2$; for the $-1$ mode $a^0 = +1$ and the velocity is $-1 - 1 = -2$. Averaging with equal weights gives zero. The optimal field at the midpoint is zero: the two modes cancel. The inference then continues along whichever mode the integration’s noise determines, which is exactly the multimodality the head represents.

WE-L.D.b (completion). With $K=1$ and the field above, what does the model produce? Completion. Starting from noise $a^0$, one Euler step with $\Delta t = 1$ gives $a^1 = a^0 + v(a^0, 0)$. With the symmetric field, points near zero receive near-zero velocity and stay near zero — the averaged, non-committal action. This is the blurring failure of $K=1$.

WE-L.D.c (faded). Derive the chunk-length rule from the deployment’s reaction requirement: $k \le f \cdot T_{react}$, where $T_{react}$ is the longest acceptable blind time. Apply it to the running case with $f = 10$ Hz and $T_{react} = 0.3$ s. Faded step: $k \le 3$. Completion. A three-action chunk at 10 Hz gives a 300 ms blind window, which is longer than the 3 ms contact requirement but acceptable because the specialist controller preempts on contact.

WE-L.D.d (contrastive near-miss). A learner trains a π0-style flow head with $K=10$ but executes only the first action of each chunk, re-planning at every step. The motion is smooth but the inference cost is tenfold and the arm stutters. Why it fails: re-planning every step destroys the chunk’s latency benefit and introduces discontinuities between successive chunks because each is integrated from independent noise. The fix: execute the whole chunk, or use temporal ensembling across overlapping chunks (as in ACT) so that successive predictions are blended rather than concatenated.

Comparative table: action-decoder families.

Design principle P36.4. Choose the decoder by the property the task demands: resolution, multimodality, smoothness, or latency. No decoder wins on all four, and the choice is a deployment decision, not a training detail.

Consolidation checkpoint 4

(a) Three capabilities. You can write the flow-matching loss and the Euler integrator. You can derive the chunk-length rule from a reaction requirement. You can state knowledge insulation and the failure it prevents.

(b) Transfer probe. A task requires two distinct grasp modes and a 100 ms blind window at 20 Hz. What chunk length and decoder? Answer: $k \le 2$ at 20 Hz; a flow or diffusion head for the multimodality, with $K$ chosen so that the integration fits within the 50 ms per-chunk budget.

(c) Remediation rule. If you cannot derive the optimal velocity at the midpoint of WE-L.D.a, re-derive Eq. (D.1) before Part E.

Part E — The running case: choosing an architecture

§L.E.1 — The requirements, stated numerically

The running case’s cell needs, in order of priority: (i) a placement tolerance of 2 cm; (ii) a contact-response requirement of 3 ms, handled by the 2.3.3 controller; (iii) instruction following for at least 20 pick-and-place variants across five objects; (iv) an onboard power budget of about 25 W; (v) a target of 90% success on seen variants and 70% on unseen objects.

The requirements convert into three filters.

The latency filter. The VLA cannot be in the contact loop, so it must be composable: it drives free-space motion and hands off before contact. Any architecture qualifies provided the handoff is implemented. The filter therefore does not eliminate architectures; it eliminates uncomposed deployments.

The power filter. About 25 W excludes a 55B model on any current accelerator and excludes cloud inference if network latency is unacceptable. It admits a 7B model with quantization and admits a 3B model comfortably. This is the filter that actually decides.

The data filter. The running case’s 2.2.4 dataset is 100k transitions, roughly 50 episodes. From 3.1.1, this is far too little for pretraining; it can only fine-tune. That eliminates “train from scratch” and points at the adaptation ladder of §L.C.2.

§L.E.2 — The selection and the composition

Running the filters leaves two candidates.

Candidate A: OpenVLA-7B, quantized, with LoRA on the vision-language path and a retrained action head. It passes the data filter because fine-tuning is the intended use; it passes the power filter only marginally, and quantization is required. Its strength is the DINOv2-plus-SigLIP fusion, which is the best of the four for the geometric precision that the running case needs to resolve a 5 cm cube at 30 cm.

Candidate B: a π0-scale (about 3B) model with a flow head, trained on the running case’s data plus a public manipulation corpus. It passes the power filter comfortably and the latency profile is better because the flow head emits chunks rather than sequential tokens. Its weakness is data: at 3B, the model has less capacity to absorb the running case’s small dataset without overfitting, so the LoRA/insulation choices matter more.

The recommendation is Candidate B when the flow head’s chunking is needed for smoothness, and Candidate A when the geometric precision of the fused encoder dominates. For the running case specifically, the 2 cm placement tolerance is generous relative to both models’ resolution, and the smoothness of the approach matters more because the 2.3.3 controller’s impedance loop amplifies jerk into force oscillation. That argues for B.

The composition is the same either way, and it is the part that determines success:

• VLA drives from home to a handoff pose 5 cm above the bin, at 10 Hz with a 3-action chunk.
• The 2.1.2 pose estimate confirms the cube is grasped; the 2.3.3 impedance controller takes over the descent.
• The 2.2.4 Q-shield monitors the VLA’s actions and preempts to the gait planner if the value falls below the offline 10th percentile.
• The 1.1.4 state machine treats the handoff as a checkpoint with health checks.

§L.E.3 — The benchmark protocol

Comparing architectures on the running case requires a protocol that isolates the four blocks.

Ablation 1: instruction grounding. Hold the object fixed, vary the instruction phrasing across five paraphrases per task. A model that fails here has a language-path deficit.

Ablation 2: visual grounding. Hold the instruction fixed, move the object to five positions including one outside the training distribution. A model that fails here has a vision-path deficit.

Ablation 3: motor precision. Hold both fixed, measure placement error distribution. A model with a coarse decoder shows a discretized error histogram; a flow head shows a continuous one.

Ablation 4: latency. Measure end-to-end from image capture to first action, at the deployment’s power budget. Report the 95th percentile, not the mean, because the tail is what causes contact failures.

The protocol’s value is diagnostic, not comparative: it tells you which block fails, which is what determines the fix.

Self-explanation 3. Why is a discretized decoder’s error histogram a fingerprint of its bin count?

Answer. Because the output can only take values on the bin grid, so the error distribution inherits the grid’s periodicity; a Fourier transform of the histogram shows a peak at the bin frequency.

Worked-example ladder.

WE-L.E.a (fully worked). OpenVLA at 7B runs at 6 Hz on a modern GPU drawing 250 W. Compute the actions per joule and compare with a 3B model at 10 Hz drawing 90 W.

Solution. OpenVLA: 6 actions/s over 250 W gives 0.024 actions per joule. The 3B: 10 over 90 gives 0.111 actions per joule, 4.6 times better. At a 25 W budget neither runs unmodified; quantization to 4-bit reduces the 3B model’s draw to roughly 25–35 W, making it the only viable candidate.

WE-L.E.b (completion). The 3B model at 10 Hz with a chunk of 3 gives a 300 ms blind window. Does the 2.3.3 controller’s 3 ms contact response still matter, and how? Completion. Yes: the controller runs at 1 kHz independently of the VLA, so contact response is unaffected; the blind window affects only the free-space approach, where a 300 ms stale observation can mean a 3 cm position error at 10 cm/s. The handoff pose must therefore be defined with a 5 cm margin, not 1 cm.

WE-L.E.c (faded). Design the evaluation matrix that separates architecture effects from data effects: vary the data volume at two levels and the architecture at two levels, and state the number of conditions. Faded step: $2 \times 2 = 4$ conditions, each with 20 trials per task. Completion. The interaction term is the quantity of interest: if the larger model’s advantage grows with data, the data is the binding constraint; if it is flat, the architecture is.

WE-L.E.d (contrastive near-miss). A team benchmarks OpenVLA and π0 on 100 trials each of seen tasks, finds OpenVLA better by 4 percentage points, and concludes OpenVLA is the better architecture. The deployment then fails on unseen objects. Why it fails: the benchmark measured only seen-task accuracy, which the §L.B.2 ablation says is not where co-fine-tuning helps; the unseen-object ablation was omitted. The fix: run all four ablations of §L.E.3, and report per-ablation results.

Comparative table: running case candidates.

Design principle P36.5. Filter architectures by hard deployment constraints first, then choose among survivors by the ablation that matches the task’s dominant risk. Comparisons without ablations rank models on the wrong axis.

Consolidation checkpoint 5

(a) Three capabilities. You can convert requirements into filters. You can design the VLA-specialist composition with handoff. You can run the four-ablation protocol and read its diagnosis.

(b) Transfer probe. The cell’s power budget rises to 150 W. Which candidate wins, and why? Answer: Candidate A becomes viable and its encoder advantage matters more, so A wins unless the smoothness ablation shows the discretization jerk exceeds the impedance controller’s tolerance.

(c) Remediation rule. If you cannot state which ablation isolates a language-path deficit, re-read §L.E.3 before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: forgetting after fine-tuning, latency tail, and long-episode drift.

The lab

Task. Implement the four-block skeleton as a parameter-counting and latency-budgeting model for the four architectures, and implement a one-dimensional flow-matching head trained on a bimodal toy distribution to demonstrate that the head captures multimodality while a per-dimension tokenizer does not.

Conditions. Python 3.10+, PyTorch 2.x, CPU sufficient. Toy action dimension 1, two modes at $+1$ and $-1$. Seed all RNGs.

Controls. (i) The skeleton accounting model with the published parameter counts; (ii) a 256-bin tokenizer trained on the bimodal data; (iii) a flow head with $K \in {1, 5, 10}$; (iv) the running case’s latency arithmetic with measured per-pass times.

Metrics.

Pre-registered hypotheses.

H1. The tokenizer trained on bimodal data produces a unimodal output distribution with mass concentrated between the modes, because per-bin cross-entropy with a single label per sample cannot represent a mixture.

H2. The flow head at $K=10$ produces samples near both modes in roughly equal proportion, because the velocity field preserves the bimodality of the conditional paths.

H3. The flow head at $K=1$ blurs toward zero, consistent with the cancellation derived in WE-L.D.a.

H4. The accounting model’s predicted parameter counts match the published figures within 10% for all four architectures.

Break-it sequence.

1. Reduce the flow head’s $K$ to 1 silently. The output distribution becomes unimodal near zero. Silent: the loss is unchanged; only the sampling reveals it.
2. Freeze the fusion block silently. Train the action head only. Paraphrase success collapses while seen-task success is unchanged. Silent: training curves look healthy.
3. Mis-weight co-fine-tuning silently. Set the web loss weight 20 times the robot weight. Seen-task success falls without an obvious training failure. Silent: both losses decrease.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The accounting model is a function, not a spreadsheet, so it can be rerun with new sizes. (2) The flow head’s samples are plotted as a histogram at each $K$. (3) The tokenizer’s failure is shown as a histogram too, not asserted. (4) The break-it results are a table. (5) The latency arithmetic uses measured per-pass times, not assumed ones.

A minimal flow head, seeded and annotated:

# lab_3_1_2.py — bimodal flow matching versus per-dimension tokenization
import numpy as np, torch, torch.nn as nn
torch.manual_seed(0); np.random.seed(0)

def sample_data(n):                                   # bimodal demonstrations
    modes = torch.where(torch.rand(n, 1) < 0.5, 1.0, -1.0)
    return modes + 0.05 * torch.randn(n, 1)           # small within-mode noise

class Velocity(nn.Module):                            # $v_theta(a^t$, t)
    def __init__(self, h=64):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(2, h), nn.Tanh(),
                                 nn.Linear(h, h), nn.Tanh(),
                                 nn.Linear(h, 1))
    def forward(self, $a_t$, t):
        return self.net(torch.$cat([a_t$, t], dim=1))

v = Velocity(); opt = torch.optim.Adam(v.parameters(), lr=1e-3)

for step in range(3000):                              # train the velocity field
    a1 = sample_data(256)                             # demonstrated action
    a0 = torch.randn_like(a1)                         # noise
    t  = torch.rand(256, 1)                           # path time
    $a_t$ = (1 - t) * a0 + t * a1                       # probability path
    target = a1 - a0                                  # conditional velocity
    loss = (($v(a_t$, t) - target) ** 2).mean()         # Eq. (D.1)
    opt.zero_grad(); loss.backward(); opt.step()

def integrate(K):                                     # Euler integration, Eq. (D.2)
    a = torch.randn(2000, 1)
    for i in range(K):
        t = torch.full((2000, 1), i / K)
        a = a + (1.0 / K) * v(a, t)
    return a.detach()

for K in (1, 5, 10):                                  # multimodality vs. step count
    s = integrate(K)
    near_pos = (s > 0.5).float().mean().item()
    near_neg = (s < -0.5).float().mean().item()
    near_zero = (s.abs() < 0.5).float().mean().item()
    print(f"K={K:2d}  +mode {near_pos:.2f}  -mode {near_neg:.2f}  blurred {near_zero:.2f}")

# Contrast: 256-bin tokenizer on the same data, argmax per sample
bins = torch.linspace(-2, 2, 256)
labels = torch.argmin((sample_data(2000) - bins).abs(), dim=1)
hist = torch.bincount(labels, minlength=256).float()
print("tokenizer mass between modes:",
      hist[90:166].sum().item() / hist.sum().item())  # bins near zero

Line 6 generates the bimodal demonstrations; line 17 trains the velocity field with the loss of Eq. (D.1); line 25 is the Euler integrator of Eq. (D.2). Lines 31–36 print the modality statistics for each $K$, which is the experiment’s core: at $K=1$ the blurred fraction is high, and it falls as $K$ rises. Lines 39–42 show the tokenizer’s mass concentrating between the modes, the failure H1 predicts.

Cheat sheet

Exercises with solutions

E 3.1.2.1. Decompose RT-1 into the four blocks and state which block its 3 Hz control rate most constrains.

S 3.1.2.1. Encoder: EfficientNet-B3 plus TokenLearner. Language: USE embedding with FiLM. Fusion: late, conditioning at each transformer layer. Decoder: discrete tokens, 11 dimensions, 256 bins. The decoder constrains the rate most, because 11 sequential forward passes per action set the floor. The lesson: the rate is set by the slowest block, and for autoregressive heads that is the decoder.

E 3.1.2.2 (transfer). A history of 4 frames at 64 tokens each plus a 20-token instruction, model width 768. Compute the attention-cost exponent and compare with 4 frames at 16 tokens each.

S 3.1.2.2. At 64 tokens: length $256 + 20 = 276$, squared $76{,}176$, times 768 gives $5.85 \times 10^7$. At 16 tokens: length $64 + 20 = 84$, squared $7{,}056$, times 768 gives $5.42 \times 10^6$. The ratio is 10.8. The lesson: the instruction offset matters most at short sequences, so token reduction has diminishing returns once the history is small.

E 3.1.2.3. State precisely what RT-2’s web-data ablation showed and what it did not show.

S 3.1.2.3. It showed that removing web co-fine-tuning degrades performance on generalization benchmarks involving unseen objects and instructions, while seen-task accuracy is largely preserved. It did not show that web data improves seen-task accuracy, nor that web data substitutes for robot data in the motor distribution. The lesson: the ablation separates semantic breadth from motor competence, and the two have different data requirements.

E 3.1.2.4 (diagnostic). A VLA fails on “put the cube on the left of the bin” but succeeds on “put the cube in the bin”. Which block, and which test confirms it?

S 3.1.2.4. The fusion block, because the relative spatial reference requires language-to-image attention. The confirming test is to place the bin so that “in” and “left of” coincide: if the model now succeeds on both phrasings, the deficit is compositional spatial fusion. The lesson: spatial relations localize failures to fusion, not to recognition.

E 3.1.2.5. Compute the LoRA parameter count for a 24-layer, 2048-wide transformer adapting only the four attention projections at rank 16.

S 3.1.2.5. $24 \cdot 4 \cdot 16 \cdot 2 \cdot 2048 = 6.29 \times 10^6$. The lesson: the count scales linearly in layers, width, and rank, so the budget can be estimated before a single gradient step.

E 3.1.2.6 (transfer). A new embodiment changes the arm from 6 to 7 DOF. Which adaptation strategy, and why not LoRA at rank 32?

S 3.1.2.6. Retrain the action head from scratch, because the output dimensionality and the kinematics both change; LoRA at rank 32 constrains the update to a low-rank subspace of the existing head’s weights, which cannot represent a different output space. Keep LoRA on the vision-language path for semantic adaptation. The lesson: output-space changes are not low-rank changes.

E 3.1.2.7. Derive the optimal flow-matching velocity at $a^t = 0.2$ for a single mode $a^1 = 1$ with $a^0 = 0$ at $t = 0.2$.

S 3.1.2.7. The path is $a^t = (1-t)a^0 + t a^1 = 0.2 \cdot 1 = 0.2$, consistent. The conditional velocity is $a^1 - a^0 = 1$, constant along the path. The optimal field at that point is 1. The lesson: for a single mode with a straight path, the field is constant; curvature and multimodality are what make the field non-trivial.

E 3.1.2.8 (diagnostic). A flow head at $K=1$ produces blurred actions. Name two remedies and the cost of each.

S 3.1.2.8. (1) Raise $K$, costing inference passes linearly. (2) Use a straightened path (as in rectified flow) so that fewer steps suffice, costing additional training. The lesson: blur at low step counts is an integration-accuracy problem, and there are two independent knobs.

E 3.1.2.9. State the chunk-length rule and compute the maximum chunk at 10 Hz with a 250 ms reaction requirement.

S 3.1.2.9. $k \le f T_{react} = 10 \cdot 0.25 = 2.5$, so $k=2$. The lesson: the blind window is a deployment constraint that bounds the chunk before any accuracy consideration.

Exit test

You are ready for Lesson 3.1.3 when you can, without notes:

1. Decompose any of the four architectures into the four blocks.
2. Distinguish late and early fusion and name the instruction type each fails.
3. Compute attention cost from history, tokens per image, and instruction length.
4. State RT-1’s token budget and why the TokenLearner existed.
5. State RT-2’s structural change and the co-fine-tuning loss.
6. State what the web-data ablation showed and what it did not.
7. Balance a co-fine-tuning loss by gradient norm.
8. Describe OpenVLA’s fused encoder and why two encoders are present.
9. Compute a LoRA parameter budget and state its limitation.
10. Match an adaptation strategy to a change type.
11. Write the flow-matching loss and the Euler integrator.
12. Explain why flow heads represent multimodality and per-dimension argmax does not.
13. State knowledge insulation and the failure it prevents.
14. Derive the blind window and choose a chunk length from a reaction requirement.
15. Run the four-ablation protocol and read its diagnosis.

Cross-references. 3.1.1 supplies the VLA interface, the three groundings, and the resolution–range trade restated in §L.B.1 and §L.C.2. 2.3.3 supplies the 3 ms contact response restated in §L.D.3 and §L.E.2. 2.3.4 supplies the distillation pattern restated in §L.B.3. 2.2.4 supplies the Q-shield restated in §L.E.2. Lesson 3.1.3 uses the four-block anatomy to determine what must be annotated and how much data each block requires. Lesson 3.1.4 uses the adaptation ladder of §L.C.2 and the ablation protocol of §L.E.3 as the skeleton of the fine-tuning pipeline.

Ready. The next lesson to write is 3.1.3 — Data Collection and Annotation for VLAs. Say “next” to generate it.