Lesson 3.1.3 — Data Collection and Annotation for VLAs

Week 18

Prerequisite lessons: 3.1.1 (introduction to VLAs), 3.1.2 (VLA architectures), 2.2.4 (offline-to-online fine-tuning), 2.3.4 (in-hand and dexterous manipulation), 2.1.2 (object detection and 6D pose estimation).

Prior results used in this lesson. From 3.1.1: effective sample size is the episode count, not the frame count, and instruction diversity is a data parameter. From 3.1.2: the four-block anatomy, the web-data ablation’s precise claim, and the gradient-norm balancing rule of Eq. (B.1). From 2.2.4: the deploy–collect–fine-tune loop and the Q-shield. From 2.3.4: the state-distribution argument for distillation. Each is restated at the point of use.

What this lesson hands forward. The episode schema, the collection cost model, the annotation contract, and the split-and-curation protocol. Lesson 3.1.4 assumes this schema exists and treats fine-tuning as the operation performed on it. Module 4.3 uses the annotation contract to define what an experiment log must contain.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, and divider are unchanged; this lesson asks what must be recorded, labelled, filtered, and split before that cell’s data can train a VLA, and what each of those steps costs in hours.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) write the episode schema as a list of fields with types and rates; (2) compute storage and bandwidth from modality, resolution, and rate; (3) state the three teleoperation families and their throughput; (4) distinguish a successful episode from a good demonstration and define the quality gate; (5) state the non-stationarity problem and how action chunking addresses it; (6) write the annotation contract with four label types and their costs; (7) compute paraphrase coverage and justify a minimum; (8) demonstrate why frame-level splitting leaks and episode-level splitting does not; (9) define deduplication, filtering, and balancing rules and their effect on effective sample size; (10) design a stratified collection plan for the running case with quality gates, annotation budget, and splits.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 3.1.1, effective sample size is the episode count rather than the frame count. State the reasoning in one sentence and give the consequence for a dataset of 100,000 frames drawn from 50 episodes.

Answer. Consecutive frames are correlated, so the information content scales with the number of independent trajectories, not with the number of frames. The consequence is that 100,000 frames from 50 episodes has an effective sample size of 50. Used in §L.D.1 to explain why deduplication at the episode level matters more than frame counts suggest.

Q2. In 3.1.2, the web-data ablation showed that removing web co-fine-tuning degrades unseen-object generalization while seen-task accuracy is preserved. What does that imply about which data must be collected for a new motor skill?

Answer. It implies that robot data is the only source for motor competence; web data supplies semantic breadth and cannot substitute for demonstrations of the new skill. Used in §L.B.1 to argue that collection volume for a new skill is set by the motor distribution, not the semantic one.

Q3. In 2.3.4, the student is trained on the teacher’s state distribution. For a VLA trained on teleoperation, what is the analogous distribution and what happens if training data is collected by a scripted policy instead?

Answer. The analogous distribution is the states the teleoperators visit under their own closed-loop control. A scripted policy visits a smoother, narrower set, so a VLA trained on it fails on the corrective states a human would produce. Used in §L.B.3 to explain why corrective demonstrations are required.

Q4. In 2.2.4, the deployment pipeline collects data and fine-tunes, then monitors and commits or rolls back. What does the collection phase of that pipeline produce that a static offline dataset does not?

Answer. It produces data on the deployed policy’s own state distribution, including the states where the policy is uncertain. Used in §L.D.3 to distinguish offline curation from online data aggregation.

Part A — The episode record

§L.A.1 — What must be in the record

An episode is the unit of robot data. It is one attempt at one task, recorded as a time-ordered sequence of observations, actions, and labels. Everything downstream — the annotation contract, the split protocol, the training loop — operates on episodes, so the schema is the foundation.

The schema has six field groups. Each field exists because some downstream consumer reads it; a field with no consumer is a collection cost with no benefit.

Group 1: observations. RGB frames, depth frames, joint positions, joint velocities, and end-effector pose. Each has a rate and a type. The rate is a collection decision that cannot be changed later.

Group 2: actions. The commanded action at each control step, in the space the policy will output. For the running case that is a 7-dimensional end-effector delta at 10 Hz, matching 3.1.1’s action space. If the recorded action is a joint torque and the policy outputs an end-effector delta, the mapping must be reconstructed from the dynamics, which introduces error.

Group 3: the instruction. Recorded as a string, with the timestamp at which it was issued. The timestamp matters: an instruction issued after the first action teaches the policy to condition on an intent that had not yet been formed.

Group 4: episode metadata. Operator identity, collection date, object identity, task identity, site and lighting condition, and the collection policy (teleoperation, scripted, or autonomous). Each of these is a potential leakage or distribution-shift axis.

Group 5: outcome labels. Success or failure, and the failure mode if known. Assigned at collection time by the operator and verified by an automated check.

Group 6: derived labels. Segmentation boundaries, success sub-goals, and grounding annotations. These are the annotation layer of Part C and are attached after collection but keyed to the episode ID.

Self-explanation 1. Why must the action be recorded in the space the policy will output, rather than in the space the controller consumes?

Answer. Because the training loss compares predicted and recorded actions; if the recorded action is in a different space, the comparison requires an inverse mapping that is not unique.

§L.A.2 — Storage and bandwidth arithmetic

The arithmetic is simple and the numbers are large, which is why the schema decision matters.

For an episode of $N$ steps at rate $f$, the storage is

with

The RGB term dominates. At $640 \times 480$, three channels, one byte per channel, one frame is 921,600 bytes. At 10 Hz, one second is 9.2 MB; a 40-second episode is 368 MB. Depth at two bytes per pixel adds 614,400 bytes per frame, or 246 MB per episode. Proprioception and actions are negligible by comparison.

This is why robot datasets are stored as compressed video plus metadata, not as raw frame arrays, and why the compression choice is part of the schema: a lossy codec that was trained for human viewing can destroy the fine texture that a policy relies on for slip detection.

Concept check. A team records RGB at 30 Hz and downsamples to 10 Hz at training time. What has the collection decision cost them?

Answer. It has cost them nothing in storage efficiency relative to recording at 10 Hz once, and it has preserved the option to train at 30 Hz later; the cost is the 3× storage during collection. The mechanism is that downsampling is irreversible but upsampling is not, so over-recording the rate is a cheap option and under-recording is an expensive mistake. [displaces M5]

Worked-example ladder.

WE-L.A.a (fully worked). Compute the storage for a 2,000-step episode recorded at 10 Hz with the running case’s camera. RGB at 640×480×3 bytes, depth at 640×480×2 bytes, proprioception as 6 floats (4 bytes each), actions as 7 floats.

Solution. $S_{rgb} = 921{,}600$ B, $S_{depth} = 614{,}400$ B, $S_{prop} = 24$ B, $S_{act} = 28$ B. Per step: 1,536,052 B, about 1.54 MB. Per episode: $2{,}000 \times 1.54$ MB $= 3.07$ GB. A 50-episode dataset is 153 GB before compression.

WE-L.A.b (completion). Compress RGB with a codec at 20:1 and depth as 8-bit at 2:1. Recompute the episode size and the dataset size.

Completion. RGB: 46,080 B per frame. Depth: 307,200 B per frame. Per step 353,308 B, about 0.35 MB. Per episode 707 MB; 50 episodes 35 GB. The completion shows that the compression choice is a 4.4× factor, which is the difference between a dataset that fits on a workstation and one that does not.

WE-L.A.c (faded). Derive the bandwidth requirement for streaming the data to a training cluster during collection, given a 250-episode-per-day collection rate.

Faded step. At 707 MB per episode and 250 episodes per day, the daily volume is 177 GB; over an 8-hour collection day that is 6.1 MB/s sustained, well within a local network. Completion. The completion shows that bandwidth is rarely the constraint; storage growth is. The schema decision that matters is retention policy, not link speed.

WE-L.A.d (contrastive near-miss). A team records only the final joint positions of each episode to save space, arguing that the policy can be trained on the final state. The trained policy has no velocity information and cannot reproduce the approach phase. Why it fails: the action is a function of the state and its history; a final state is one sample from a trajectory, not a demonstration of it. The fix: record the full trajectory at the control rate, and compress it rather than subsample it.

Comparative table: field groups and their consumers.

Design principle P37.1. Record the richest representation the storage budget permits, and downsample in the training pipeline rather than at collection. Downsampling is reversible only in the direction that costs storage.

Consolidation checkpoint 1

(a) Three capabilities. You can write the six field groups with types and rates. You can compute episode storage from modality and rate. You can state which collection decisions are irreversible.

(b) Transfer probe. A depth sensor returns 16-bit values at 30 Hz. The policy consumes 8-bit depth at 10 Hz. Should collection record 16-bit at 30 Hz or 8-bit at 10 Hz? Answer: 16-bit at 30 Hz, because the higher precision supports the 2.1.2 pose estimator and the higher rate supports slip detection; both downsamples are then training-time choices.

(c) Remediation rule. If you cannot compute the storage of WE-L.A.a, redo Eq. (A.1) before Part B.

Part B — Collection

§L.B.1 — The three collection families

Robot data is collected by one of three mechanisms, and they produce different distributions.

Scripted or classical policies. The 1.1.4 pipeline, or the 2.3.2 planner, drives the arm and the data is logged. Throughput is high, cost is low, and the distribution is narrow and smooth: the scripted policy visits only the states its program produces. It cannot produce the corrective states that a human produces when something goes slightly wrong, because the script never goes slightly wrong.

Teleoperation. A human drives the arm through a leader arm, a VR controller, a spacemouse, or kinesthetic teaching. Throughput is lower, cost is higher, and the distribution is broad and non-stationary: humans pause, overshoot, and correct. That non-stationarity is a feature for coverage and a nuisance for learning.

Autonomous rollout with human correction. A policy drives and a human intervenes when it errs. This is the DAgger family (Ross, Gordon, Bagnell 2011, tier A, S) and it is the mechanism that produces data on the policy’s state distribution rather than the expert’s, which is the distribution the policy will actually occupy at deployment.

The choice is not exclusive. A practical pipeline uses scripted collection for the bulk of free-space motion, teleoperation for the contact-rich segments, and autonomous rollout plus correction once a first policy exists.

§L.B.2 — The quality gate: successful is not the same as good

A successful episode reaches the goal. A good demonstration reaches the goal by a path worth imitating. The two are different, and confusing them is the third misconception.

Three failure patterns produce successful-but-bad demonstrations. A lucky recovery succeeds after an error that a policy trained on it will reproduce without the recovery, because the recovery is a small part of the trajectory. A hesitant episode pauses repeatedly at decision points; a policy trained on it learns to hesitate. A jerky episode has high-frequency corrections that the action space cannot represent smoothly, and the policy learns the jitter.

The quality gate is a set of automated and human checks applied before an episode enters the dataset:

• Success, verified by a state check rather than by the operator’s opinion.
• Smoothness, measured as the mean absolute jerk of the end-effector trajectory, below a threshold calibrated from the best quartile of pilot data.
• Duration, within a band around the median for the task, so that neither rushed nor hesitant episodes dominate.
• Contact-force peak, below the 2.3.3 limit, so that no episode demonstrates a grasp that would be rejected by the safety layer.
• Instruction timing, with the string recorded before the first action.

Episodes that fail the gate are not discarded; they are relabelled as negative examples with their failure mode, because recovery training needs them and because a policy that has never seen a failure cannot recognize one.

Self-explanation 2. Why does a hesitant demonstration teach hesitation even though every episode in the dataset is successful?

Answer. Because the loss is over the whole trajectory, and the pause occupies many frames; the policy minimizes average error and therefore reproduces the most common behaviour at each state, which is pausing.

§L.B.3 — The non-stationarity problem and corrective data

Human demonstrations violate the Markov assumption in a specific way: the operator’s action depends on their intent and history, not only on the current observation. The same observation can be followed by different actions depending on whether the operator is about to commit or about to reconsider. A policy trained with a per-step regression loss sees this as noise and averages the two, producing a tentative motion that commits to neither.

Action chunking (Zhao et al. 2023, tier A, S; restated from 3.1.2 §L.D.3) addresses this by making the prediction target a short trajectory rather than a single action. The policy commits to a chunk, and the chunk can be one of several modes rather than an average. This is why chunking is not only a latency optimization; it is a response to demonstration non-stationarity.

The second response is corrective data. The 3.1.2 §L.B.3 result — that co-fine-tuning does not fix a state-distribution gap — implies that a policy trained only on clean demonstrations will fail on the states it reaches after a small error. Those states are not in the clean data. Collecting corrections on the policy’s own rollouts closes the gap, and it is the mechanism 2.2.4’s deploy–collect–fine-tune loop already implements.

Worked-example ladder.

WE-L.B.a (fully worked). An operator completes a pick-and-place in 25 seconds including reset. Compute the throughput in episodes per hour, and the operator-hours for 4,000 episodes.

Solution. $3{,}600 / 25 = 144$ episodes per hour. For 4,000 episodes: $4{,}000/144 = 27.8$ operator-hours, about 3.5 eight-hour days with one operator.

WE-L.B.b (completion). The quality gate rejects 30% of episodes and resets add 10 seconds. Recompute throughput and operator-hours.

Completion. Cycle time 35 s gives 103 episodes per hour, of which 72 pass the gate. For 4,000 accepted episodes the operator must record 5,556 attempts, taking 54 operator-hours. The completion shows that the gate’s rejection rate is a direct multiplier on collection cost and must be budgeted.

WE-L.B.c (faded). Derive the corrective-data requirement: how many interventions per deployment hour are needed to keep the policy’s state distribution covered?

Faded step. Estimate the rate at which the policy reaches states outside the demonstration distribution, using the Q-shield’s rejection rate as a proxy; each rejection is a state that needs a correction. Completion. The rate is task-dependent; the operational rule is to collect until the rejection rate over a fixed evaluation window stops falling.

WE-L.B.d (contrastive near-miss). A team collects 2,000 episodes from a single expert operator at maximum speed, achieving a 98% success rate. The trained policy is fast and erratic and fails whenever the cube is slightly off the demonstrated position. Why it fails: one operator gives one style, the speed maximizes jitter, and the narrow position coverage means the off-position states are unrepresented. The fix: multiple operators, a duration band in the quality gate, and position randomization in the collection protocol.

Comparative table: collection families.

Design principle P37.2. Collect on the distribution the policy will occupy, not on the distribution the expert prefers. Clean demonstrations are necessary and insufficient.

Consolidation checkpoint 2

(a) Three capabilities. You can compute throughput and operator-hours. You can write the five quality-gate checks. You can state why chunking responds to non-stationarity.

(b) Transfer probe. A cell has 500 clean demonstrations and a deployed policy with a 15% intervention rate. What is the next collection step? Answer: collect the interventions as corrective data, label them, and add them to the dataset; the clean demonstrations cannot be extended to cover the deployed distribution.

(c) Remediation rule. If you cannot state why a successful episode may be a bad demonstration, re-read §L.B.2 before Part C.

Part C — Annotation

§L.C.1 — The annotation contract

An annotation contract specifies, for each label type, who assigns it, when, in what format, and against what verification. It exists because annotation requirements flow backward into collection: a label that requires a field you did not log cannot be produced.

Four label types matter for VLAs.

Instruction labels. The string issued before the episode began, plus at least three paraphrases. The paraphrases can be assigned after collection by a separate annotator working from the recorded object and task metadata, but the primary instruction must be recorded live, because it captures the operator’s phrasing.

Success and failure labels. A binary outcome plus a failure-mode tag drawn from a fixed vocabulary. The vocabulary should include the three groundings of 3.1.1 — visual, linguistic, motor — so that failures can be attributed. Automated verification from the final state is mandatory; operator self-report alone is unreliable at scale.

Segmentation labels. Boundaries between phases: reach, grasp, lift, transport, place, release. These support auxiliary losses, reward shaping, and diagnosis, and they are the cheapest labels to produce because they can be semi-automated from contact events and velocity profiles with human verification at the boundaries.

Grounding labels. For a subset of keyframes, the 6D pose of the referenced object and a bounding box. The 2.1.2 estimator produces these automatically; human annotation verifies a sample. The subset should over-represent the frames where the estimator is least confident.

Concept check. A team realizes after collection that it never recorded which object the instruction referred to. Which label types can still be produced and which cannot?

Answer. Instruction paraphrases can still be produced from video review, slowly and inconsistently. Grounding labels can still be produced by the pose estimator. Success labels can be produced from the final state. But the live instruction cannot be recovered, and the object reference must be inferred from the video, which reintroduces exactly the ambiguity the instruction was meant to resolve. The mechanism is that intent is a collection-time signal. [displaces M2]

§L.C.2 — Paraphrase coverage

The 3.1.1 result stands: a model trained with one phrasing per task learns instruction–task mappings, not task concepts, and fails on paraphrases. The fix is coverage, and coverage has a minimum.

Let a task have $P$ distinct phrasings in the training data and let the deployment’s phrasing distribution have support over a set $\mathcal{P}_{deploy}$. If $\mathcal{P}_{deploy} \not\subseteq \mathcal{P}_{train}$, the model must generalize across phrasings, which requires either semantic pretraining (the 3.1.2 web-data path) or enough phrasing diversity that the model learns the paraphrase relation. Empirically, three to five phrasings per task is the threshold below which paraphrase generalization is unreliable, and the phrasings should vary structure (verb choice, word order, reference style), not only synonyms.

The critical detail is distribution: the paraphrases must be spread within episodes, not concentrated in separate episodes. If each phrasing appears in its own episodes, the model can learn “this episode’s phrasing implies this episode’s trajectory” without learning the paraphrase relation.

§L.C.3 — Annotation cost

Annotation cost is the product of per-episode time, label count, and labour rate. It is often larger than the collection cost for small datasets, and it scales with the label types.

Worked-example ladder.

WE-L.C.a (fully worked). Annotating one episode with instruction plus three paraphrases takes 1.5 minutes; success and failure mode takes 0.3 minutes; segmentation takes 2 minutes; grounding for 10 keyframes takes 4 minutes. Compute the total for 4,000 episodes at 25 currency units per hour.

Solution. Per episode: $1.5 + 0.3 + 2 + 4 = 7.8$ minutes $= 0.13$ hours. For 4,000 episodes: 520 hours. At 25 units per hour: 13,000 units. The collection cost of WE-L.B.b was 54 operator-hours; annotation is roughly ten times larger.

WE-L.C.b (completion). Drop grounding labels to 2 keyframes per episode and semi-automate segmentation so that it takes 0.5 minutes. Recompute.

Completion. Per episode: $1.5 + 0.3 + 0.5 + 0.8 = 3.1$ minutes $= 0.0517$ hours. Total 207 hours, 5,170 units. The completion shows that the label types dominate the cost, and that the cheapest labels are the ones that can be semi-automated.

WE-L.C.c (faded). Derive the marginal value of a fourth paraphrase: how much unseen-phrasing accuracy does it buy, and at what cost?

Faded step. Measure paraphrase generalization as a function of phrasing count on a held-out phrasing set; the curve typically saturates around four. Completion. The fourth paraphrase buys little; the second and third buy most of the benefit. Budget for three and spend the remainder on grounding labels, which have a steeper value curve because they feed the 2.1.2 auxiliary loss.

WE-L.C.d (contrastive near-miss). A team assigns all four paraphrases to the same annotator, who develops a private vocabulary. The trained model fails on the operators’ actual phrasing. Why it fails: the annotator’s vocabulary is itself a distribution, and the model learned it rather than the deployment’s. The fix: draw paraphrases from the operators who will use the system, or from a pool instructed to match their register.

Comparative table: label types and their properties.

Design principle P37.3. Specify the annotation contract before the first episode is collected; every label that requires a field you did not log is a label you cannot buy later at any price.

Consolidation checkpoint 3

(a) Three capabilities. You can write the four label types with their verification. You can compute annotation cost. You can state the minimum paraphrase count and the distribution requirement.

(b) Transfer probe. A dataset has 5 paraphrases per task but each paraphrase appears only in its own 10 episodes. Does it satisfy the coverage requirement? Answer: no; the coverage is by label count but not by distribution within episodes, so the model can still learn phrasing-to-episode associations.

(c) Remediation rule. If you cannot state which labels cannot be recovered after collection, re-read §L.C.1 before Part D.

Part D — Curation and hygiene

§L.D.1 — Effective sample size and deduplication

From 3.1.1: effective sample size is the episode count. Deduplication operates at the same level.

Exact duplicates arise when the same episode is exported twice or when a reset procedure is recorded identically across attempts. They inflate the sample count without adding information and bias the loss toward their content.

Near duplicates are more subtle: two episodes from the same operator, on the same object, at the same position, with the same phrasing, differing only in timing noise. They contribute little diversity while consuming capacity. The detection rule is a distance in a low-dimensional summary — final object pose, trajectory length, instruction string, operator — with a threshold calibrated so that episodes differing only by timing are collapsed.

Correlated episodes are not duplicates but behave like them for the purposes of variance: a burst of 200 episodes collected in one afternoon under one lighting condition has far less than 200 episodes’ worth of environmental diversity. The curation rule is to record the collection batch in metadata and to ensure splits do not concentrate a batch on one side.

§L.D.2 — Leakage: the frame split failure

The standard machine-learning split is random over samples. For robot data it is wrong, and the wrongness is large enough to be worth demonstrating arithmetically.

Suppose 50 episodes of 2,000 frames each, split 80/20 by frame. A test frame’s immediate neighbours in time are almost certainly in the training set, and their observations differ by a few pixels. A model that memorizes training frames by nearest neighbour will score near-perfectly on the test set while having learned nothing about the task.

The correct split is by episode, stratified by task, object, operator, and collection batch. With 50 episodes, the split is 40/5/5; the test set is five episodes, which is small but honest.

The magnitude of the leakage is computable. If the frame-level nearest-neighbour distance within an episode is $\epsilon$ and the between-episode distance is $D$, a nearest-neighbour classifier evaluated on a frame split achieves error at the rate at which $\epsilon$-neighbourhoods cross the split boundary — which for a contiguous split is near zero. The lesson is that the frame split measures continuity, not generalization.

Self-explanation 3. Why does a frame-level split produce a test metric that is nearly independent of the model’s quality?

Answer. Because the test frames are interpolations of training frames, so any model with local smoothness scores well; the metric measures the smoothness prior, not the task.

§L.D.3 — Filtering, balancing, and mixing

Filtering decides which episodes enter the training set. The rules from Part B: successful and smooth episodes are positive examples; gate failures are negative examples with a failure tag; episodes with sensor faults are discarded. The filter must be applied before splitting, not after, because a post-split filter changes the split’s proportions.

Balancing decides how often each episode is seen. The natural balance is by task, object, and instruction. The failure mode is balancing by frame count: a task with longer episodes gets more frames and dominates the loss even if its episode count is equal. The correct balance is by episode, with a cap on any single episode’s weight.

Mixing with web data is the 3.1.2 Eq. (B.1) rule restated: weight by gradient norm, not by sample count. At robot-data scale the web set is thousands of times larger, so equal weighting drowns the robot signal.

Online aggregation is the 2.2.4 deploy–collect–fine-tune loop. It is not a curation step but a curation cycle: the dataset grows with the deployed policy’s state distribution, and the filter and balance rules must be reapplied each cycle. The risk is that the dataset drifts toward the policy’s own biases, which is why the Q-shield’s rejection rate is a useful monitor.

Worked-example ladder.

WE-L.D.a (fully worked). A dataset has 50 episodes of 2,000 frames. An 80/20 frame split gives 80,000 training and 20,000 test frames. An 80/20 episode split gives 40 training and 10 test episodes. Compute the ratio of training information in the two cases.

Solution. The frame split’s training set covers all 50 episodes, so it contains information from every trajectory, including the test trajectories. The episode split’s training set contains 40 independent trajectories. The frame split therefore leaks the test trajectories entirely, and its test metric is an interpolation error, not a generalization error.

WE-L.D.b (completion). With 50 episodes, is the 10-episode test set large enough? Compute the standard error of a success-rate estimate at $p = 0.9$.

Completion. $\text{SE} = \sqrt{p(1-p)/n} = \sqrt{0.09/10} = 0.095$. A 95% interval is roughly $\pm 0.19$, far too wide to distinguish a 90% policy from a 70% one. The completion shows the tension: honest splits need many episodes, which is why collection volume is set by the test set’s resolution requirement, not by the training set’s.

WE-L.D.c (faded). Derive the deduplication threshold from the requirement that two episodes differing only by timing noise are collapsed and two differing by a 1 cm object displacement are kept.

Faded step. Summarize each episode by its final object pose and duration; compute pairwise distances; set the threshold between the timing-noise distance and the 1 cm displacement. Completion. The threshold is empirical; the principle is that the threshold must be justified by the task’s tolerance, not chosen by convenience.

WE-L.D.d (contrastive near-miss). A team balances by frame count and finds that the transport-heavy tasks dominate the loss while the grasp tasks are under-learned. Why it fails: transport episodes are longer, so equal episode counts give unequal frame counts, and the loss weights frames. The fix: weight by episode, cap per-episode contribution, or sample batches by task.

Comparative table: curation operations and their targets.

Design principle P37.4. Split, filter, and balance by episode; anything done by frame over-weights long episodes and leaks neighbours.

Consolidation checkpoint 4

(a) Three capabilities. You can demonstrate frame-split leakage arithmetically. You can compute test-set standard error. You can state the three balancing rules.

(b) Transfer probe. A dataset has 200 episodes but 150 were collected in one batch. What is the curation action? Answer: record the batch in metadata, ensure the split does not concentrate the batch on one side, and treat the effective environmental diversity as closer to the number of batches than the number of episodes.

(c) Remediation rule. If you cannot state why a frame split inflates the metric, re-read §L.D.2 before Part E.

Part E — The running case: the full plan

§L.E.1 — The task population and the sampling decision

The cell must handle 20 tasks across five objects: cube, cup, screwdriver, bolt, sponge. The full cross-product is 100 combinations, and collecting all of them is neither affordable nor necessary.

The sampling rule is stratified by task with object coverage at the margin. Each of the 20 tasks is collected with all five objects, but the episode count per task–object pair is lower than a full collection: 40 episodes per pair gives 4,000 episodes, and a pair that proves difficult receives a second allocation only if the 2.3.3 force budget shows it is at risk.

At 103 attempts per operator-hour (WE-L.B.b) and a 70% gate pass rate, 4,000 accepted episodes require 5,714 attempts, or 55 operator-hours. With five operators that is 11 hours each, comfortably under two working days.

§L.E.2 — The annotation plan and the gates

Instructions. The primary instruction is recorded live. Three paraphrases per episode are assigned by an annotator drawn from the operator pool, so the vocabulary matches deployment. The phrasings vary verb, word order, and reference style.

Success and failure. Verified automatically from the final state, with the failure-mode vocabulary of 3.1.1 §L.E.1: visual grounding, linguistic grounding, motor grounding, and a fourth tag for sensor fault.

Segmentation. Semi-automated from the momentum observer’s contact events (2.3.3) and the velocity profile. Human verification only at boundaries.

Grounding. The 2.1.2 estimator produces 6D poses for every keyframe; human verification on a 5% sample stratified by estimator confidence, with the low-confidence frames over-represented.

The collection gates. Before an episode enters the dataset:

• success verified from the final state;
• mean absolute jerk below the pilot best-quartile threshold;
• duration within $\pm 40$ of the task median;
• peak contact force below 20 N;
• instruction timestamp before the first action;
• all six observation fields present at every step.

An episode failing any gate becomes a negative example with its failure tag, unless it also fails a sensor-consistency check, in which case it is discarded.

§L.E.3 — The splits and the evaluation protocol

The split is 80/10/10 by episode, stratified by task, object, operator, and collection batch. With 4,000 episodes the test set is 400 episodes, giving a standard error at $p = 0.9$ of $\sqrt{0.09/400} = 0.015$, a 95% interval of $\pm 0.03$. That is enough to distinguish a 90% policy from an 85% one but not from an 88% one; if the deployment’s acceptance criterion needs the finer distinction, the test set must be doubled.

Three held-out axes are reported separately: seen task with unseen object, seen object with unseen phrasing, and unseen task. The three correspond to the three groundings of 3.1.1, and separating them is what makes the evaluation diagnostic rather than comparative.

Self-explanation 1. Why report the three held-out axes separately rather than one aggregate success rate?

Answer. Because an aggregate hides which grounding failed; a model that scores 85% overall may be at 95% on seen tasks and 40% on unseen phrasings, and the fix differs entirely between those cases.

Worked-example ladder.

WE-L.E.a (fully worked). Compute the total plan: episodes, attempts, operator-hours, annotation-hours, and the test-set standard error.

Solution. Episodes 4,000; attempts 5,714; operator-hours 55; annotation at 3.1 minutes per episode (WE-L.C.b) gives 207 hours; test set 400 episodes; standard error 0.015.

WE-L.E.b (completion). Reduce to 10 tasks with the same object coverage and recompute the plan.

Completion. 2,000 episodes, 2,857 attempts, 28 operator-hours, 103 annotation-hours, test set 200, standard error 0.021. The completion shows that halving the tasks halves the cost but widens the test interval by 40%, which may make the evaluation non-discriminating.

WE-L.E.c (faded). The gate pass rate falls from 70% to 55% because operators are rushed. Recompute the attempts and state the corrective action.

Faded step. Attempts $= 4{,}000/0.55 = 7{,}273$; operator-hours $= 70.6$. Completion. The corrective action is not to collect more but to slow the operators, because a rushed attempt that fails the smoothness gate costs the same as a good one and returns nothing.

WE-L.E.d (contrastive near-miss). A team collects the full 100-task–object cross-product with 10 episodes each, giving 1,000 episodes. The test set is 100 episodes and the model scores 88%. On deployment it fails on the cup. Why it fails: the stratified coverage is too thin to expose the cup’s specific failure (it is the only object whose grasp requires the 2.3.1 force closure at a curved surface), and the aggregate metric hid it. The fix: allocate episodes by difficulty, not uniformly, and report per-object results.

Comparative table: the plan at three scales.

Design principle P37.5. Allocate collection by difficulty and by the test set’s required resolution, not uniformly across the task grid.

Consolidation checkpoint 5

(a) Three capabilities. You can compute the full collection and annotation plan. You can set the gate thresholds from pilot data. You can report the three held-out axes separately.

(b) Transfer probe. The cup’s grasp requires a curved-surface force closure that the cube’s does not. What does this change in the plan? Answer: the cup receives a second episode allocation and its own test stratum; uniform allocation would under-cover the only object whose grasp model is qualitatively different.

(c) Remediation rule. If you cannot compute the test-set standard error, redo WE-L.D.b before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: single-batch test set, concentrated paraphrases, and correlated episodes.

The lab

Task. Build a synthetic episode dataset with the six-field schema, demonstrate that a frame-level split inflates the test metric while an episode-level split does not, and run the collection and annotation cost model for the running case’s plan.

Conditions. Python 3.10+, NumPy, scikit-learn for the nearest-neighbour baseline. Seed all RNGs. The synthetic data has 50 episodes of 50 steps, each with a task-dependent latent offset so that task structure is learnable and neighbour structure is exploitable.

Controls. (i) Nearest-neighbour predictor on a frame split; (ii) the same predictor on an episode split; (iii) the cost model at the three scales of §L.E.3; (iv) a deduplication pass that collapses near-identical episodes.

Metrics.

Pre-registered hypotheses.

H1. The nearest-neighbour predictor on a frame-level split achieves test error below 0.05 in normalized action units, because test frames are neighbours of training frames.

H2. The same predictor on an episode-level split has test error at least five times larger, because the test trajectories are unseen.

H3. Deduplication at a threshold derived from the timing-noise scale removes a substantial fraction of a deliberately duplicated dataset while leaving the episode-split error essentially unchanged.

H4. The cost model reproduces the §L.E.3 figures within 5%.

Break-it sequence.

1. Split by frame silently. The metric looks excellent. Silent: the pipeline runs and the number is plausible.
2. Concentrate one batch in the test set silently. The test error rises without any code error. Silent: the split is technically by episode.
3. Assign all paraphrases to one annotator silently. Phrasing diversity collapses. Silent: the label count is unchanged.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The split granularity is a named parameter, not a default. (2) Both metrics are reported side by side, so the leakage is visible as a ratio. (3) Deduplication is a function with a justified threshold, not a hard-coded constant. (4) The break-it results are a table. (5) The cost model is a function of episode count, gate pass rate, and label types.

A runnable core, seeded and annotated:

# lab_3_1_3.py — schema, leakage demonstration, cost model
import numpy as np
from sklearn.neighbors import KNeighborsRegressor
np.random.seed(0)

$N_EP$, EP_LEN, $N_TASK$ = 50, 50, 10
episodes = []
for e in $range(N_EP$):
    task = e % $N_TASK$                                  # balanced task assignment
    offset = np.array([task * 0.4, -task * 0.3])       # task-level latent
    steps = np.arange(EP_LEN)[:, None] / EP_LEN        # normalized time
    traj = offset + np.hstack([steps, 1 - steps])      # smooth task trajectory
    obs = traj + 0.02 * np.random.randn(EP_LEN, 2)     # observation noise
    act = np.diff(traj, axis=0, prepend=traj[:1])      # action is the delta
    episodes.append({"ep": e, "task": task,
                     "obs": obs, "act": act})

X = np.vstack([ep["obs"] for ep in episodes])          # all frames
Y = np.vstack([ep["act"] for ep in episodes])
ep_id = np.concatenate([[ep["ep"]] * EP_LEN for ep in episodes])

# Frame-level split: random over frames (WRONG)
rng = np.random.default_rng(0)
perm = rng.permutation(len(X))
$n_test$ = len(X) // 5
tr_f, te_f = $perm[n_test$:], perm[:$n_test$]
nn_f = $KNeighborsRegressor(n_neighbors=1$).fit(X[tr_f], Y[tr_f])
err_frame = np.mean(np.linalg.norm(nn_f.predict(X[te_f]) - Y[te_f], axis=1))

# Episode-level split: contiguous whole episodes (RIGHT)
test_eps = set(range(0, $N_EP$, 5))                      # every fifth episode
mask_te = np.isin(ep_id, list(test_eps))
tr_e, te_e = np.where(~mask_te)[0], np.where(mask_te)[0]
nn_e = $KNeighborsRegressor(n_neighbors=1$).fit(X[tr_e], Y[tr_e])
err_ep = np.mean(np.linalg.norm(nn_e.predict(X[te_e]) - Y[te_e], axis=1))

print(f"frame-split error  = {err_frame:.4f}")
print(f"episode-split error= {err_ep:.4f}")
print(f"leakage ratio      = {err_ep / err_frame:.1f}")

# Cost model
def $plan(n_tasks$, $n_objects$, per_pair, gate_pass, cycle_s,
         annot_min_per_ep, rate):
    eps = $n_tasks$ * $n_objects$ * per_pair
    attempts = eps / gate_pass
    op_hours = attempts * cycle_s / 3600.0
    annot_hours = eps * annot_min_per_ep / 60.0
    test_n = int(0.1 * eps)
    se = np.sqrt(0.9 * 0.1 / max(test_n, 1))
    return dict(episodes=eps, attempts=attempts, op_hours=op_hours,
                annot_hours=annot_hours, test_se=se)

for tasks in (20, 10):
    print(tasks, plan(tasks, 5, 40, 0.70, 35, 3.1, 25))

Lines 5–12 build the synthetic episodes with a task-level latent offset, so a nearest-neighbour predictor can succeed on seen tasks and fail on unseen ones. Lines 22–30 are the frame split, which mixes frames from the same trajectory across train and test; lines 32–38 are the episode split. The printed leakage ratio is the experiment’s core result: it should be well above 5. Lines 44–58 are the cost model of §L.E.1 and §L.E.3, parameterized by gate pass rate, cycle time, and annotation time per episode.

Cheat sheet

Exercises with solutions

E 3.1.3.1. Write the six schema field groups and name one downstream consumer for each.

S 3.1.3.1. Observations (encoder block), actions (decoder loss), instruction (language encoder), metadata (splits and balancing), outcome (filtering and evaluation), derived labels (auxiliary losses). The lesson: every field must have a consumer, or it is a cost without benefit.

E 3.1.3.2 (transfer). An episode is 300 steps at 10 Hz with 640×480 RGB at 1 byte per channel and 16-bit depth. Compute the uncompressed episode size and the size after 20:1 RGB compression and 2:1 depth.

S 3.1.3.2. Per step: RGB 921,600 B, depth 614,400 B, total 1,536,000 B. Per episode 461 MB uncompressed. Compressed: RGB 46,080 B, depth 307,200 B, total 353,280 B per step, 106 MB per episode. The lesson: compression is a 4.3× factor and should be decided with the schema.

E 3.1.3.3. An operator completes an episode in 30 s including reset; the gate passes 65%. Compute attempts and operator-hours for 3,000 accepted episodes.

S 3.1.3.3. Attempts $= 3{,}000/0.65 = 4{,}615$; at 30 s each that is 38.5 hours. The lesson: the gate pass rate is a direct cost multiplier and belongs in the plan, not in the retrospective.

E 3.1.3.4 (diagnostic). A dataset has 4 paraphrases per task but all assigned by one annotator. The model fails on the operators’ phrasing. Name the cause and the fix.

S 3.1.3.4. The annotator’s private vocabulary is its own distribution, and the model learned it. The fix is to draw paraphrases from the deployment operator pool or to instruct the annotator to match their register. The lesson: annotation is a data-generating process with its own distribution.

E 3.1.3.5. Demonstrate in two sentences why a frame-level split inflates the metric.

S 3.1.3.5. Test frames have near-identical neighbours in the training set because adjacent frames differ by a few pixels, so a model with any local smoothness interpolates them. The metric therefore measures continuity rather than task generalization. The lesson: the split granularity determines what the metric means.

E 3.1.3.6 (transfer). A dataset has 400 episodes; the test split is 40. What success-rate difference can be resolved at 95% confidence near $p = 0.9$?

S 3.1.3.6. $\text{SE} = \sqrt{0.09/40} = 0.047$, so the 95% interval is about $\pm 0.09$; differences below roughly 13 percentage points are not resolvable. The lesson: the test split’s size sets the evaluation’s resolution, and the resolution requirement sets the collection volume.

E 3.1.3.7 (diagnostic). A policy trained on 4,000 episodes hesitates at grasp. Name two data causes and the discriminating test.

S 3.1.3.7. (1) Demonstrations include pauses, so the duration gate was absent or too loose; test by inspecting the duration distribution and the pause count. (2) The loss is per-step without chunking, so the two modes average; test by switching to chunked prediction and observing whether hesitation falls. The lesson: hesitation is usually a demonstration-quality or a target-representation problem, not a capacity problem.

E 3.1.3.8. Compute the annotation cost for 2,500 episodes at 4 minutes per episode and 30 currency units per hour, then recompute with grounding semi-automated to 1 minute.

S 3.1.3.8. At 4 min: $2{,}500 \times 4/60 = 166.7$ hours, 5,000 units. At 1 min: 41.7 hours, 1,250 units. The lesson: grounding dominates annotation cost and is the label most amenable to automation.

E 3.1.3.9 (transfer). A team collects 100 task–object pairs with 10 episodes each and reports 88% aggregate success. Deployment fails on one object. What is the diagnosis and the plan change?

S 3.1.3.9. Uniform allocation under-covers the difficult object, and the aggregate metric hides the per-object failure. The plan change is difficulty-weighted allocation and per-object reporting. The lesson: aggregate metrics conceal the failures that matter; stratify the report.

Exit test

You are ready for Lesson 3.1.4 when you can, without notes:

1. Write the six schema field groups with types, rates, and consumers.
2. Compute episode storage from modality, resolution, and rate.
3. Name the three collection families and their distribution properties.
4. Distinguish a successful episode from a good demonstration.
5. Write the five quality-gate checks.
6. State the non-stationarity problem and how chunking addresses it.
7. State why corrective data is required and how it is collected.
8. Write the four label types with their verification.
9. Compute annotation cost and identify the dominant term.
10. State the paraphrase minimum and the within-episode distribution requirement.
11. Demonstrate frame-split leakage arithmetically.
12. Compute test-set standard error and size the split from it.
13. State the deduplication, filtering, and balancing rules.
14. Design a stratified collection plan with gates, annotation budget, and splits.
15. Report the three held-out axes separately and explain why.

Cross-references. 3.1.1 supplies effective sample size and instruction diversity restated in §L.D.1 and §L.C.2. 3.1.2 supplies the web-data ablation and the gradient-norm mixing rule restated in §L.B.1 and §L.D.3. 2.2.4 supplies the deploy–collect–fine-tune loop restated in §L.B.3 and §L.D.3. 2.3.4 supplies the state-distribution argument restated in §L.B.3. 2.1.2 supplies the pose estimator used to semi-automate grounding labels in §L.C.1. Lesson 3.1.4 treats fine-tuning as the operation performed on the dataset this lesson specifies.

Ready. The next lesson to write is 3.1.4 — VLA Fine-Tuning: the pipeline and the pitfalls. Say “next” to generate it.