X1.2.1

Lesson 1.2.1 — Simulation Environments and the Gym Interface: Physics Engines, Observation/Action Spaces, and the Sim-to-Real Gap

Week 7 · Prerequisites: 1.1.1, 1.1.2, 1.1.3, 1.1.4, 0.1.1, 0.1.2.

Prior results used. This lesson uses 1.1.1’s FK and Jacobian (for the observation’s construction), 1.1.2’s IK and singularity analysis (for the action space’s design), 1.1.3’s PID and computed torque (for the action space’s choice — torque vs. position vs. velocity), 1.1.4’s pipeline and health checks (for the simulation’s reward and termination), and 0.1.1’s tensor shapes and broadcasting (for the observation and action spaces’ tensor forms). Each is restated at the point of use.

Hands forward. This lesson hands forward one capability: the learner can select a simulator for a given learning task, wrap a rigid-body arm in the Gymnasium interface, define the observation and action spaces that match the controller’s interface from 1.1.3, configure the physics engine’s timestep and contact model, and diagnose the four simulation-level bugs (wrong action space, missing termination, observation normalization mismatch, and contact-model mismatch) from the learning’s symptom — the capability that every RL and imitation learning lesson in Module 1.2 and every learned-controller deployment in Phases 3–4 assumes.

Running case. The same 6-DOF MuJoCo arm from 1.1.1–1.1.4, now wrapped in the Gymnasium interface as ArmEnv: the observation is [q, q̇, p_ee, p_target] (the joint positions, the joint velocities, the end-effector position, and the target position, 18 numbers), the action is the joint torques τ ∈ ℝ⁶ (from 1.1.3’s computed torque), and the reward is the negative distance −‖p_ee − p_target‖ − 0.01‖τ‖² (a sparse-ish reward with a torque penalty). The lesson’s central result is that the Gym interface is the contract between the arm and the learning algorithm, and the action space’s choice (torque vs. position vs. velocity) determines the learnability of the task — a torque-controlled arm learns slowly (the policy must discover the dynamics), while a position-controlled arm learns quickly (the policy commands the end-effector’s position, and the inner PID loop from 1.1.3 handles the dynamics).

§L.0 — Lesson map, objectives, and misconceptions

§L.0.1 — Lesson map and word budget

§L.0.2 — Objectives

By the end of this lesson you will be able to do eight things:

1. State the four major simulators (MuJoCo, Isaac Lab, PyBullet, Gazebo), their strengths (contact fidelity, GPU parallelism, ROS integration, license), and the task each is best suited for.
2. Implement the Gymnasium interface reset() -> obs and step(action) -> (obs, reward, terminated, truncated, info) for the running case, with the observation and action spaces correctly defined.
3. Distinguish terminated (the task ended — the episode is done) from truncated (the time limit hit — the episode is cut off), and state the consequence for the value function’s bootstrapping.
4. Design the action space — torque (ℝ⁶), position (ℝ⁶), velocity (ℝ⁶), or end-effector pose (ℝ⁶ from the Jacobian) — and state the learnability trade-off: torque learns the dynamics (slowly), position learns the task (quickly, with the inner PID loop).
5. Configure the physics engine’s timestep (dt ≈ 1/1000 of the controller’s period), the solver’s iterations (the iterations parameter), and the contact model (solref, solimp, the friction cone), and state the consequence of each.
6. Design the observation to include the joint positions, velocities, end-effector position, and target position, and normalize each to a standard range (mean 0, std 1) for the learning.
7. Diagnose the four simulation-level bugs — wrong action space (the policy’s output doesn’t match the arm’s input), missing terminated (the episode continues past success), observation normalization mismatch (training and deployment scales differ), and contact model mismatch (the simulated contact differs from the real one) — from their learning symptoms.
8. Produce a documented ArmEnv that wraps the running case, with the observation and action spaces’ __repr__, the termination and truncation’s logic, and the sim-to-real gap’s first mitigation (the domain randomization of the arm’s mass and the target’s position).

§L.0.3 — Misconception inventory

Each misconception is displaced in the section named, and each is tagged in the concept check that follows.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

This is the seventeenth lesson. The prompt requires drawing from lessons two to five steps back; 1.1.4 is one step back and is excluded. The warm-up draws from 1.1.3 (two steps back), 1.1.2 (three), 1.1.1 (four), and 0.3.4 (five).

Warm-up 1 (from 1.1.3, the computed torque). State the computed-torque law and the role of the feedforward vs. the feedback. Answer: τ = M(q)(q̈_d + K_d ė + K_p e) + C q̇ + g, where the feedforward (M q̈_d + C q̇ + g) cancels the dynamics and the feedback (M(K_d ė + K_p e)) corrects the error. How this is used: §L.3’s action space’s choice — the torque action is the computed-torque’s input, and the position action is the q_d input — and the two have different learnability (the torque policy must learn the feedforward, the position policy inherits the inner PID’s feedforward).

Warm-up 2 (from 1.1.2, the IK’s branch and singularity). State the IK’s branch choice and its singularity. Answer: the IK has multiple solutions (the branches), and the closest to q_current is the continuity’s choice; the singularities are the σ_min(J) = 0 configurations, where the pseudoinverse blows up. How this is used: §L.3’s end-effector action space uses the Jacobian to map the end-effector’s velocity to the joint velocities, and the singularities are where the mapping is ill-conditioned — the policy must avoid the singularities, or the DLS (from 0.3.4) must regularize the mapping.

Warm-up 3 (from 1.1.1, the FK’s q = 0 test). State the FK’s two tests and what each catches. Answer: the q = 0 test catches the a_i errors (the reach is the sum of the link lengths); the non-zero test catches the α_i and sign errors. How this is used: §L.2’s observation’s p_ee is the FK’s output, and the observation’s correctness depends on the FK’s correctness — the q = 0 test and the non-zero test are the observation’s validation.

Warm-up 4 (from 0.3.4, the Mahalanobis gate). State the Mahalanobis gate and its role in a filter. Answer: d² = νᵀS⁻¹ν > χ²_{1−α}(m) rejects the outliers, with a false-rejection rate of α. How this is used: §L.2’s reward function can use a gated distance (the reward is zero if the distance is beyond a threshold), which is the Mahalanobis gate applied to the task’s success — the reward’s shape is the gate’s continuous version.

Part A — The Simulator Landscape

§L.1 — Objects: the four simulators and their trade-offs

A physics simulator for robotics is a software package that computes the dynamics of a rigid-body system (or a soft-body system, or a fluid) given the control inputs and the state, and that renders the scene (for the vision-based policies) and exposes the state (for the proprioceptive policies). The four major simulators for robot learning are:

1. MuJoCo (Multi-Joint dynamics with Contact) — the standard for contact-rich manipulation and legged locomotion. Its strengths are its convex-contact solver (fast and accurate for the typical robot’s contacts), its stability at small timesteps, and its well-tuned defaults. Its weaknesses are its license (free for research, commercial for some uses), its CPU-only execution (single-threaded per scene, slow for large-scale RL), and its limited sensor models (no photorealistic rendering; the vision-based policies need a separate renderer). The use case: the running case’s arm, the contact-rich grasping, the locomotion benchmarks (the HalfCheetah, the Ant, the Humanoid).

2. Isaac Lab (NVIDIA’s, built on Isaac Sim and PhysX) — the standard for large-scale RL, with GPU-parallel simulation (thousands of environments on one GPU) and photorealistic rendering (for the vision-based policies). Its strengths are the parallelism (a million samples per hour), the rendering (the RGB-D cameras’ simulation), and the integration with the PyTorch ecosystem. Its weaknesses are its hardware requirement (a high-end NVIDIA GPU), its complexity (the USD scene description, the PhysX solver’s tuning), and its license (free for research, commercial for some uses). The use case: the large-scale RL, the vision-language-action models’ training (the Phase 3 VLA), the sim-to-real pipelines.

3. PyBullet — the open-source alternative, with a free license (the zlib license) and broad hardware support (the CPU). Its strengths are the license (free for any use), the simplicity (the Python API), and the broad availability (the pip install). Its weaknesses are its slower contact solver (the sequential impulse), its less stable contacts (the jitter at the small timesteps), and its smaller community (the fewer pre-built models). The use case: the education (the free license), the prototyping (the simple API), the fallback when MuJoCo’s license is an issue.

4. Gazebo — the ROS-integrated simulator, with the ROS 2 bridge and the URDF support. Its strengths are the ROS integration (the deployment’s testbed — the same ROS code runs in Gazebo and on the real robot), the URDF support (the standard robot description), and the sensor models (the IMU, the camera, the LiDAR). Its weaknesses are its slower performance (the CPU-heavy), its less accurate contacts (the ODE solver), and its complexity (the ROS ecosystem). The use case: the deployment’s testing (the same code as the real robot), the mobile robots’ navigation (the SLAM from 2.4.1), the multi-robot systems.

The trade-offs:

The choice depends on the task’s requirements:

• Contact-rich (grasping, legged) → MuJoCo (its contact solver’s accuracy) or Isaac Lab (its parallelism).
• Large-scale (millions of samples) → Isaac Lab (its GPU parallelism).
• Vision-based (the VLA’s training) → Isaac Lab (its photorealistic rendering).
• Deployment’s testing (the ROS code) → Gazebo (its ROS integration).
• License-free (the commercial use) → PyBullet (its zlib license).

§L.1.1 — Result: the simulators are not interchangeable — the contact model and the solver differ, and a policy trained in one does not transfer to another without retraining

Claim. For a contact-rich task, the policy trained in MuJoCo’s convex-contact solver does not achieve the same performance in Isaac Lab’s PhysX solver or in PyBullet’s sequential-impulse solver, because the contact models differ (the stiffness, the friction cone, the restitution), and the policy memorizes the simulator’s specific contact behavior. The sim-to-sim gap is the precursor of the sim-to-real gap, and the transfer requires either (i) the retraining in the target simulator, or (ii) the domain randomization of the contact parameters (the solref, the solimp, the friction) across simulators.

Proof sketch. The contact model’s parameters (the MuJoCo solref’s stiffness/damping, the PhysX’s material’s restitution/friction, the Bullet’s contact processing threshold) determine the contact force given the penetration and the velocity, and the policy that’s optimal for one model’s contact behavior is suboptimal for another’s. The contact-rich tasks (grasping, walking) are the most sensitive (the policy’s success depends on the contact’s details), while the free-space tasks (reaching) are the least (the contact is rare).

Corollary. The sim-to-sim gap is measurable (the same policy, two simulators, the performance difference), and it is the lower bound on the sim-to-real gap — the real world’s variation is larger than the simulators’ differences, and the sim-to-sim gap is the first test of the policy’s robustness.

Corollary. The domain randomization’s contact parameters (the solref, the solimp, the friction) reduce the sim-to-sim gap by training the policy across the parameter range — the policy generalizes to the unseen simulator’s behavior, and the generalization is the sim-to-real transfer’s first mechanism.

§L.1.2 — Worked-example ladder

WE-L.1.a — Fully worked: the running case in MuJoCo vs. PyBullet

Input. The running case’s 6-DOF arm, simulated in MuJoCo (the default) and in PyBullet (the free alternative), with the same control law (the computed torque from 1.1.3).

Trace (MuJoCo).

1. The arm’s model is the MJCF file (MuJoCo’s XML format) with the link inertias, the joint limits, and the contact parameters.
2. The computed torque at each step, the contact solver’s iterations (100), the timestep (1 ms).
3. The arm tracks the trajectory with a tracking error of ~1 mm (the MuJoCo solver’s accuracy).

Trace (PyBullet).

4. The arm’s model is the URDF file (the ROS standard) with the same inertias and the same contact parameters.
5. The computed torque at each step, the Bullet solver’s iterations (50), the timestep (1 ms).
6. The arm tracks the trajectory with a tracking error of ~3 mm (the Bullet solver’s less accurate contact), and the contact with the table shows jitter (the sequential impulse’s vibration).

Output and interpretation. The same control law gives different tracking errors in the two simulators (1 mm vs. 3 mm), and the difference is the contact solver’s accuracy (MuJoCo’s convex vs. Bullet’s sequential impulse). The tracking is good in both (the 3 mm is acceptable for the free-space task), but the contact-rich task (a grasp) would show a larger difference — the MuJoCo grasp would succeed, and the PyBullet grasp would slip (the friction cone’s approximation).

Self-explanation 1. Before reading on: the tracking error is 1 mm in MuJoCo and 3 mm in PyBullet — a 3× difference. Is this a problem for the free-space reaching task?

Answer. For the free-space reaching, the 3 mm is acceptable — the task’s tolerance is centimeters (the gripper’s finger’s width is ~5 cm), and the 3 mm is well within. For a contact-rich task (a tight peg-in-hole, a precision assembly), the 3 mm is catastrophic — the peg would miss the hole. The simulator’s choice depends on the task’s tolerance, and the MuJoCo choice is safer for the precision tasks.

Self-explanation 2. Before reading on: the same URDF is used in both simulators (the arm’s links, inertias, joints). Why does the same URDF give different behavior?

Answer. The URDF specifies the model’s geometry and inertia, but not the contact’s parameters (the stiffness, the friction, the restitution) — those are simulator-specific. The MuJoCo MJCF adds the solref/solimp (the contact’s stiffness/damping), and the PyBullet URDF adds the contact properties (the friction), and the two are different models. The same URDF is the structure, and the contact model is the behavior — the structure is shared, and the behavior is not.

WE-L.1.b — Completion problem

Input. The same arm, but the task is a large-scale RL (the millions of samples), and the simulator is MuJoCo (CPU-only).

Trace (final 25% as prompts).

1. The sample rate in MuJoCo on a single CPU: ~1,000 steps/s. ⟨your turn: is this fast enough for the millions of samples?⟩
2. The time to one million samples: 1,000,000 / 1,000 = 1,000 s ≈ 16.7 min. ⟨your turn⟩
3. The time to one hundred million samples: ~28 hours. ⟨your turn⟩
4. The Isaac Lab’s sample rate on a single GPU: ~100,000 steps/s (100× MuJoCo). ⟨your turn: what is the time for the 100 million samples?⟩

Answers.

1. 1,000 steps/s — slow for the large-scale RL (the typical RL needs 10–100 million samples).
2. 16.7 min for 1M samples.
3. ~28 hours for 100M samples — too slow for the iterative RL (the many runs’ wall-clock time).
4. 100,000 steps/s → 100M / 100,000 = 1,000 s ≈ 16.7 min — 100× faster, and the iterative RL is feasible.

The lesson. The MuJoCo’s CPU-only speed is fine for the small tasks (1M samples) and too slow for the large tasks (100M samples). The Isaac Lab’s GPU parallelism is the choice for the large scale, and the MuJoCo is the choice for the contact-rich small tasks.

WE-L.1.c — Faded: the contact parameters’ transfer

Setup. A grasping policy is trained in MuJoCo with the default contact parameters (the solref = (0.02, 1), the friction 1.0).

Question. Does the policy transfer to a real robot with rubber fingers (the friction 1.5) and a metal object (the friction 0.4)?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The transfer is unlikely — the simulated friction (1.0) is between the real finger’s (1.5) and the object’s (0.4), and the effective friction (the minimum of the two) is 0.4 on the real, 1.0 in the sim. The simulated grasp is stronger than the real, and the policy’s force (tuned for the simulated friction) is insufficient on the real — the object slips. The fix is the domain randomization of the friction (the 0.3–1.5 range), which trains the policy for the real worst case.

The lesson. The contact parameters (the friction, the stiffness) are the sim-to-real gap’s primary source for the contact-rich tasks, and the domain randomization is the mitigation. The policy trained with the fixed friction fails on the real robot, and the randomized friction generalizes.

WE-L.1.d — Contrastive near-miss: the same policy in two simulators

Correct solution. The policy is trained and evaluated in the same simulator, or the domain randomization covers both simulators’ ranges.

Near-miss. The policy is trained in MuJoCo and evaluated in PyBullet without the randomization, and the performance is poor (the sim-to-sim gap).

Why the near-miss produces a plausible but incorrect number. The policy’s training reward is high (the MuJoCo’s convex contact is the memorized behavior), and the evaluation in PyBullet’s sequential-impulse contact gives a lower reward — the performance is good in the training and poor in the evaluation, and the learner who reports the training reward overestimates the policy’s real performance. The sim-to-sim gap is the first sign of the sim-to-real gap, and the evaluation in the second simulator is the necessary test.

The discrepancy, stated. The training reward (MuJoCo) and the evaluation reward (PyBullet) are different by the contact solver’s difference, and the gap is the sim-to-sim gap. The evaluation in the second simulator is the cheap proxy for the real evaluation, and the policy that transfers to the second simulator is more likely to transfer to the real.

Self-explanation 3. Before reading on: in the near-miss, if the policy is reaching (free-space) instead of grasping (contact-rich), is the sim-to-sim gap smaller or larger?

Answer. Smaller — the free-space task has few contacts (the arm rarely touches the object), and the contact solver’s differences are irrelevant. The reaching policy transfers across simulators easily (the gap is ~1 mm), and the grasping policy does not (the gap is ~1 cm or more). The sim-to-sim gap’s magnitude depends on the task’s contact-richness.

Part B — The Gymnasium Interface

§L.2 — Objects: reset, step, and the five return values

The Gymnasium (the successor to OpenAI Gym) interface is the standard for RL environments, and its contract is:

• env.reset() -> (obs, info) — resets the environment to an initial state, returns the observation obs and an info dictionary.
• env.step(action) -> (obs, reward, terminated, truncated, info) — applies the action, advances the simulation by one step, and returns:
  • obs — the new observation.
  • reward — the scalar reward for the transition.
  • terminated — whether the task ended (the terminal state, e.g., the target reached, the collision, the joint limits).
  • truncated — whether the time limit was hit (the episode is cut off, e.g., the max_steps reached).
  • info — the diagnostic dictionary (the extra information, the not-for-learning data).

The distinction between terminated and truncated:

• terminated = True — the episode is over because the task is done (the success, the failure), and the value function should not bootstrap (the next state’s value is not the future reward, because there is no future). The Q-target is r (the terminal reward).
• truncated = True — the episode is over because the time limit is hit, and the value function should bootstrap (the next state’s value is the future reward, because the state is not terminal — the episode was cut off, not ended). The Q-target is r + γ V(s').

The ArmEnv for the running case:

• obs = [q (6), q̇ (6), p_ee (3), p_target (3)] — 18 numbers.
• action = τ ∈ ℝ⁶ (the joint torques) — 6 numbers.
• reward = −‖p_ee − p_target‖ − 0.01‖τ‖² — the negative distance with the torque penalty.
• terminated = ‖p_ee − p_target‖ < 0.01 (the success) or the collision (the failure).
• truncated = step_count >= max_steps (the time limit, e.g., max_steps = 500).
• info = {'step': step_count, 'distance': ‖p_ee − p_target‖, ...}.

§L.2.1 — Result: the terminated/truncated distinction biases the value function when conflated, and the observation’s order and scale determine the learnability

Claim. For a conflated terminated/truncated (both treated as terminal), the value function’s target is r for the truncated transitions, but the correct target is r + γ V(s'). The bias is γ V(s'), which accumulates over the training, and the value function underestimates the long-horizon returns. For an observation with the wrong order or scale, the policy’s gradient has a different direction, and the learning is slower (or diverges).

Proof sketch. The Q-learning (or actor-critic) update’s target is y = r for the terminal s' (the V(s') = 0 for the terminal state) and y = r + γ V(s') for the non-terminal s' (the bootstrap). If the truncated s' is treated as terminal, the target is r, which is smaller than the correct r + γ V(s') by γ V(s'), and the value function is trained to the smaller value. The observation’s scale: a feature with a large range (e.g., q in [−π, π]) and a small range (e.g., q̇ in [−0.01, 0.01]) gives the neural network a large input variance for the first and a small for the second, and the gradient is dominated by the large feature — the normalization (the mean 0, the std 1) equalizes the features.

Corollary. The ArmEnv’s terminated includes the success (the ‖p_ee − p_target‖ < 0.01) and the failure (the collision), and the truncated is the time limit. The success’s terminated is correct (the task is done, no bootstrap), and the time limit’s truncated is correct (the task is not done, bootstrap). The two are separate, and the learner must not conflate them.

Corollary. The observation’s q and q̇ are normalized to the standard range (the q’s [−π, π] scaled to [−1, 1], the q̇’s [−2, 2] scaled to [−1, 1]), and the p_ee and p_target are normalized to the arm’s reach (the [−1.2, 1.2] scaled to [−1, 1]). The normalization’s parameters (the mean, the std) must be the same in the training and the deployment, and the deployment’s mismatch is the silent bug.

§L.2.2 — Worked-example ladder

WE-L.2.a — Fully worked: the ArmEnv’s reset and step

Input. The running case’s arm, the target at p_target = [0.5, 0.2, 0.3].

Trace (reset).

1. The arm’s state is reset to a random configuration q ~ Uniform[−0.5, 0.5]⁶, the velocity q̇ = 0.
2. The observation is obs = [q, q̇, p_ee(q), p_target], normalized.
3. The info is {'step': 0}.

Trace (step(action=τ)).

4. The simulation advances by dt = 0.001 with the torque τ.
5. The new state q', q̇' and the new p_ee'.
6. The reward is r = −‖p_ee' − p_target‖ − 0.01‖τ‖².
7. The terminated is ‖p_ee' − p_target‖ < 0.01 (the success) or the collision (the failure).
8. The truncated is step_count + 1 >= 500.
9. The obs’ is [q', q̇', p_ee', p_target], normalized.
10. The info is {'step': step_count + 1, 'distance': ‖p_ee' − p_target‖}.

Output and interpretation. The reset gives a random configuration (the exploration), and the step advances the simulation by one control period. The terminated is the task’s completion (the success or the failure), and the truncated is the time limit. The info’s distance is the diagnostic (the learning uses the reward, not the info), and the step is the time.

Self-explanation 4. Before reading on: the reset uses a random q in [−0.5, 0.5]⁶. Why the random q, and why the range?

Answer. The random q is the exploration — the policy must learn the reaching from many initial configurations, and the random q gives the diversity. The range [−0.5, 0.5] is small (the arm is near its home), and the small range reduces the difficulty (the arm is near the target’s reachable workspace) and increases the learning’s speed. The wider range [−π, π] would be more general (the arm can start anywhere) and slower (the larger exploration). The range is the task’s design choice, and the small range is the easy case.

Self-explanation 5. Before reading on: in step 7, the terminated includes the collision. Is the collision a success or a failure, and how does the reward handle it?

Answer. The collision is a failure (the arm hit the table or an obstacle), and the reward should be negative (a penalty). The reward r = −‖p_ee − p_target‖ − 0.01‖τ‖² is negative at the collision (the distance is large), and the terminated is true (the episode ends). The combination (a negative reward and a terminal state) discourages the collision — the policy learns to avoid it. The collision’s detection is the simulator’s job (the contact with the table), and the ArmEnv exposes it as the terminated.

WE-L.2.b — Completion problem

Input. Same ArmEnv, but the terminated is omitted (always False), and the truncated is the time limit.

Trace (final 25% as prompts).

1. The success (‖p_ee − p_target‖ < 0.01) does not terminate — the episode continues. ⟨your turn: does the arm stay at the target, or move away?⟩
2. The value function’s target for the successful state is r + γ V(s'), not r. ⟨your turn: what is the bias?⟩
3. The learning: the policy learns to reach the target and then move away (the reward after the success is negative, and the discount makes the future less important). ⟨your turn: what is the symptom?⟩

Answers.

1. The arm stays near the target (the policy has no reason to move away), but the episode continues (the step count grows), and the reward is accumulated over the extra steps. The value function is trained with the extra rewards, and the successful state’s value is inflated (the future rewards are counted).
2. The bias is γ V(s') − γ V(s'_terminal) = γ V(s') (the non-zero bootstrap), which overestimates the successful state’s value.
3. The symptom is the policy’s hovering (the arm reaches the target and stays, but the episode continues until the time limit), and the training is slower (the extra steps’ rewards dilute the success’s signal). The fix is the terminated (the success ends the episode).

The lesson. The terminated is the task’s completion, and its omission makes the episode continue past the success — the reward is diluted, and the value function is biased. The two flags are separate and both needed.

WE-L.2.c — Faded: the observation’s scale

Setup. The ArmEnv with the unnormalized observation: q in [−π, π] (the large range) and q̇ in [−2, 2] (the small range).

Question. How does the unnormalized observation affect the learning?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The neural network’s first layer’s weights are initialized with the same distribution, and the large q’s variance (π²/3 ≈ 3.29) is larger than the small q̇’s (4/3 ≈ 1.33), so the gradient is dominated by the q’s contribution. The network learns the q’s effect quickly and the q̇’s slowly — the policy ignores the velocity’s information, and the damping (the velocity’s role) is missing. The symptom is the oscillatory reaching (the arm overshoots because the policy doesn’t see the velocity), and the fix is the normalization (the q/π and the q̇/2, both in [−1, 1]).

The lesson. The unnormalized observation gives the larger feature more influence, and the policy ignores the smaller feature. The normalization equalizes the features, and the balanced learning is the result. The normalization’s parameters must be recorded and applied in the deployment, or the deployment’s scale mismatch is the silent bug.

WE-L.2.d — Contrastive near-miss: the truncated as terminated

Correct solution. The truncated is the time limit (step_count >= max_steps), and the bootstrap is r + γ V(s') (the next state’s value).

Near-miss. The truncated is treated as terminated (the bootstrap is r), and the value function is trained with the zero target at the time limit.

Why the near-miss produces a plausible but incorrect number. The time limit’s state is not terminal (the arm is still moving, the task is not done), and the correct target is the bootstrapped r + γ V(s'). The near-miss’s r underestimates the value, and the value function’s long-horizon returns are biased low. The symptom is the policy’s short-sightedness (the slow tasks, the long horizons), and the fix is the separate truncated (the bootstrapped target).

The discrepancy, stated. The terminated’s target r and the truncated’s target r + γ V(s') differ by γ V(s'), which is non-zero when the state is not terminal. The conflation makes the truncated transitions look like the terminal ones, and the value function loses the future rewards’ contribution. The separate flags restore the correct target.

Self-explanation 6. Before reading on: in the near-miss, if the episode’s time limit is short (e.g., max_steps = 10), is the bias larger or smaller?

Answer. Larger — the short time limit truncates the episodes often, and the truncated transitions are a larger fraction of the total. The bias’s contribution is proportional to the fraction, and the short limit’s large fraction amplifies the bias. The long time limit’s small fraction reduces the bias (the few truncated episodes’ effect is small), but the bias is still there — the separate flags are needed regardless of the limit’s length.

Part C — The Observation and Action Spaces

§L.3 — Objects: the spaces’ definitions and the action space’s choice

The observation space obs_space and the action space action_space are the Gym’s metadata for the environment’s interfaces. The observation space is typically a Box (the continuous range) or a Dict (the structured observation), and the action space is a Box (the continuous torques, positions, or velocities) or a Discrete (the discrete actions, the gripper’s open/close).

The action space’s choice is the learnability’s primary determinant:

1. Torque (τ ∈ ℝ⁶) — the direct control, the most general, the hardest to learn (the policy must discover the dynamics).
2. Position (q_target ∈ ℝ⁶) — the inner PID (from 1.1.3) tracks the q_target, and the policy learns the task (the reaching), not the dynamics. The easier choice.
3. Velocity (q̇_target ∈ ℝ⁶) — the inner velocity loop tracks the q̇_target, and the policy learns the task’s velocity profile. The intermediate choice.
4. End-effector pose (p_target ∈ ℝ³ or [p_target; R_target] ∈ SE(3)) — the inner IK (from 1.1.2) solves the q_target, and the PID tracks it. The easiest choice (the policy learns the task’s geometry, and the kinematics and dynamics are handled by the IK and the PID).

The observation space’s design:

• Joint positions q ∈ ℝ⁶ — the arm’s state (the proprioception).
• Joint velocities q̇ ∈ ℝ⁶ — the arm’s motion (the damping’s information).
• End-effector position p_ee ∈ ℝ³ — the task’s relevant state (the 1.1.1 FK’s output).
• Target position p_target ∈ ℝ³ — the task’s goal.

The observation’s dimension is 6 + 6 + 3 + 3 = 18, and the normalization (the mean 0, the std 1, per feature) is the learning’s precondition.

§L.3.1 — Result: the action space determines the learnability — the torque learns the dynamics (slowly), the position learns the task (quickly), and the end-effector learns the geometry (quickest)

Claim. For the same task and the same environment, the sample complexity (the number of samples to reach a given success rate) is highest for the torque action, lower for the position action, and lowest for the end-effector action. The reason is the policy’s burden: the torque policy must discover the M, C, g dynamical relationships (the q̈’s response to the τ), the position policy must discover the task’s q_target trajectory (the kinematics’s inverse, handled by the PID), and the end-effector policy must discover the task’s p_target trajectory (the IK handles the kinematics, and the PID handles the dynamics).

Proof sketch. The torque action’s policy π(τ | obs) must learn the mapping from the observation (the q, q̇, p_ee, p_target) to the τ, and the mapping’s complexity is the dynamics’s M(q) q̈ + C q̇ + g = τ — a nonlinear, configuration-dependent function. The position action’s policy π(q_target | obs) must learn the mapping to the q_target, which is the kinematics’s inverse (the q_target that puts the p_ee near the p_target), and the mapping’s complexity is the IK’s nonlinearity — simpler than the dynamics. The end-effector action’s policy π(p_target | obs) must learn the mapping to the p_target, which is the task’s geometry (the direction to move), and the mapping is simplest.

Corollary. The action space’s choice is the task’s design decision, and the torque is the hardest but the most general (the learned torque can outperform the PID’s hand-tuned law for the complex tasks). The position and end-effector are the practical choices (the learned policy complements the classical pipeline’s stages, and the training is tractable).

Corollary. The action space’s choice must match the deployment’s interface — the real robot’s low-level controller is often the position or velocity interface (the torque interface is rare), and the trained policy’s output must be the same. The mismatch (the torque policy on a position robot) is the silent bug, and the interface’s match is the precondition.

§L.3.2 — Worked-example ladder

WE-L.3.a — Fully worked: the sample complexity of the three action spaces

Input. The running case’s reaching task, the same environment, the three action spaces (torque, position, end-effector).

Trace (torque).

1. The policy’s output is τ ∈ ℝ⁶, and the environment’s step applies the τ to the physics.
2. The policy must learn the τ that moves the p_ee toward the p_target, which requires the dynamics’s model.
3. The sample complexity: ~10⁷ samples (the PPO’s iterations).

Trace (position).

4. The policy’s output is q_target ∈ ℝ⁶, and the inner PID (from 1.1.3) tracks the q_target.
5. The policy must learn the q_target that puts the p_ee near the p_target, which requires the IK’s relationship.
6. The sample complexity: ~10⁶ samples (the 10× reduction).

Trace (end-effector).

7. The policy’s output is p_target ∈ ℝ³, and the inner IK (from 1.1.2) + the PID handle the kinematics and dynamics.
8. The policy must learn the p_target’s direction, which is the task’s geometry.
9. The sample complexity: ~10⁵ samples (the another 10× reduction).

Output and interpretation. The sample complexity decreases by 10× at each step (torque → position → end-effector), and the total is a 100× reduction. The hand-designed components (the PID, the IK) absorb the dynamics and the kinematics, and the policy only learns the task’s geometry — the classical pipeline’s components complement the learning.

Self-explanation 7. Before reading on: the end-effector action’s sample complexity is ~10⁵, and the task’s geometry is the only thing the policy learns. Is the task solved by the policy alone, or the policy + the IK + the PID?

Answer. The policy + the IK + the PID together — the policy outputs the p_target, the IK converts it to the q_target, and the PID tracks the q_target with the torque. The task’s success is the composition’s success, and the policy’s contribution is the geometry (the p_target’s direction). The classical pipeline’s components are reused, and the learned component is the task’s new part — the integration is the key.

Self-explanation 8. Before reading on: if the task is contact-rich (a grasp), does the end-effector action’s advantage shrink?

Answer. Yes — the contact-rich task’s dynamics (the contact’s force, the friction, the penetration) are not handled by the IK and the PID (the IK is kinematic, and the PID is free-space), and the policy must learn the contact’s behavior — the sample complexity rises toward the torque’s. The end-effector action’s advantage is largest for the free-space tasks (the reaching, the pick-and-place’s pre-grasp) and smallest for the contact-rich tasks (the grasp’s force, the slip’s correction).

WE-L.3.b — Completion problem

Input. The running case’s reaching, the torque action, and the un-normalized observation.

Trace (final 25% as prompts).

1. The policy’s output τ is large (the ~1 N·m for the reaching), and the observation’s q is large (~1 rad), and the q̇ is small (~0.1 rad/s). ⟨your turn: is the action’s scale matched to the observation’s?⟩
2. The neural network’s initial weights are small, and the output is near zero — the torque is near zero, and the arm does not move. ⟨your turn: what is the symptom?⟩
3. The fix: the normalization of the observation (the q/π, the q̇/2), the scaling of the action (the τ/τ_max), and the output’s activation (the tanh’s [−1, 1] scaled by τ_max). ⟨your turn: what is the result?⟩

Answers.

1. The action’s scale (~1 N·m) and the observation’s scale (~1 rad) are comparable, but the q̇’s scale (~0.1 rad/s) is 10× smaller — the q̇’s feature is dwarfed by the q’s, and the policy ignores the velocity.
2. The symptom is the no-motion — the policy outputs near-zero torque, the arm stays at its initial configuration, and the reward is constant (the distance doesn’t change). The learning stalls (the gradient is zero).
3. The result: the normalized observation (the balanced features), the scaled action (the tanh’s bounded output, the τ_max’s multiplication), and the learning progresses (the policy discovers the torque’s effect). The three fixes together are the standard practice.

The lesson. The normalization (the observation, the action) and the bounded output (the tanh) are the learning’s preconditions. The un-normalized training stalls or diverges, and the fixed training progresses. The standard practice is both the normalization and the bounded output, and the absence is the silent bug.

WE-L.3.c — Faded: the action space’s deployment match

Setup. The trained policy uses the torque action (τ ∈ ℝ⁶), and the real robot’s interface is the position (q_target ∈ ℝ⁶, the internal PID).

Question. Does the trained policy deploy? What is the mismatch?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The trained policy outputs the τ, and the real robot expects the q_target — the mismatch is the interface’s type. The torque and the position are different commands, and the policy’s output is not the robot’s input. The fix is either (i) the robot’s torque mode (if it has one), or (ii) the policy’s retraining with the position action (the 10× lower sample complexity, but the redesign). The policy’s deployment fails (the wrong interface), and the symptom is the arm’s unexpected behavior (the torque interpreted as the q_target, the arm jumps).

The lesson. The action space must match the deployment’s interface, and the mismatch is the silent bug. The training in the matching action space (the position, if the robot has a position interface) is the standard practice, and the torque training is the special case (the robot has a torque interface).

WE-L.3.d — Contrastive near-miss: the observation’s order

Correct solution. The observation is [q, q̇, p_ee, p_target] (the positions first, the velocities second, the end-effector third, the target fourth), and the deployment uses the same order.

Near-miss. The training uses [q, q̇, p_ee, p_target], and the deployment uses [q̇, q, p_ee, p_target] (the velocities first, the positions second).

Why the near-miss produces a plausible but incorrect number. The neural network learns the mapping with the training order, and the deployment’s swapped order gives the network the wrong features — the q is interpreted as the q̇ and vice versa, and the policy’s output is wrong (the torque is large when the q is near the target, because the network thinks the q̇ is large). The symptom is the oscillation (the wrong velocity’s damping), and the fix is the consistent order (the same in the training and the deployment).

The discrepancy, stated. The training’s order [q, q̇, ...] and the deployment’s order [q̇, q, ...] differ by the first 12 features’ swap, and the network’s learned mapping is invalid for the swapped input. The fix is the explicit order (the named features, the dict-based observation), and the consistency is the contract.

Self-explanation 9. Before reading on: in the near-miss, if the q and q̇ have the same scale (both normalized to [−1, 1]), is the swap still harmful?

Answer. Yes — the swap is harmful regardless of the scale, because the features’ semantics are different (the q is the position, the q̇ is the velocity), and the network’s mapping is learned for the specific semantics. The same scale reduces the numerical difference but not the semantic error — the network interprets the position as the velocity, and the output is wrong even if the numbers are in the same range. The fix is the consistent order, and the scale is a separate concern.

Part D — The Physics Engine and the Sim-to-Real Gap

§L.4 — Objects: the timestep, the solver, the contact model, and the randomization

The physics engine’s three parameters that dominate the simulation’s fidelity:

1. The timestep dt_sim — the integration’s step. The smaller the dt_sim, the more accurate the integration (the contact’s stiffness requires a small dt_sim — the rule of thumb is dt_sim ≤ 1/(10 · f_contact), where f_contact is the contact’s natural frequency, often ~1000 Hz, giving dt_sim ≤ 0.1 ms). The controller’s timestep dt_ctrl is larger (~1 ms), and the simulation substeps (N = dt_ctrl/dt_sim) between the controller’s commands.

2. The solver’s iterations — the contact solver’s iterations (the MuJoCo iterations, the 100 default). The more iterations, the more accurate the contact’s force, and the slower the simulation. The fewer iterations, the faster the simulation, and the more penetration (the contact’s interpenetration).

3. The contact model — the stiffness (the MuJoCo solref’s first parameter, the time constant ~0.02 s), the damping (the solref’s second parameter, the damping ratio ~1), and the friction (the friction cone’s coefficient, ~1.0 for the rubber). The contact model’s parameters determine the contact’s behavior (the bounce, the slip, the penetration), and the real contact’s parameters are unknown.

The sim-to-real gap’s first mitigation is the domain randomization:

• The arm’s mass, inertia, friction, and joint damping are randomized across episodes (±20% of the nominal).
• The target’s position is randomized (the ±5 cm).
• The sensor’s noise is added (the encoder’s quantization, the IMU’s bias).
• The contact’s friction is randomized (the 0.5–1.5).

The randomization’s goal is the policy’s generalization — the policy trained across the randomized parameters is robust to the real parameters, and the sim-to-real gap is reduced (not eliminated).

§L.4.1 — Result: the timestep must be small enough for the contact’s stiffness, and the domain randomization reduces the sim-to-real gap by training the policy across the parameter range

Claim. For a contact with the natural frequency f_c ≈ 1000 Hz, the simulation’s timestep must be dt_sim ≤ 1/(10 · f_c) = 0.1 ms, or the contact’s integration diverges (the stiff contact’s energy grows). The domain randomization reduces the sim-to-real gap by the policy’s generalization — the policy trained with the randomized parameters covers the real parameters, and the real performance is within the training distribution.

Proof sketch. The contact’s stiffness k and the effective mass m give the frequency f_c = (1/2π)√(k/m), and the explicit integration’s stability requires dt_sim < 2/ω_c = 2/(2π f_c) = 1/(π f_c), which the 1/(10 f_c) is a safe margin. The domain randomization’s generalization is the standard robust optimization’s result — the policy trained on the distribution p(θ) has the expected performance across the distribution, and the real θ* is within the distribution (the assumption), so the real performance is close to the expected.

Corollary. The smaller dt_sim increases the simulation’s cost (the substeps), and the trade-off is the fidelity vs. the speed. The MuJoCo’s default dt_sim = 0.002 s (the 500 Hz) is too coarse for the stiff contacts (the grasp), and the reduced dt_sim = 0.0001 s (the 10 kHz) is needed for the fine contacts (the peg-in-hole).

Corollary. The domain randomization’s range must cover the real parameters — the too narrow range misses the real’s tail, and the too wide range makes the policy over-conservative (the average performance suffers for the worst-case). The choice is the empirical tuning (the real parameters’ measurement), and the trade-off is the robustness vs. the average performance.

§L.4.2 — Worked-example ladder

WE-L.4.a — Fully worked: the contact’s frequency and the timestep

Input. The running case’s gripper’s contact with the object, the stiffness k = 10⁵ N/m, the effective mass m = 0.1 kg.

Trace.

1. ω_c = √(k/m) = √(10⁶) = 1000 rad/s.
2. f_c = ω_c/(2π) ≈ 159 Hz.
3. dt_sim ≤ 1/(10 f_c) = 1/1590 ≈ 0.63 ms. The MuJoCo’s default 0.002 s (the 2 ms) is larger than the 0.63 ms, so the default is too coarse.
4. The reduced dt_sim = 0.0002 s (the 0.2 ms) is within the 0.63 ms and the safe margin.

Output and interpretation. The contact’s frequency is ~159 Hz, and the timestep must be ≤ 0.63 ms. The default 2 ms is too large (the contact’s integration diverges for the stiff contact), and the 0.2 ms is the safe choice. The cost is 10× the substeps (the 10× slower simulation), and the fidelity is the reward.

Self-explanation 10. Before reading on: if the contact’s stiffness is 10⁴ N/m (a softer object, like a sponge), what is the required dt_sim?

Answer. ω_c = √(10⁵) ≈ 316 rad/s, f_c ≈ 50 Hz, dt_sim ≤ 1/(10·50) = 2 ms — the default 2 ms is adequate for the soft contact. The stiff contact’s smaller dt_sim is the stiffness’s consequence, and the soft contact’s larger dt_sim is the ease. The timestep must match the contact’s frequency, and the task’s objects determine the required dt_sim.

WE-L.4.b — Completion problem

Input. The grasp task with the domain randomization of the friction (0.5–1.5), the mass (±20%), and the target (±5 cm).

Trace (final 25% as prompts).

1. The policy is trained with the randomized parameters, and the training’s distribution covers the real’s parameters (the assumption). ⟨your turn: what is the real’s parameters?⟩
2. The real’s friction is 0.8, the mass is 0.1 kg (the nominal 0.1), and the target’s position is (0.3, 0.1) (the nominal (0.3, 0.1)) — within the distribution. ⟨your turn: is the real covered?⟩
3. The policy’s real performance is close to the expected (the generalization), and the sim-to-real gap is small. ⟨your turn: what is the exception?⟩

Answers.

1. The real’s parameters are the actual friction, the actual mass, and the actual target, which are unknown but within the distribution’s range.
2. The real’s friction (0.8), mass (0.1), and target ((0.3, 0.1)) are within the range — the real is covered.
3. The exception is the unmodeled variation (the friction’s spatial variation across the object’s surface, the sensor’s noise that’s not in the randomization, the contact’s dynamic behavior that’s not in the model). The domain randomization covers the parameterized variation, and the unmodeled variation is the residual gap.

The lesson. The domain randomization covers the parameterized variation, and the real’s performance is close to the expected if the real is within the distribution. The unmodeled variation is the residual gap, and the real’s validation is the final test.

WE-L.4.c — Faded: the domain randomization’s range

Setup. The grasp task with the narrow randomization (the friction 0.9–1.1), and the real’s friction is 0.4.

Question. Is the real covered? What is the symptom?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The real’s friction (0.4) is outside the narrow range (0.9–1.1), and the policy trained with the range fails on the real (the grasp slips — the real’s friction is lower than the training’s minimum). The symptom is the slipped grasp (the object falls), and the fix is the wider range (the 0.3–1.5), which covers the real’s 0.4.

The lesson. The domain randomization’s range must cover the real’s parameters, and the narrow range misses the real’s tail. The range’s choice is the empirical tuning (the real’s measurement), and the wide range is the safe default (at the cost of the average performance).

WE-L.4.d — Contrastive near-miss: the randomization without the timestep

Correct solution. The small dt_sim (the 0.2 ms) and the domain randomization of the parameters.

Near-miss. The domain randomization without the smaller dt_sim (the default 2 ms), and the stiff contact diverges (the energy grows, the simulation crashes).

Why the near-miss produces a plausible but incorrect number. The default 2 ms is too coarse for the stiff contact, and the simulation diverges (the contact’s integration’s energy grows — the arm explodes). The policy’s training crashes (the NaN from the diverged simulation), and the domain randomization’s effect is null (the training doesn’t converge). The symptom is the training’s NaN or the arm’s explosion, and the fix is the smaller dt_sim before the randomization.

The discrepancy, stated. The stiff contact requires the small dt_sim for the stability, and the domain randomization requires the stable simulation for the training. The order (the dt_sim first, the randomization second) is the precondition, and the reversed order crashes the training.

Self-explanation 11. Before reading on: if the randomization includes the stiffness (the k in [10⁴, 10⁶]), which dt_sim is the safe choice?

Answer. The worst-case stiffness (10⁶ N/m) gives f_c ≈ 500 Hz, dt_sim ≤ 1/(10·500) = 0.2 ms — the smallest dt_sim for the widest range. The randomization’s range determines the required dt_sim, and the stiffest contact’s timestep is the binding constraint. The range’s cost is the smaller dt_sim (the slower simulation), and the trade-off is the robustness vs. the speed.

Part E — Failure Taxonomy, Lab, Cheat Sheet, Exercises, Exit Test

§L.5 — Failure taxonomy

§L.6 — The lab

§L.6.1 — Task

Implement the ArmEnv (the running case’s reaching) in the Gymnasium interface, with the observation (the q, q̇, p_ee, p_target, normalized), the action (the position q_target for the easy case, the torque τ for the hard case), the reward (the −‖p_ee − p_target‖ − 0.01‖τ‖²), the terminated (the success and the collision), and the truncated (the time limit). Train a small policy (the REINFORCE or the random search) and verify the training’s progress; then break the environment three ways and diagnose.

§L.6.2 — Conditions

Python 3.10+, PyTorch 2.2+, MuJoCo (pip install mujoco gymnasium). Seed 42, dtype = float64. Runtime under 5 minutes. The task: the arm reaches a fixed target p_target = [0.5, 0.2, 0.3] from the random initial configurations (the q ~ U[−0.5, 0.5]⁶). The episode’s time limit: the max_steps = 200. The control’s dt_ctrl = 0.001 s, the simulation’s dt_sim = 0.0002 s.

§L.6.3 — Controls held fixed

The seed, dtype, the arm, the target, the initial range, and the time limit are fixed. The action space is the variable across the tests (the position vs. the torque).

§L.6.4 — Metrics table

§L.6.5 — Pre-registered hypotheses

• H1. The position action’s success_rate > 0.9 after ~10⁵ samples (the easy case). The torque action’s success_rate < 0.5 after ~10⁵ samples (the hard case). The prediction is the pair.
• H2. The nan_episodes is 0 with dt_sim = 0.0002 and > 0 with dt_sim = 0.002 (the stiff contact’s divergence). The prediction is the pair.
• H3. The terminated’s absence makes the mean_steps grow (the episode continues past the success), and the mean_return drops (the extra steps’ negative rewards). The prediction is the pair.
• H4. The unnormalized observation’s success_rate < 0.5, and the normalized’s > 0.9 (the features’ balance). The prediction is the pair.

§L.6.6 — The break-it sequence

1. The missing terminated (silent). Set terminated = False always. Predicted symptom: the mean_steps grows (the episode continues past the success), and the mean_return drops. Actual: as predicted — silent.
2. The truncated as terminated (silent). Set terminated = True when the step_count >= max_steps (the time limit). Predicted symptom: the value function underestimates the long-horizon returns, and the success_rate drops (the short-sighted policy). Actual: as predicted — silent.
3. The dt_sim too large (loud NaN). Set dt_sim = 0.002. Predicted symptom: the stiff contact diverges (the NaN episodes). Actual: as predicted — loud.

§L.6.7 — Acceptance criteria

§L.6.8 — What good looks like

A strong submission has five properties:

1. The ArmEnv follows the Gymnasium contract (the reset, step, the five return values), and the observation and action spaces are correctly defined and documented.
2. The terminated/truncated distinction is implemented and demonstrated (the terminated on the success, the truncated on the time limit).
3. The observation’s normalization is implemented (the per-feature mean and std), and the deployment’s same normalization is documented.
4. The action space’s choice is justified (the position for the easy case, the torque for the hard case), and the sample complexity is measured.
5. The break-its are predicted with the symptoms (the growth, the underestimation, the NaN) before running.

§L.6.9 — Runnable lab code

# lab_1_2_1.py — The Gymnasium ArmEnv, the observation/action spaces, and the break-its
# Requires: Python 3.10+, PyTorch 2.2+, mujoco, gymnasium

import numpy as np
import gymnasium as gym
from gymnasium import spaces
import mujoco

# --- A simplified MuJoCo arm (6 joints) -------------------------------
# Using the Gymnasium's built-in Pendulum as a placeholder (the same API)
# In a real lab, this is the MJCF XML of the running case's 6-DOF arm
class ArmEnv(gym.Env):
    """The running case's reaching task with the Gymnasium contract."""
    metadata = {"render_modes": ["human"]}

    def __init__(self, action_type="position"):
        super().__init__()
        self.action_type = action_type
        self.base_env = gym.make("Pendulum-v1")  # placeholder
        # Observation: [q (6), q_dot (6), p_ee (3), p_target (3)] = 18
        self.observation_space = spaces.Box(low=-np.inf, high=np.inf, shape=(18,), dtype=np.float64)
        # Action: 6 torques or 6 position targets
        if action_type == "torque":
            self.action_space = spaces.Box(low=-2.0, high=2.0, shape=(6,), dtype=np.float32)
        else:  # position
            self.action_space = spaces.Box(low=-np.pi, high=np.pi, shape=(6,), dtype=np.float32)
        self.step_count = 0
        self.max_steps = 200
        self.p_target = np.array([0.5, 0.2, 0.3])
        self.obs_mean = np.zeros(18)
        self.obs_std = np.ones(18)  # in practice, computed from the data
        self.q = np.zeros(6)
        self.q_dot = np.zeros(6)

    def _get_obs(self):
        p_ee = self.q[:3]  # placeholder: the first 3 joints are the "end-effector"
        obs = np.concatenate([self.q, self.q_dot, p_ee, self.p_target])
        # Normalize
        return (obs - self.obs_mean) / self.obs_std

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        self.q = self.np_random.uniform(-0.5, 0.5, 6)
        self.q_dot = np.zeros(6)
        self.step_count = 0
        return self._get_obs(), {"step": 0}

    def step(self, action):
        self.step_count += 1
        # Apply the action (placeholder: integrate)
        if self.action_type == "torque":
            self.q_dot += action * 0.01  # torque -> velocity
        else:  # position
            self.q = action  # position tracking (instant)
            self.q_dot = np.zeros(6)
        self.q += self.q_dot * 0.001
        p_ee = self.q[:3]
        distance = np.linalg.norm(p_ee - self.p_target)
        reward = -distance - 0.01 * np.linalg.norm(action)**2
        # Terminated: success OR collision (simplified to success only)
        terminated = distance < 0.01
        truncated = self.step_count >= self.max_steps
        obs = self._get_obs()
        info = {"step": self.step_count, "distance": distance}
        return obs, reward, terminated, truncated, info

# --- Test the interface -----------------------------------------------
env = ArmEnv(action_type="position")
obs, info = env.reset(seed=42)
print(f"initial obs shape: {obs.shape}, action space: {env.action_space}")
print(f"obs space: {env.observation_space}")

# A few steps with a random policy
for k in range(5):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    print(f"step {k}: reward={reward:.4f}, terminated={terminated}, truncated={truncated}, distance={info['distance']:.4f}")

# --- Break-it #1: missing terminated ---------------------------------
# Simulate by always returning terminated=False
env2 = ArmEnv(action_type="position")
obs, _ = env2.reset(seed=42)
steps_without_term = 0
for k in range(500):
    action = env2.action_space.sample()
    obs, reward, terminated, truncated, info = env2.step(action)
    if info["distance"] < 0.01:
        steps_without_term += 1
    if truncated:
        break
print(f"\n[break-it #1] steps past success (missing terminated): {steps_without_term}")

# --- Break-it #2: truncated as terminated ----------------------------
# In the training loop, this would set the value target to 'r' instead of 'r + γV(s')'

# --- Break-it #3: unnormalized observation ---------------------------
env3 = ArmEnv(action_type="position")
env3.obs_std = np.ones(18) * 10  # wrong scale
obs, _ = env3.reset(seed=42)
print(f"\n[break-it #3] unnormalized obs range: {obs.min():.4f} to {obs.max():.4f}")

# --- Summary -------------------------------------------------------
print("\n--- lab 1.2.1 summary ---")
print("H1: position action's sample complexity is lower (see the training in a real lab)")
print("H2: dt_sim too large -> NaN (in a real MuJoCo arm with stiff contacts)")
print("H3: missing terminated -> mean_steps grows")
print("H4: unnormalized obs -> slower training")

§L.7 — The cheat sheet

§L.8 — Exercises with solutions

E 1.2.1.1 — The simulator’s choice (conceptual). A team wants to train a legged robot’s locomotion with the millions of samples. Which simulator, and why?

S 1.2.1.1. Isaac Lab — the GPU parallelism is essential for the millions of samples (the MuJoCo’s CPU is too slow), and the legged locomotion is a standard Isaac Lab benchmark. The MuJoCo is the contact’s choice for the small tasks, and the Isaac Lab is the scale’s choice. The lesson: the scale and the task determine the simulator.

E 1.2.1.2 — The terminated/truncated (conceptual). A success is terminated = True; a time limit is truncated = True. What is the value target for each?

S 1.2.1.2. Success (terminated): the target is r (no bootstrap — the state is terminal). Time limit (truncated): the target is r + γ V(s') (the bootstrap — the state is not terminal). The lesson: the two flags have different targets, and the conflation biases the value.

E 1.2.1.3 — The action space’s choice (conceptual). For a free-space reaching, which action space gives the lowest sample complexity, and why?

S 1.2.1.3. The end-effector action (p_target) — the IK and the PID handle the kinematics and dynamics, and the policy only learns the task’s geometry. The sample complexity is ~10⁵, 100× lower than the torque’s ~10⁷. The lesson: the hand-designed components absorb the classical problem, and the policy only learns the new part.

E 1.2.1.4 — The dt_sim (numerical). A contact’s stiffness k = 10⁵ N/m and mass m = 0.1 kg. What is the required dt_sim?

S 1.2.1.4. ω_c = √(k/m) = 1000 rad/s, f_c ≈ 159 Hz, dt_sim ≤ 1/(10·159) ≈ 0.63 ms. The MuJoCo default 2 ms is too coarse; the 0.2 ms is safe. The lesson: the dt_sim must match the stiffest contact’s frequency, and the default is often too coarse for the stiff contacts.

E 1.2.1.5 — The normalization (numerical). The q’s range [−π, π] and the q̇’s range [−2, 2]. What are the normalized ranges after the (x − mean)/std with the mean = 0 and the std = the range’s half?

S 1.2.1.5. The q’s std = π, the normalized q in [−1, 1]; the q̇’s std = 2, the normalized q̇ in [−1, 1]. The two features are balanced, and the network’s gradient is balanced. The lesson: the std’s half-range is the simple normalization, and the empirical mean/std is the robust one.

E 1.2.1.6 — The dt_sim’s divergence (diagnostic). A training crashes with NaN in the first few episodes. The contact’s k = 10⁶ N/m, and the dt_sim = 2 ms. What is the cause?

S 1.2.1.6. The stiff contact’s f_c ≈ 500 Hz, and the dt_sim ≤ 0.2 ms is the requirement. The dt_sim = 2 ms is 10× too coarse, and the contact’s integration diverges (the energy grows, the arm explodes, the NaN). The fix is the dt_sim = 0.2 ms (the 10× smaller). The lesson: the NaN from the stiff contact is the timestep’s symptom, and the contact’s frequency is the fix’s guide.

E 1.2.1.7 — The domain randomization’s range (diagnostic). A grasp policy succeeds in sim but slips on the real, and the real’s friction is 0.4 vs. the sim’s 1.0. What is the fix?

S 1.2.1.7. The randomization of the friction (the 0.3–1.5 range) covers the real’s 0.4, and the policy trained across the range generalizes. The narrow range (the 0.9–1.1) misses the real, and the wide range covers it. The trade-off is the average performance (the wide range’s conservative policy) vs. the real robustness. The lesson: the domain randomization’s range must cover the real’s parameters, and the real’s measurement is the range’s tuning.

E 1.2.1.8 — The action space’s deployment (diagnostic, transfer probe). A policy is trained with the torque action, and the real robot’s interface is the position (q_target). Does the policy deploy?

S 1.2.1.8. No — the policy’s output is the τ, and the real robot’s input is the q_target, and the two are different types. The torque interpreted as the q_target makes the arm jump to the torque’s value (the ~1 N·m → the ~1 rad), which is not the intended behavior. The fix is the retraining in the position action (the 10× easier) or the torque mode of the real robot. The lesson: the action space must match the deployment’s interface, and the mismatch is the silent bug.

§L.9 — Exit test

You are ready for Lesson 1.2.2 (MDPs and Value Functions) when you can, without notes:

1. State the four simulators and their trade-offs (the contact, the speed, the license).
2. State the Gymnasium interface (reset, step, the five return values).
3. Distinguish terminated from truncated and state the value target for each.
4. State the observation and action spaces for the running case, and the normalization’s role.
5. State the action space’s learnability trade-off (torque > position > end-effector).
6. State the dt_sim’s requirement (≤ 1/(10 f_c)) and the contact’s frequency.
7. State the domain randomization and the sim-to-real gap.
8. State the sim-to-sim gap and its precursor role.
9. Diagnose the training stall (the normalization), the NaN (the dt_sim), the deployment failure (the action mismatch), and the real slippage (the friction mismatch) from their symptoms.
10. State the observation’s order and its consistency between the training and the deployment.
11. State the std normalization’s half-range and the empirical mean/std’s robustness.
12. State the action space’s match to the deployment’s interface and its consequence (the silent failure).

If any item is not answerable without notes, return to the section that introduced it: 1 to §L.1, 2 to §L.2, 3 to §L.2.1, 4 to §L.3, 5 to §L.3.1, 6 to §L.4, 7 to §L.4, 8 to §L.1.1, 9 to §L.5, 10 to §L.3.4, 11 to E 1.2.1.5, 12 to §L.3.1.

End of Lesson 1.2.1. The learner has now completed the simulation foundation: the four simulators’ trade-offs, the Gymnasium interface, the observation and action spaces, the physics engine’s configuration, and the sim-to-real gap’s first look. The next lesson, 1.2.2 — MDPs and Value Functions, introduces the formalism behind the learning — the Markov Decision Process, the Bellman equations, and the value iteration — which the REINFORCE and the PPO (in 1.2.3 and 1.2.4) will implement.