X2.1.1

Lesson 2.1.1 — Camera Models and Image Formation: The Pinhole, the Intrinsics, and the Projection Equation

Week 9 · Prerequisites: 1.2.1, 1.2.2, 0.2.1, 0.2.2, 0.2.4.

Prior results used. This lesson uses 1.2.1’s ArmEnv’s observation space (the image as a perceptual input), 1.2.2’s state as the sufficient statistic (the perception’s target), 0.2.1’s SE(3) composition and inverse, 0.2.2’s SO(3) rotations, and 0.2.4’s from-to convention and frame audit — the same composition that the perception-to-IK boundary used. Each is restated at the point of use.

Hands forward. This lesson hands forward one capability: the learner can state the pinhole camera model, write the projection equation x = K [R | t] X, compute the intrinsics from a camera’s specifications, undistort a measured image point, back-project a pixel to a ray, and diagnose the four camera-model bugs (wrong intrinsics, wrong extrinsic direction, the Z = 0’s perspective division, and the missing distortion’s correction) from their reprojection’s symptoms — the capability that every subsequent perception lesson (2.1.2 detection, 2.1.3 6D pose, 2.1.4 visual servoing, 2.4.1 SLAM) assumes.

Running case. The same 6-DOF MuJoCo arm from 1.1.1–1.2.5, with a camera mounted on the end-effector (from 0.2.4’s pipeline). The camera’s intrinsics are f_x = f_y = 500 pixels, c_x = 320, c_y = 240, the image is 640 × 480, the extrinsics are the composition T_Cam^World = (T_World^Cam)^{-1} from 0.2.4’s T_W^EE · T_EE^Cam, and the object is a cube at the world position p^W = [0.5, 0.2, 0.3] (the same target from the pick-and-place task). The lesson’s central result is that the projection x = K [R | t] X collapses the 3D point to a 2D pixel, losing the depth (the Z), and the inverse is a ray (the pixel + the camera’s center), not a point — and the extrinsic’s direction (the world-to-camera vs. the camera-to-world) is the most common bug, silent in the single-view projection and visible in the multi-view triangulation.

§L.0 — Lesson map, objectives, and misconceptions

§L.0.1 — Lesson map and word budget

§L.0.2 — Objectives

By the end of this lesson you will be able to do eight things:

1. State the pinhole camera model — the image point x = [u, v, 1]ᵀ, the 3D point X = [X, Y, Z, 1]ᵀ, the intrinsic K, and the extrinsic [R | t] — and write the projection equation s x = K [R | t] X (the s is the scale).
2. Compute the intrinsic matrix K from a camera’s focal length, pixel size, and resolution, and state the skew’s (usually zero) role and the principal point’s meaning.
3. Apply the extrinsic [R | t] with the correct direction (the world-to-camera for the projection, the camera-to-world for the back-projection), and verify the direction by projecting the camera’s own origin (the camera-to-world’s t) to the pixel (0, 0).
4. Compute the projection by hand for a specific 3D point, and state the depth’s loss (the Z’s ambiguity) and the back-projection’s ray (the pixel + the camera’s center).
5. State the field of view FOV = 2 arctan(W/(2 f)) and the focal length’s effect (the larger f, the narrower the FOV, the more zoomed), and compute the FOV for a specific camera.
6. Apply the radial and the tangential distortion models x_undistorted = x_distorted (1 + k_1 r² + k_2 r⁴ + k_3 r⁶) + (tangential), and state why the undistortion is required for the geometric accuracy (the radial’s barrel or pincushion’s effect).
7. Diagnose the four camera-model bugs — wrong intrinsics (the projection’s offset), wrong extrinsic direction (the object’s mirrored in the multi-view), the Z = 0’s division (the point at the camera’s center), and the missing distortion (the systematic radial error) — from their reprojection’s symptoms.
8. Produce a working projection and back-projection implementation for the running case’s camera, with the intrinsics calibrated (a synthetic calibration), the extrinsics from the FK + the calibration, the distortion’s correction, and the verification against the known 3D points.

§L.0.3 — Misconception inventory

Each misconception is displaced in the section named, and each is tagged in the concept check that follows.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

This is the twenty-second lesson. The prompt requires drawing from lessons two to five steps back; 1.2.5 is one step back and is excluded. The warm-up draws from 1.2.4 (two steps back), 1.2.3 (three), 1.2.2 (four), and 1.2.1 (five). The geometric prerequisites (0.2.1–0.2.4) are more than five steps back and are restated at the point of use in Parts A–B.

Warm-up 1 (from 1.2.1, the observation space). State the ArmEnv’s observation space and the normalization’s role. Answer: the observation is [q, q̇, p_ee, p_target] (18 numbers), normalized to the mean 0, std 1; the normalization balances the features and enables the learning. How this is used: §L.1’s camera’s observation is the image (H × W × 3), and its normalization (the pixel’s [0, 255] to the [0, 1] or the [-1, 1]) is the same balance, and the image’s inclusion in the observation is the perception’s addition to the 1.2.1’s ArmEnv.

Warm-up 2 (from 1.2.2, the state’s sufficiency). State the Markov property and the state’s sufficient statistic’s role. Answer: the state is the sufficient statistic for the future (P(s' | s, a, history) = P(s' | s, a)), and the policy can be a function of the current state. How this is used: §L.3’s back-projection is the perception’s partial observation — the image is the 2D projection, and the state’s 3D position is the sufficient (the depth’s missing), and the perception’s task is to recover the state from the image (the inverse problem, the depth’s ambiguity).

Warm-up 3 (from 1.2.3, the policy’s input). State the policy π_θ(a | s) and its input. Answer: the policy is the distribution over the actions given the state, and the state is the input (the observation, the proprioception). How this is used: §L.1’s projection is the observation model y = h(s) (the camera’s image is a function of the 3D state), and the policy’s input is the image (the visual policy) or the estimated state (the state-based policy), and the perception’s role is to convert the image to the state.

Warm-up 4 (from 1.2.4, the rollout). State the PPO’s rollout and the terminated/truncated’s distinction. Answer: the rollout is the sequence of (s, a, r, s') transitions, and the terminated is the task’s end, the truncated is the time limit. How this is used: §L.1’s camera’s image is the visual observation that the rollout records (the 1.2.1’s observation space extended), and the image’s timestamp must match the state’s (the synchronization), or the policy sees the stale image (the motion’s blur, the latency’s delay).

Part A — The Pinhole Model and the Intrinsics

§L.1 — Objects: the pinhole, the projection, and the K matrix

The pinhole camera model is the idealized camera that projects a 3D point X^C = [X, Y, Z]ᵀ (in the camera’s frame) to the image point (u, v) on the image plane. The projection is:

where:

• f_x, f_y are the focal lengths in pixels (the physical focal length f divided by the pixel’s size dx, dy).
• c_x, c_y are the principal point’s coordinates (the image’s center, in pixels).
• The division by Z is the perspective (the far objects are smaller, the near ones larger).

The homogeneous form is:

where the s = Z is the scale (the depth), and the matrix K is the intrinsic:

The s_k is the skew (the pixel’s non-orthogonality, usually zero for the modern cameras), and the f_x, f_y differ by the pixel aspect ratio (f_x/f_y = dy/dx for the non-square pixels).

The extrinsic [R | t] is the world-to-camera transform (the T_Cam^World in the 0.2.4’s from-to convention), and the full projection is:

where the X^W is the world’s homogeneous coordinate (the 4-vector), and the [R | t] is the 3 × 4 matrix that converts the world to the camera and the K converts the camera to the pixel.

The full pipeline:

1. The world point X^W (the 4-vector).
2. The camera point X^C = [R | t] X^W (the 3-vector).
3. The normalized image point x_n = [X/Z, Y/Z, 1]ᵀ.
4. The pixel x = K x_n (the 2D point, up to the scale s = Z).

§L.1.1 — Result: the projection is the K [R | t]’s homogeneous product with the scale s = Z, and the depth’s loss is the irreversible part — the pixel is the 3D point’s 2D shadow, and the depth is not recoverable from the single view

Claim. The projection s x = K [R | t] X^W is a linear equation in the homogeneous coordinates, and the scale s = Z (the camera’s depth) is the only non-linear part (the division). The projection collapses the 3D point to the 2D pixel, and the inverse is not unique — the pixel corresponds to a ray from the camera’s center through the pixel, and the depth along the ray is the lost information.

Proof sketch. The projection’s matrix P = K [R | t] is a 3 × 4 matrix of rank 3 (the K’s rank 3 and the [R | t]’s rank 3), and its null space is the 1-dimensional camera’s center (the P’s null vector is the C = [C_x, C_y, C_z, 1]ᵀ with the P C = 0). The null space is the depth’s freedom — the points along the ray C + λ (K^{-1} x - C) all project to the same x, and the λ is the depth (the lost). The inverse requires the λ’s additional information (the stereo, the known plane, the structured light).

Corollary. The depth’s loss is the pinhole’s fundamental limitation, and the RGB-D camera (the Microsoft Kinect, the Intel RealSense) adds the depth (the structured light or the time-of-flight) to recover the λ. The monocular camera’s depth is the learned (the monocular depth’s network, the Phase 3’s VLA’s implicit depth) or the multi-view’s (the stereo, the SLAM’s triangulation).

Corollary. The projection’s scale s = Z is the camera’s depth, and the division by Z is the perspective — the far objects’ smaller pixels. The division’s Z = 0 (the point at the camera’s center) is the undefined (the perspective’s singularity), and the Z → 0’s numerical instability is the camera’s near plane’s boundary.

§L.1.2 — Worked-example ladder

WE-L.1.a — Fully worked: the projection of a 3D point to a pixel

Input. The camera’s intrinsics f_x = 500, f_y = 500, c_x = 320, c_y = 240, s_k = 0 (the synthetic calibration), the camera at the world origin with the identity rotation (R = I, t = 0), the 3D point X^W = [0.5, 0.2, 1.0]ᵀ (the 1 meter in front).

Trace.

1. The camera point X^C = [0.5, 0.2, 1.0]ᵀ (the identity extrinsic).
2. The normalized image point x_n = [0.5/1.0, 0.2/1.0] = [0.5, 0.2].
3. The pixel u = 500 · 0.5 + 320 = 250 + 320 = 570.
4. The pixel v = 500 · 0.2 + 240 = 100 + 240 = 340.
5. The pixel (570, 340) — the point is right of the center (u > 320) and below (v > 240), and within the image’s [0, 639] × [0, 479].

Output and interpretation. The (570, 340) is the pixel, and the interpretation is the inverse: the u = 570 means the point is 250/500 = 0.5 normalized units to the right, and the v = 340 means it is 100/500 = 0.2 normalized units below. The depth Z = 1.0 is the scale (the lost), and the same pixel’s any depth along the ray (0.5λ, 0.2λ, λ) gives the same (570, 340).

Self-explanation 1. Before reading on: in step 3, the u = 570 — if the Z were 0.5 (the closer), what is the u?

Answer. The x_n = 0.5/0.5 = 1.0, and the u = 500·1.0 + 320 = 820 — outside the image’s [0, 639]. The closer point projects further from the center (the larger x_n), and the outside the image is the field of view’s limit. The same world point at the different depth projects to the different pixel, and the single view cannot distinguish the depth from the position.

Self-explanation 2. Before reading on: in step 4, the v = 340 — if the c_y = 0 (the top-left’s origin), what is the v?

Answer. The v = 500·0.2 + 0 = 100 — the top-left’s (0, 0) coordinate, and the 100 is the row’s index. The principal point (c_x, c_y)’s choice is the image’s coordinate’s origin, and the common choices are the top-left (c = 0) and the center (c = [W/2, H/2]). The same physical point projects to the different (u, v) depending on the principal point’s convention, and the consistency (the calibration’s and the deployment’s) is the key.

WE-L.1.b — Completion problem

Input. The same camera, the 3D point X^W = [1.0, 0.0, 2.0]ᵀ.

Trace (final 25% as prompts).

1. The normalized image point x_n = [1.0/2.0, 0.0/2.0] = [0.5, 0.0]. ⟨your turn⟩
2. The pixel u = 500·0.5 + 320 = 570. ⟨your turn⟩
3. The pixel v = 500·0.0 + 240 = 240. ⟨your turn⟩
4. The pixel (570, 240) — the point is on the image’s center row (v = c_y). ⟨your turn⟩

Answers.

1. x_n = [0.5, 0.0]. ✓
2. u = 570. ✓
3. v = 240. ✓
4. The point is on the center’s row. ✓

The lesson. The Y^W = 0 gives the v = c_y (the image’s center’s row), and the horizon (the Y = 0) is the center’s row. The interpretation: the point is level with the camera (the Y = 0), and the pixel is on the center. The depth’s Z = 2.0 is the larger (the farther), and the same X = 1.0 gives the same u = 570 (the same normalized 0.5), but the physical size is larger (the 2 meters away, the same pixel offset).

WE-L.1.c — Faded: the intrinsics from the specifications

Setup. A camera with the focal length f = 4 mm, the pixel size dx = dy = 2 µm (the 0.002 mm), the resolution 1920 × 1080.

Question. What is the K matrix?

You must produce the intermediate steps. The setup and the answer are given.

Answer.

1. The f_x = f/dx = 4/0.002 = 2000 pixels.
2. The f_y = f/dy = 2000 pixels.
3. The c_x = 1920/2 = 960, the c_y = 1080/2 = 540.
4. The K = [[2000, 0, 960], [0, 2000, 540], [0, 0, 1]].

The lesson. The focal length in pixels is the physical f divided by the pixel’s size, and the principal point is the image’s center. The K’s values are the camera’s properties (the lens’s f, the sensor’s dx, the resolution), and the calibration’s task is to measure them (the checkerboard’s calibration). The synthetic K from the specifications is the approximation, and the actual K (the manufacturing’s tolerance) is the calibrated.

WE-L.1.d — Contrastive near-miss: the f_x = f_y assumption

Correct solution. The f_x and f_y are separate (the pixel’s aspect ratio’s possible difference).

Near-miss. The f_x = f_y (the square pixels).

Why the near-miss produces a plausible but incorrect number. The modern cameras often have square pixels (the f_x = f_y), and the assumption works for them. The older or the specialized cameras (the anamorphic lenses, the non-square CCDs, the analog video) have the non-square pixels (the f_x/f_y ≠ 1), and the assumption introduces a systematic aspect ratio’s error — the reprojection is stretched in the one direction, and the pose’s estimation is biased. The symptom is the reprojection’s systematic error (the larger at the edges), and the fix is the separate f_x, f_y.

The discrepancy, stated. The f_x = f_y and the f_x ≠ f_y differ by the pixel aspect ratio (dy/dx), and the non-square pixel’s f_x/f_y = dy/dx. The calibration’s f_x, f_y are separately estimated, and the assumption’s validity depends on the camera’s specifications (the datasheet’s pixel size). The check is the calibration’s residual — the square’s assumption gives a larger residual if the pixels are non-square.

Self-explanation 3. Before reading on: the f_x = 2000, f_y = 2000 (the square), and the actual f_y = 2100 (the non-square, the dy/dx = 1.05), what is the reprojection’s error for the v = 540 (the center) and the v = 100 (the near the edge)?

Answer. The v’s actual pixel is the f_y · Y/Z + c_y = 2100 · Y/Z + 540, and the assumed f_y = 2000 gives the 2000 · Y/Z + 540. The difference is the 100 · Y/Z, and the Y/Z = (v - 540)/2000 (the assumed). At the center (v = 540), the Y/Z = 0, and the error is zero. At the near the edge (v = 100), the Y/Z = -0.22, and the error is the 100 · (-0.22) = -22 pixels — the large systematic. The edge’s error is the larger, and the center’s is the zero. The aspect ratio’s error is the systematic, the larger at the edges, and the checkerboard’s calibration’s residual is the diagnostic.

Part B — The Extrinsics and the Frame Audit

§L.2 — Objects: the world-to-camera, the camera-to-world, and the projection

The extrinsic [R | t] is the world-to-camera transform — the T_Cam^World in the 0.2.4’s from-to convention:

The camera’s pose in the world (the T_World^Cam in the 0.2.4’s from-to) is the inverse:

The projection uses the T_Cam^World (the world-to-camera), and the back-projection uses the T_World^Cam (the camera-to-world) — the two are inverses, and the substitution of the one for the other mirrors the projection.

The verification of the direction: the camera’s own origin in the world is the T_World^Cam’s translation (the -R^⊤ t), and the projection of the camera’s origin should be the undefined (the Z = 0’s division), and the projection of the point just in front of the camera’s origin (the T_World^Cam · [0, 0, 1, 1]ᵀ) should be the pixel (c_x, c_y) (the center). The check is the direction’s correctness.

The running case’s extrinsic is the composition from 0.2.4:

where the T_W^B is the base’s pose (the calibration), the T_B^{EE}(q) is the FK from 1.1.1 (the end-effector’s pose in the base), and the T_{EE}^{Cam} is the camera’s mount (the calibration). The T_Cam^World is the extrinsic, and the T_World^Cam is the camera’s pose (the inverse).

§L.2.1 — Result: the projection’s extrinsic is the world-to-camera, and the camera’s pose is the inverse — the direction’s substitution mirrors the projection, and the symptom is the reprojection’s systematic error in the multi-view

Claim. The projection x = K [R | t] X^W uses the world-to-camera [R | t] = T_Cam^World, and the camera’s pose T_World^Cam is the inverse. The substitution of the pose for the extrinsic (the wrong direction) replaces the T_Cam^World with the T_World^Cam, and the projection’s result is the mirrored (the rotated by the R^⊤ R and the translated by the t - R^⊤ t). The symptom is the reprojection’s systematic error in the multi-view (the single view’s error is the consistent, and the triangulation’s error is the inconsistent).

Proof sketch. The T_Cam^World’s inverse is the T_World^Cam = [[R^⊤, -R^⊤ t], [0, 1]], and the substitution’s projection is the K T_World^Cam X^W instead of the K T_Cam^World X^W. The two differ by the T_Cam^World T_World^Cam = I (the composition’s identity), and the wrong’s projection is the K T_World^Cam X^W, which is the correct’s K (T_Cam^World)^{-1} X^W — the inverse’s projection. The single view’s wrong projection is the consistent (the same direction’s wrong, the image is mirrored), and the multi-view’s wrong is the inconsistent (the two cameras’ wrong projections don’t agree).

Corollary. The direction’s check is the projection of the camera’s own origin (the T_World^Cam’s t), which should be the undefined (the Z = 0’s division), and the projection of the point just in front (the T_World^Cam · [0, 0, 1, 1]ᵀ), which should be the center (c_x, c_y). The check’s failure is the direction’s bug, and the fix is the inverse (0.2.1’s (R^⊤, -R^⊤ t)).

Corollary. The running case’s extrinsic is the FK + the camera mount’s composition (the 1.1.4’s T_W^{Obj}’s structure), and the direction’s bug is the same as the 0.2.4’s wrong-direction error — the symptom is the object’s mirrored position (the projection’s K T_World^Cam X^W instead of the K T_Cam^World X^W), and the fix is the inverse.

§L.2.2 — Worked-example ladder

WE-L.2.a — Fully worked: the extrinsic and its inverse

Input. The camera at the world position t_{cam}^W = [0.5, 0, 0.3]ᵀ (the camera’s origin in the world) with the rotation R_{cam}^W = Rz(π/2) (the camera looks along the world’s -y). The camera’s pose T_World^Cam = (R_{cam}^W, t_{cam}^W). The world point X^W = [0.5, 0.2, 0.3]ᵀ (the same as the 0.2.4’s target).

Trace (the camera’s pose, the T_World^Cam).

1. The R_{cam}^W = Rz(π/2) = [[0, -1, 0], [1, 0, 0], [0, 0, 1]].
2. The t_{cam}^W = [0.5, 0, 0.3].
3. The T_World^Cam = [[R_{cam}^W, t_{cam}^W], [0, 1]].

Trace (the extrinsic, the inverse T_Cam^World).

4. The R^⊤ = Rz(-π/2) = [[0, 1, 0], [-1, 0, 0], [0, 0, 1]].
5. The -R^⊤ t = -Rz(-π/2)·[0.5, 0, 0.3] = -[0, -0.5, 0.3] = [0, 0.5, -0.3].
6. The T_Cam^World = [[R^⊤, -R^⊤ t], [0, 1]] = [[[0, 1, 0, 0], [-1, 0, 0, 0.5], [0, 0, 1, -0.3]], [0, 0, 0, 1]].

Trace (the projection).

7. The camera point X^C = R^⊤ X^W + (-R^⊤ t) = [[0, 1, 0], [-1, 0, 0], [0, 0, 1]]·[0.5, 0.2, 0.3] + [0, 0.5, -0.3].
8. The R^⊤ X^W = [0.2, -0.5, 0.3].
9. The X^C = [0.2, 0, 0].
10. The pixel u = 500·(0.2/0) + 320 — the Z = 0’s division — the undefined!

Output and interpretation. The world point [0.5, 0.2, 0.3] is the same z = 0.3 as the camera’s z = 0.3, and the camera’s Z = 0 means the point is on the camera’s focal plane — the projection’s undefined. The point is level with the camera (the same z), and the camera’s -y look direction doesn’t see the point at the same z (the side of the camera). The example is the Z = 0’s singularity (the 1.2.1’s M4), and the fix is the closer point (the different z) or the camera’s orientation’s change.

Self-explanation 4. Before reading on: the X^C = [0.2, 0, 0] has the Z = 0 — what does this mean physically?

Answer. The Z = 0 means the point is on the camera’s image plane (the focal plane’s perpendicular distance is zero), and the projection’s division by Z is the undefined (the point is at the camera’s side, not the front). The physical meaning: the point is level with the camera’s optical axis (the same z), and the camera doesn’t see it (it is at the horizon). The Z = 0’s division is the singularity, and the practical’s check is the Z > 0’s validation (the point in the front).

Self-explanation 5. Before reading on: if the t_{cam}^W = [0.5, 0.2, 0.3] (the same as the point), the X^C = [0, 0, 0] — the projection is the undefined. What is the physical meaning?

Answer. The X^C = [0, 0, 0] means the point is at the camera’s origin (the same position), and the projection is the undefined (the camera cannot see itself). The physical meaning: the point is inside the camera, and the pixel is the undefined (the image is the infinitely close). The practical’s check is the Z > 0 (the point in the front), and the Z = 0 (the point at the camera’s side) and the Z < 0 (the point behind) are the invalid.

WE-L.2.b — Completion problem

Input. The same camera, but the point X^W = [0.5, -0.2, 0.3]ᵀ (the world’s -y’s side, the camera’s front).

Trace (final 25% as prompts).

1. The R^⊤ X^W = [[0, 1, 0], [-1, 0, 0], [0, 0, 1]]·[0.5, -0.2, 0.3] = [-0.2, -0.5, 0.3]. ⟨your turn⟩
2. The X^C = [-0.2, -0.5, 0.3] + [0, 0.5, -0.3] = [-0.2, 0, 0]. ⟨your turn⟩
3. The Z = 0 — the same singularity! ⟨your turn⟩
4. The point is at the same z as the camera, and the camera’s front is the world’s -y — the point’s z is the same as the camera’s z, and the projection is the undefined regardless of the y’s sign. ⟨your turn⟩

Answers.

1. [-0.2, -0.5, 0.3]. ✓
2. [-0.2, 0, 0]. ✓
3. The Z = 0. ✓
4. The Z = 0’s singularity regardless of the y’s sign. ✓

The lesson. The z = 0.3’s same as the camera’s z = 0.3 makes the Z = 0 — the camera’s image plane is the z = 0 (the perpendicular to the optical axis), and the point at the same z is on the image plane (the undefined). The camera’s orientation (the -y’s look) means the world’s -y is the camera’s +Z (the front), and the point’s z must be the larger (the further in the front) for the Z > 0. The Z = 0’s singularity is the camera’s image plane, and the practical’s point must be the in front (Z > 0).

WE-L.2.c — Faded: the running case’s projection

Setup. The running case’s camera at the end-effector’s pose T_B^EE = FK(q) for the q = [0, π/2, 0, 0, 0, 0] (the arm’s bend), the camera’s mount T_EE^Cam = [[I, [0, 0, 0.1]], [0, 1]] (the 10 cm along the end-effector’s z), the base at the world origin, the object at p^W = [0.5, 0.2, 0.3].

Question. What is the extrinsic T_Cam^World, and the object’s projection?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The end-effector’s pose T_B^EE is the FK from 1.1.1 at the q, and the camera’s pose T_World^Cam = T_W^B · T_B^EE · T_EE^Cam (the 0.2.4’s composition), and the extrinsic T_Cam^World = (T_World^Cam)^{-1}. The object’s camera point X^C = R^⊤ (p^W - t_{cam}^W) with the R^⊤ and the t_{cam}^W from the camera’s pose. The pixel x = K X^C/Z. The specific values depend on the FK’s output and the camera’s mount, and the verification is the projection of the object’s known p^W to the pixel’s plausible location (the within the image, the consistent with the camera’s look).

The lesson. The running case’s projection is the composition of the FK (the arm’s motion), the camera’s mount (the calibration), and the intrinsics (the lens), and the three are the camera model’s components. The direction’s check is the camera’s origin’s projection (the undefined), and the point in front (the center). The runtime’s cost is the 4×4 matrix multiply + the 3×3’s K’s application, and the real-time’s feasibility is the per-pixel’s fast.

WE-L.2.d — Contrastive near-miss: the camera’s pose as the extrinsic

Correct solution. The extrinsic T_Cam^World (the world-to-camera), the inverse of the camera’s pose.

Near-miss. The camera’s pose T_World^Cam (the camera-to-world), used as the extrinsic.

Why the near-miss produces a plausible but incorrect number. The single view’s wrong projection is the mirrored (the camera’s image is the back of the scene), and the symptom is the object’s wrong position (the left is the right, the up is the down). The multi-view’s wrong is the inconsistent (the two cameras’ wrong projections don’t triangulate), and the 3D’s reconstruction’s failure. The symptom is the 3D’s wrong position (the large error), and the fix is the inverse (0.2.1’s (R^⊤, -R^⊤ t)).

The discrepancy, stated. The T_Cam^World and the T_World^Cam are inverses, and the substitution’s projection is the opposite (the mirrored). The single view’s error is the systematic (the consistent’s wrong), the multi-view’s error is the inconsistent (the triangulation’s failure). The direction’s check (the camera’s origin’s projection) is the fix’s test, and the running case’s composition (the 1.1.4’s T_W^{Obj}’s structure) is the same as the 0.2.4’s audit.

Self-explanation 6. Before reading on: the single view’s T_World^Cam’s wrong projection is the mirrored — if the object is at the camera’s left, the wrong’s projection is the right’s pixel. Is the wrong’s projection consistent with the camera’s physical model?

Answer. The wrong’s projection is consistent with the wrong’s camera (the camera’s origin at the world’s -t_{cam}, the orientation at the R^⊤), which is the different camera (the mirrored’s view). The wrong’s projection is the valid (a different camera’s valid projection), and the single view’s wrong is the undetectable (the same image’s structure, the different interpretation). The multi-view’s wrong is the detectable (the two views’ inconsistency), and the fix is the direction’s correction.

Part C — The Projection’s Geometry: FOV, Depth, and Back-Projection

§L.3 — Objects: the field of view, the depth’s ambiguity, and the back-projection’s ray

The field of view (the FOV) is the angle subtended by the image’s width (or the height) at the camera’s center:

where W, H are the image’s width and height in pixels, and the f_x, f_y are the focal lengths. The larger the f (the longer the lens), the narrower the FOV (the more zoomed), and the smaller the f (the wider the lens), the wider the FOV (the more scene).

The depth’s ambiguity is the projection’s loss — the same pixel corresponds to the ray (x_n λ, y_n λ, λ) for any λ > 0, and the depth’s λ is the lost. The back-projection is the ray’s computation:

where x_n = (u - c_x)/f_x, y_n = (v - c_y)/f_y, and the R is the camera’s pose’s rotation (the T_World^Cam’s R_{cam}^W). The ray’s origin is the camera’s center t_{cam}^W, and the ray’s direction is the camera’s frame’s (x_n, y_n, 1) rotated to the world. The back-projection is the inverse of the projection (the pixel to the ray), and the depth’s λ is the free parameter along the ray.

The back-projection’s applications:

• The stereo’s triangulation: the two cameras’ rays intersect at the 3D point, and the intersection’s depth is the triangulated.
• The SLAM’s feature’s 3D position: the monocular’s feature is the ray, and the multiple views’ rays triangulate the 3D point.
• The visual servoing: the pixel’s ray is the target’s direction, and the robot’s motion aligns the ray with the camera’s axis.

§L.3.1 — Result: the back-projection is the ray (the pixel + the camera’s center), and the depth’s λ is the free parameter — the stereo and the multi-view are the triangulation’s constraints that fix the λ

Claim. The back-projection of the pixel (u, v) is the ray C + λ R [x_n, y_n, 1]ᵀ with the camera’s center C and the camera’s rotation R, and the depth λ is the free parameter. The stereo’s two cameras’ rays intersect at the 3D point (the λ’s fixed by the intersection), and the multi-view’s triangulation is the least-squares’s intersection (the λ’s estimated from the multiple rays).

Proof sketch. The projection’s inverse is the x_n = K^{-1} x (the normalized image point), and the camera’s frame’s point is the X^C = λ [x_n, y_n, 1]ᵀ (the scale λ’s free). The world’s point is the X^W = R X^C + C (the camera’s pose’s application), and the ray is the C + λ R x_n (the origin C, the direction R x_n). The λ is the depth (the lost), and the stereo’s second camera’s ray intersects the first’s at the λ’s specific value (the triangulation’s solution).

Corollary. The monocular’s depth is the learned (the monocular depth’s network), the prior (the known object’s size), or the motion’s (the structure from the motion, the SLAM’s multi-view’s triangulation). The RGB-D camera’s depth is the sensor’s (the structured light, the time-of-flight), and the depth’s λ is the measured (the not free).

Corollary. The back-projection’s direction’s correctness is the camera’s pose’s T_World^Cam’s rotation R (the camera’s orientation), and the wrong direction (the R^⊤) mirrors the ray. The fix is the 0.2.1’s inverse, and the verification is the projection’s ray’s reprojection (the pixel’s back-projection’s ray’s projection should be the same pixel).

§L.3.2 — Worked-example ladder

WE-L.3.a — Fully worked: the FOV and the back-projection’s ray

Input. The same camera: f_x = f_y = 500, W = 640, H = 480. The camera at the world’s origin with the identity rotation. The pixel (u, v) = (570, 340) (the same from WE-L.1.a).

Trace (the FOV).

1. The FOV_x = 2 arctan(640/(2·500)) = 2 arctan(0.64) ≈ 2·0.569 = 1.138 rad ≈ 65.2°.
2. The FOV_y = 2 arctan(480/1000) = 2 arctan(0.48) ≈ 2·0.448 = 0.896 rad ≈ 51.3°.

Trace (the back-projection).

3. The x_n = (570 - 320)/500 = 0.5, the y_n = (340 - 240)/500 = 0.2.
4. The camera’s frame’s ray is the λ [0.5, 0.2, 1]ᵀ.
5. The world’s ray (the identity rotation) is the λ [0.5, 0.2, 1]ᵀ (the same, since the camera is at the origin).

Output and interpretation. The FOV is the 65.2° × 51.3° — the horizontal’s wider (the 640 vs. the 480’s aspect), and the image’s corners are the FOV/2’s directions. The back-projection’s ray is the λ [0.5, 0.2, 1]ᵀ, and the λ = 1 gives the (0.5, 0.2, 1.0) — the same as the WE-L.1.a’s original point. The λ = 2 gives the (1.0, 0.4, 2.0) — the same pixel (the projection 0.5, 0.2’s normalized), the different 3D point. The depth’s λ is the free.

Self-explanation 7. Before reading on: the FOV’s 65.2° × 51.3° — what is the aspect ratio, and how does it relate to the image’s 640 × 480?

Answer. The aspect ratio is the 640/480 = 1.333 (the 4:3), and the FOV’s ratio is the tan(65.2°/2)/tan(51.3°/2) = tan(32.6°)/tan(25.65°) = 0.639/0.480 = 1.331. The two are approximately equal (the difference is the arctan’s nonlinearity), and the exact’s relation is the tan(FOV_x/2)/tan(FOV_y/2) = W/H·(f_y/f_x), which is the W/H (the aspect’s ratio) for the f_x = f_y. The FOV’s ratio is the aspect’s ratio (the small approximation’s error), and the image’s shape is the FOV’s shape.

Self-explanation 8. Before reading on: the ray’s λ = 1 gives the (0.5, 0.2, 1.0) — the same as the WE-L.1.a’s original. Is the depth’s λ = 1 the specific value, or the arbitrary?

Answer. The λ = 1 is the arbitrary (the same pixel’s any λ gives the same projection), and the WE-L.1.a’s λ = 1 is the coincidence (the original point’s Z = 1.0 is the chosen’s λ). The ray’s parameterization is the λ’s arbitrary, and the specific λ is the additional constraint (the known object’s size, the stereo’s triangulation, the monocular depth’s network). The ray is the lost’s representation, and the λ is the recovered’s task.

WE-L.3.b — Completion problem

Input. The same camera, the pixel (u, v) = (100, 100) (the top-left’s corner).

Trace (final 25% as prompts).

1. The x_n = (100 - 320)/500 = -0.44, the y_n = (100 - 240)/500 = -0.28. ⟨your turn⟩
2. The camera’s ray is the λ [-0.44, -0.28, 1]ᵀ. ⟨your turn⟩
3. The ray’s direction’s angles: the atan2’s (-0.44, 1) ≈ -23.7°, the atan2(-0.28, 1) ≈ -15.6°. ⟨your turn⟩
4. The angles are within the FOV’s half (±32.6°, ±25.65°), and the pixel is inside the FOV. ⟨your turn⟩

Answers.

1. x_n = -0.44, y_n = -0.28. ✓
2. The ray λ [-0.44, -0.28, 1]ᵀ. ✓
3. The angles -23.7°, -15.6°. ✓
4. The pixel inside the FOV. ✓

The lesson. The top-left’s pixel’s normalized (-0.44, -0.28) is the negative (the left and the up), and the ray’s direction is the negative’s angle (the -23.7° and the -15.6°). The angles are within the FOV’s half (the ±32.6°, ±25.65°), and the pixel is inside the FOV’s range. The projection and the back-projection are the inverse’s pair, and the FOV’s boundary is the image’s edge.

WE-L.3.c — Faded: the stereo’s triangulation

Setup. The two cameras (the left at the origin, the right at the [0.1, 0, 0]’s baseline), the same pixel (570, 340) in the both (the object’s same pixel).

Question. What is the triangulated 3D point?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The left’s ray is the λ_L [0.5, 0.2, 1]ᵀ, the right’s ray is the [0.1, 0, 0]ᵀ + λ_R [0.5, 0.2, 1]ᵀ. The intersection’s λ_L [0.5, 0.2, 1] = [0.1, 0, 0] + λ_R [0.5, 0.2, 1], and the components’ equations: the 0.5 λ_L = 0.1 + 0.5 λ_R, the 0.2 λ_L = 0.2 λ_R, the λ_L = λ_R. The second gives the λ_L = λ_R, and the first gives the 0.5 λ_L = 0.1 + 0.5 λ_L, which is the 0 = 0.1 — the inconsistent! The same pixel in the both cameras is the parallel rays (the no intersection), and the object is at the infinity (the parallel’s no triangulation). The different pixels are the realistic (the stereo’s disparity), and the intersection is the finite’s λ.

The lesson. The stereo’s triangulation requires the different pixels in the two cameras (the disparity), and the same pixel is the parallel rays (the no finite intersection). The disparity d = u_L - u_R is the depth’s inverse (Z = f · b / d with the b the baseline), and the smaller the d (the farther the object), the larger the Z. The stereo’s depth is the triangulation’s result, and the monocular’s depth is the learned’s alternative.

WE-L.3.d — Contrastive near-miss: the back-projection without the camera’s rotation

Correct solution. The back-projection’s ray R x_n + C with the camera’s rotation R (the T_World^Cam’s R_{cam}^W).

Near-miss. The back-projection’s ray x_n + C (the no rotation, the camera’s frame’s identity).

Why the near-miss produces a plausible but incorrect number. The no rotation’s ray is the camera’s frame’s direction (the identity), and the world’s ray is the camera’s pose’s rotation’s application. The no rotation’s ray is the wrong (the camera’s orientation is ignored), and the symptom is the object’s wrong 3D position (the large error), the visual servoing’s wrong motion (the camera moves the wrong direction), and the triangulation’s wrong depth. The fix is the camera’s rotation’s application (the R x_n), and the verification is the ray’s reprojection (the pixel’s back-projection’s projection should be the same pixel).

The discrepancy, stated. The R x_n and the x_n differ by the camera’s rotation R, and the wrong’s ray is the camera’s frame’s (the identity’s), the correct’s is the world’s (the rotated’s). The camera’s orientation is the essential (the ray’s direction), and the no rotation is the wrong. The fix is the rotation’s application, and the verification is the reprojection’s consistency.

Self-explanation 9. Before reading on: the no-rotation’s x_n + C — if the camera’s rotation is the identity (the aligned with the world), does the no rotation work?

Answer. Yes — if the camera’s rotation is the identity (the camera’s frame is the world’s), the R x_n = x_n, and the no rotation coincides with the correct. The no rotation’s failure is the non-identity’s rotation (the camera’s orientation), and the symptom is the large error. The correct’s R x_n is the general (the any rotation), and the no rotation is the special’s identity’s case.

Part D — The Lens Distortion

§L.4 — Objects: the radial, the tangential, and the undistortion

The lens distortion is the deviation of the real camera from the pinhole’s ideal, and the two common models:

1. The radial distortion — the barrel (the k_1 < 0) or the pincushion (the k_1 > 0), the symmetric’s radial:

where r² = x_d² + y_d² (the normalized image point’s radius), and the k_1, k_2, k_3 are the radial’s coefficients (the calibrated).

2. The tangential distortion — the decentering of the lens (the misalignment), the tangential:

where the p_1, p_2 are the tangential’s coefficients.

The full distortion model:

The distortion’s effect is the systematic (the larger at the edges), and the undistortion is the correction (the x_undist is the ideal’s pinhole’s point). The calibration (the checkerboard) estimates the k_1, k_2, p_1, p_2, and the undistortion is the pre-processing (the before the projection’s geometry).

The running case’s camera: the typical’s k_1 ≈ -0.3, k_2 ≈ 0.1, p_1 ≈ 0, p_2 ≈ 0 (the mild’s barrel), and the undistortion’s effect is the tens of pixels at the edges.

§L.4.1 — Result: the distortion is the systematic’s error (the larger at the edges), and the undistortion is the required for the geometric’s accuracy — the omission’s bias is the radial’s k_1 r²’s magnitude

Claim. The distortion is the systematic’s (the deterministic, the calibrated), and its error is the k_1 r²’s magnitude (the larger at the edges, the zero at the center). The undistortion is the required for the geometric’s accuracy (the pose’s estimation, the triangulation, the visual servoing), and the omission’s bias is the radial’s error (the tens of pixels at the edges for the wide-angle lenses).

Proof sketch. The distortion’s radial r_d = r_u (1 + k_1 r_u² + ...) is the r_u’s perturbation, and the k_1 r_u² is the relative’s error (the k_1 r_u²’s magnitude at the radius). The undistortion’s correction is the r_u’s recovery (the r_d’s inverse), and the omission’s bias is the r_d - r_u’s magnitude (the k_1 r_u³). The wide-angle’s k_1 ≈ -0.3 gives the 30%‘s relative’s error at the r_u = 1, and the pixel’s error is the f · 0.3 (the 150 pixels for the f = 500).

Corollary. The undistortion is the calibrated’s correction (the checkerboard’s estimation), and the calibration’s accuracy is the undistortion’s accuracy. The no calibration is the no correction (the raw’s distortion), and the symptom is the systematic’s error (the larger at the edges), the reprojection’s residual’s pattern (the radial’s systematic).

Corollary. The modern’s machine’s learning’s perception (the deep’s network) learns the distortion’s correction implicitly (the training’s data includes the distorted images, and the network adapts), and the explicit’s undistortion is the geometric’s applications (the pose’s estimation, the triangulation) where the pixel’s accuracy is the required.

§L.4.2 — Worked-example ladder

WE-L.4.a — Fully worked: the radial distortion’s error

Input. The normalized image point (x_d, y_d) = (0.4, 0.3), the r² = 0.4² + 0.3² = 0.25, the k_1 = -0.3, the k_2 = 0.1, the k_3 = 0.

Trace (the undistorted point).

1. The radial’s factor 1 + k_1 r² + k_2 r⁴ = 1 + (-0.3)·0.25 + 0.1·0.0625 = 1 - 0.075 + 0.00625 = 0.93125.
2. The x_undist = 0.4 · 0.93125 = 0.3725.
3. The y_undist = 0.3 · 0.93125 = 0.2794.
4. The difference x_undist - x_d = -0.0275, y_undist - y_d = -0.0206.

Trace (the pixel’s error).

5. The pixel’s difference f · (-0.0275) = 500 · (-0.0275) = -13.75 pixels.
6. The pixel’s difference f · (-0.0206) = -10.3 pixels.

Output and interpretation. The distortion’s k_1 = -0.3 gives the 13.75 pixels’ error in the x and the 10.3 in the y for the (0.4, 0.3)’s normalized (the moderate’s radius). The larger’s r = 0.5 (the edge) gives the larger (the k_1 r² = -0.075 and the k_2 r⁴ = 0.00625’s sum is the -0.069, and the r = 0.5’s x = 0.5 gives the x_undist = 0.5·0.931 = 0.4655’s 17 pixels). The systematic’s error is the larger at the edges, and the undistortion’s correction is the required.

Self-explanation 10. Before reading on: the k_1 = -0.3’s negative is the barrel (the barrel’s distortion). What is the visual’s effect?

Answer. The barrel’s negative’s k_1 makes the edges’s pulled inward (the x_undist < x_d), and the image’s looks like the barrel (the center’s magnified, the edges’s compressed). The pincushion’s positive’s k_1 makes the edges’s pushed outward (the image’s looks like the pincushion). The visual’s distortion is the typical for the wide-angle’s lenses (the fisheye’s large’s barrel), and the undistortion is the **k_1’s correction (the edges’s pulled back to the pinhole’s straight’s lines).

WE-L.4.b — Completion problem

Input. The same point (0.4, 0.3), but the k_1 = +0.3 (the pincushion).

Trace (final 25% as prompts).

1. The radial’s factor 1 + 0.3·0.25 + 0.1·0.0625 = 1.075 + 0.00625 = 1.08125. ⟨your turn⟩
2. The x_undist = 0.4 · 1.08125 = 0.4325, the y_undist = 0.3 · 1.08125 = 0.3244. ⟨your turn⟩
3. The difference +0.0325, +0.0244 (the positive, the edges’s pushed outward). ⟨your turn⟩
4. The pixel’s difference +16.25, +12.2 (the positive). ⟨your turn⟩

Answers.

1. The factor 1.08125. ✓
2. The x_undist = 0.4325, y_undist = 0.3244. ✓
3. The difference +0.0325, +0.0244. ✓
4. The pixel’s +16.25, +12.2. ✓

The lesson. The k_1 = +0.3’s pincushion pushes the edges’s outward (the positive’s difference), and the sign’s opposite of the barrel’s. The undistortion’s direction is the sign’s opposite (the barrel’s edges’s pulled inward’s correction is the outward), and the distortion’s calibrated’s k_1 is the sign’s source.

WE-L.4.c — Faded: the tangential distortion

Setup. The same point (0.4, 0.3), the p_1 = 0.01, the p_2 = 0.005, the no radial (k_1 = k_2 = 0).

Question. What is the tangential’s correction?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The r² = 0.25, and the tangential’s x_undist = 0.4 + 2·0.01·0.4·0.3 + 0.005·(0.25 + 2·0.16) = 0.4 + 0.0024 + 0.005·0.57 = 0.4 + 0.0024 + 0.00285 = 0.40525. The y_undist = 0.3 + 0.01·(0.25 + 2·0.09) + 2·0.005·0.4·0.3 = 0.3 + 0.01·0.43 + 0.0012 = 0.3 + 0.0043 + 0.0012 = 0.3055. The difference is the +0.00525 and the +0.0055 — the small (the tangential’s small’s p_1, p_2), and the pixel’s difference is the 2.6 and the 2.75 (the smaller than the radial’s).

The lesson. The tangential’s effect is the small (the p_1, p_2’s small), and the radial’s effect is the larger (the k_1’s larger). The modern’s calibration’s p_1, p_2 is the near zero (the lens’s decentering is the small), and the radial is the dominant. The undistortion’s full’s model (the radial + the tangential) is the standard, and the radial’s only is the approximation.

WE-L.4.d — Contrastive near-miss: the no distortion’s correction

Correct solution. The undistortion’s correction (the radial + the tangential), the calibrated’s k_1, k_2, p_1, p_2.

Near-miss. The no distortion’s correction (the raw’s pixel, the no k_1, k_2, p_1, p_2).

Why the near-miss produces a plausible but incorrect number. The raw’s pixel is the distorted (the actual’s lens’s projection), and the pinhole’s assumption is the wrong (the undistorted’s). The reprojection’s residual is the radial’s systematic (the larger at the edges), and the pose’s estimation is the biased (the camera’s pose’s wrong). The symptom is the systematic’s error (the reprojection’s pattern), and the fix is the undistortion’s calibrated’s correction.

The discrepancy, stated. The x_undist and the x_d differ by the distortion’s correction (the k_1 r² and the k_2 r⁴), and the no correction’s pixel is the x_d’s f · x_d, the correct’s is the f · x_undist. The difference is the f · (x_undist - x_d) (the f · k_1 r²’s magnitude), and the edge’s error is the large. The fix is the undistortion, and the verification is the reprojection’s residual’s no’s systematic.

Self-explanation 11. Before reading on: the no distortion’s k_1 = -0.3’s error at the edge (r = 0.5) is the 17 pixels — is the neural’s network’s perception’s affected?

Answer. The neural’s network’s perception (the deep’s detection, the VLA’s fine-tuning) learns the distortion’s correction implicitly (the training’s data includes the distorted images, and the network adapts). The 17 pixels’ error is the relative’s (the image’s 640’s width), and the network can tolerate the mild’s (the k_1 = -0.3), but the wide-angle’s (the fisheye’s k_1 = -0.8) is the large (the 50 pixels), and the explicit’s undistortion is the required (the geometric’s tasks). The network’s implicit’s learning and the explicit’s undistortion are the complementary (the detection’s vs. the pose’s estimation’s).

Part E — Failure Taxonomy, Lab, Cheat Sheet, Exercises, Exit Test

§L.5 — Failure taxonomy

§L.6 — The lab

§L.6.1 — Task

Implement the projection and the back-projection for the running case’s camera, verify the direction (the camera’s origin’s projection, the point in front’s center), the intrinsics (the K from the specifications), and the distortion’s undistortion (the synthetic’s k_1, k_2); break the four ways and diagnose.

§L.6.2 — Conditions

Python 3.10+, PyTorch 2.2+, CPU. Seed 42, dtype = float64. Runtime under 3 minutes. The camera’s K = [[500, 0, 320], [0, 500, 240], [0, 0, 1]], the resolution 640 × 480, the distortion’s k_1 = -0.3, k_2 = 0.1, p_1 = p_2 = 0, the object’s X^W = [0.5, 0.2, 1.0] (the camera’s origin with the identity rotation).

§L.6.3 — Controls held fixed

The seed, dtype, the K, the resolution, the distortion, and the object’s X^W are fixed. The variants are the wrong intrinsics (the f = 500 → 400), the wrong direction (T_World^Cam instead of the T_Cam^World), the Z = 0’s point ([0.5, 0.2, 0]), and the no distortion’s correction.

§L.6.4 — Metrics table

§L.6.5 — Pre-registered hypotheses

• H1. The object [0.5, 0.2, 1.0]’s projection is the (570, 340) with the reprojection_error < 0.1 pixels. The prediction is the pixel’s pair.
• H2. The camera’s origin’s projection (the [0, 0, 0]’s world’s point) is the NaN (the Z = 0’s division). The prediction is the NaN.
• H3. The point in front [0, 0, 1]’s projection is the (c_x, c_y) = (320, 240). The prediction is the pixel’s center.
• H4. The FOV is the 65.2° × 51.3° (the computed). The prediction is the pair.
• H5. The distortion’s correction for the (0.4, 0.3)’s normalized is the -13.75 pixels (the x). The prediction is the value.

§L.6.6 — The break-it sequence

1. The wrong intrinsics (silent). The f = 400 (the wrong). Predicted symptom: the projection’s offset (the systematic), the larger at the edges. Actual: as predicted — silent.
2. The wrong direction (silent). The T_World^Cam as the extrinsic. Predicted symptom: the projection’s mirrored (the left is the right), the camera’s origin’s projection is the defined (the not NaN). Actual: as predicted — silent.
3. The Z = 0 (loud NaN). The point [0.5, 0.2, 0] (the image plane). Predicted symptom: the division’s NaN or inf. Actual: as predicted — loud.
4. The no distortion (silent). The no undistortion. Predicted symptom: the systematic’s error (the radial’s), the larger at the edges. Actual: as predicted — silent.

§L.6.7 — Acceptance criteria

§L.6.8 — What good looks like

A strong submission has five properties:

1. The K is derived from the specifications (the f, the dx, the resolution), and the projection matches the hand-computed (570, 340).
2. The direction’s check is demonstrated (the camera’s origin’s NaN, the front point’s center), and the wrong direction’s symptom is shown.
3. The back-projection is verified (the x → ray → x’s consistency), and the depth’s ambiguity is demonstrated (the two different λ’s give the same pixel).
4. The distortion’s correction is demonstrated (the raw vs. the undistorted’s difference), and the edge’s larger error is shown.
5. The four break-its are predicted with the symptoms (the offset, the mirrored, the NaN, the systematic) before running.

§L.6.9 — Runnable lab code

# lab_2_1_1.py — The camera model, the projection, the back-projection, and the distortion
# Requires: Python 3.10+, PyTorch 2.2+

import torch
import numpy as np

torch.manual_seed(42); dtype, device = torch.float64, torch.device("cpu")

# --- The camera model --------------------------------------------------
class PinholeCamera:
    def __init__(self, fx, fy, cx, cy, W, H, dist_coeffs=None):
        self.K = torch.tensor([[fx, 0, cx], [0, fy, cy], [0, 0, 1]], dtype=dtype)
        self.fx, self.fy, self.cx, self.cy = fx, fy, cx, cy
        self.W, self.H = W, H
        self.dist = dist_coeffs if dist_coeffs else [0.0, 0.0, 0.0, 0.0, 0.0]
    
    def project(self, X_cam):
        """X_cam: (3,) camera-frame point -> (u, v) or None if Z <= 0."""
        Z = X_cam[2]
        if Z <= 0:
            return None
        x_n = X_cam[0] / Z
        y_n = X_cam[1] / Z
        u = self.fx * x_n + self.cx
        v = self.fy * y_n + self.cy
        return torch.tensor([u, v], dtype=dtype)
    
    def undistort(self, x_dist):
        """x_dist: (2,) normalized distorted point -> (2,) undistorted."""
        x, y = x_dist
        r2 = x*x + y*y
        k1, k2, p1, p2, k3 = self.dist
        radial = 1 + k1*r2 + k2*r2*r2 + k3*r2*r2*r2
        x_undist = x * radial + 2*p1*x*y + p2*(r2 + 2*x*x)
        y_undist = y * radial + p1*(r2 + 2*y*y) + 2*p2*x*y
        return torch.tensor([x_undist, y_undist], dtype=dtype)
    
    def back_project(self, uv, R, C, lam=1.0):
        """uv: (2,) pixel, R: (3,3) camera rotation, C: (3,) camera center."""
        x_n = (uv[0] - self.cx) / self.fx
        y_n = (uv[1] - self.cy) / self.fy
        ray_cam = torch.tensor([x_n, y_n, 1.0], dtype=dtype) * lam
        ray_world = R @ ray_cam + C
        return ray_world
    
    def fov(self):
        return 2 * np.arctan(self.W / (2*self.fx)) * 180 / np.pi, 2 * np.arctan(self.H / (2*self.fy)) * 180 / np.pi

# --- The test camera ---------------------------------------------------
cam = PinholeCamera(500, 500, 320, 240, 640, 480, [0.0, 0.0, 0.0, 0.0, 0.0])

# H1: the projection of the object
X_cam = torch.tensor([0.5, 0.2, 1.0], dtype=dtype)
uv = cam.project(X_cam)
print(f"H1: projection of {X_cam.tolist()} = {uv.tolist()} (expected [570, 340])")

# H2: the camera's origin (Z = 0)
X_origin = torch.tensor([0.0, 0.0, 0.0], dtype=dtype)
print(f"H2: camera origin -> {cam.project(X_origin)} (expected None/NaN)")

# H3: the point in front (Z = 1, X = Y = 0)
X_front = torch.tensor([0.0, 0.0, 1.0], dtype=dtype)
print(f"H3: point in front -> {cam.project(X_front).tolist()} (expected [320, 240])")

# H4: the FOV
print(f"H4: FOV = {cam.fov()} degrees (expected ~65.2 x 51.3)")

# H5: the distortion
x_n_dist = torch.tensor([0.4, 0.3], dtype=dtype)
cam_dist = PinholeCamera(500, 500, 320, 240, 640, 480, [-0.3, 0.1, 0.0, 0.0, 0.0])
x_n_undist = cam_dist.undistort(x_n_dist)
print(f"H5: undistorted {x_n_dist.tolist()} -> {x_n_undist.tolist()}")
print(f"    pixel difference = {(x_n_undist - x_n_dist) * 500}")

# --- Break-it #1: the wrong intrinsics --------------------------------
cam_wrong = PinholeCamera(400, 400, 320, 240, 640, 480, [0.0, 0.0, 0.0, 0.0, 0.0])
uv_wrong = cam_wrong.project(X_cam)
print(f"\n[break-it #1] wrong f=400: {uv_wrong.tolist()} vs. correct {uv.tolist()}")

# --- Break-it #2: the wrong direction (identity as the "pose") ------
# Simulate: the camera's pose is the identity, and the "extrinsic" is the wrong-direction
R_pose = torch.eye(3, dtype=dtype)  # the identity rotation
C = torch.tensor([0.0, 0.0, 0.0], dtype=dtype)  # at the origin
# The wrong direction's "extrinsic" is the pose (the camera-to-world), the correct is the world-to-camera
# Since the pose is the identity, the two are the same; use a non-identity case
R_pose_nonid = torch.tensor([[0., -1, 0], [1, 0, 0], [0, 0, 1]], dtype=dtype)
C_nonid = torch.tensor([0.5, 0.0, 0.0], dtype=dtype)
# The correct extrinsic: the world-to-camera = R^T, -R^T t
R_correct = R_pose_nonid.T
C_correct = -R_correct @ C_nonid
# The correct camera point: R_correct @ X_world + C_correct... wait, the extrinsic's rotation and translation
X_world = torch.tensor([0.5, 0.2, 1.0], dtype=dtype)
X_cam_correct = R_correct @ X_world + C_correct
# The wrong (the pose): R_pose @ X_world + C_pose (the pose's direct application)
X_cam_wrong = R_pose_nonid @ X_world + C_nonid
print(f"\n[break-it #2] correct X_cam = {X_cam_correct.tolist()}, wrong X_cam = {X_cam_wrong.tolist()}")
print(f"    correct pixel = {cam.project(X_cam_correct).tolist() if cam.project(X_cam_correct) is not None else 'None'}")
print(f"    wrong pixel = {cam.project(X_cam_wrong).tolist() if cam.project(X_cam_wrong) is not None else 'None'}")

# --- Break-it #3: the Z = 0 -----------------------------------------
print(f"\n[break-it #3] Z=0 projection -> {cam.project(torch.tensor([0.5, 0.2, 0.0], dtype=dtype))} (expected None)")

# --- Break-it #4: the no distortion ---------------------------------
x_n_dist_test = torch.tensor([0.4, 0.3], dtype=dtype)
# No distortion's "correction" is the raw (the identity)
print(f"\n[break-it #4] no-distortion raw = {x_n_dist_test.tolist()}, corrected = {x_n_undist.tolist()}")
print(f"    error (pixels) = {((x_n_undist - x_n_dist_test) * 500).tolist()}")

# --- Back-projection's consistency ----------------------------------
uv_test = torch.tensor([570.0, 340.0], dtype=dtype)
R_identity = torch.eye(3, dtype=dtype)
C_identity = torch.tensor([0.0, 0.0, 0.0], dtype=dtype)
ray = cam.back_project(uv_test, R_identity, C_identity, lam=1.0)
uv_reproj = cam.project(ray)
print(f"\n[back-projection] uv = {uv_test.tolist()} -> ray = {ray.tolist()} -> reproj = {uv_reproj.tolist()}")

# --- Summary ---------------------------------------------------------
print("\n--- lab 2.1.1 summary ---")
print(f"H1: projection = {uv.tolist()} (expected [570, 340])")
print(f"H2: camera origin -> {cam.project(X_origin)} (expected None)")
print(f"H3: front point -> {cam.project(X_front).tolist()} (expected [320, 240])")
print(f"H4: FOV = {cam.fov()}")
print(f"H5: distortion = {((x_n_undist - x_n_dist_test) * 500).tolist()}")

§L.7 — The cheat sheet

§L.8 — Exercises with solutions

E 2.1.1.1 — The projection (numerical). The K = [[500, 0, 320], [0, 500, 240], [0, 0, 1]], the X_cam = [0.3, 0.1, 0.6]. Compute the pixel.

S 2.1.1.1. The x_n = 0.5, 0.167, the u = 500·0.5 + 320 = 570, the v = 500·0.167 + 240 ≈ 323.5. The lesson: the Z = 0.6’s smaller (the closer), and the u = 570’s same as the Z = 1.0’s if the X’s proportional (the X/Z = 0.5), the perspective’s scale’s effect.

E 2.1.1.2 — The intrinsic from the specs (numerical). The f = 5 mm, the dx = 2.5 µm, the resolution = 1024 × 768. Compute the K.

S 2.1.1.2. The f_x = 5/0.0025 = 2000, the f_y = 2000, the c_x = 512, the c_y = 384. The lesson: the same formula, and the pixel’s size’s smaller gives the larger f_x.

E 2.1.1.3 — The direction’s check (diagnostic). The camera’s origin’s projection is the (320, 240) instead of the NaN. What is the bug?

S 2.1.1.2. The wrong direction — the T_World^Cam (the pose) is used as the extrinsic (T_Cam^World). The camera’s origin in the world is the T_World^Cam’s translation, and the projection with the wrong extrinsic gives the defined pixel (the camera’s origin as a world point, projected). The fix is the inverse. The lesson: the direction’s check is the camera’s origin’s undefined’s projection, and the defined’s is the wrong.

E 2.1.1.4 — The back-projection (numerical). The pixel (100, 100), the camera at the origin with the identity. Compute the ray.

S 2.1.1.4. The x_n = (100 - 320)/500 = -0.44, the y_n = (100 - 240)/500 = -0.28, the ray λ [-0.44, -0.28, 1]. The lesson: the top-left’s negative’s normalized, and the ray’s direction is the negative’s angle.

E 2.1.1.5 — The FOV (numerical). The f = 500, the W = 640. Compute the FOV.

S 2.1.1.5. The 2 arctan(640/1000) = 2 arctan(0.64) ≈ 65.2°. The lesson: the larger f is the narrower FOV (the f = 1000 gives the 2 arctan(0.32) ≈ 35.4°).

E 2.1.1.6 — The distortion (numerical). The k_1 = -0.3, the (x, y) = (0.4, 0.3). Compute the undistorted.

S 2.1.1.6. The r² = 0.25, the radial = 1 - 0.3·0.25 = 0.925, the x_undist = 0.4·0.925 = 0.37, the y_undist = 0.3·0.925 = 0.2775. The lesson: the barrel’s negative’s k_1 pulls the edges inward, and the pixel’s difference is the -15 and -11.

E 2.1.1.7 — The no distortion (diagnostic). The reprojection’s residual is the radial’s (the larger at the edges). What is the fix?

S 2.1.1.7. The undistortion’s calibrated’s correction (the k_1, k_2, p_1, p_2 from the checkerboard’s calibration). The raw’s pixel is the distorted, and the pinhole’s assumption is the wrong. The lesson: the systematic’s radial’s residual is the no distortion’s signature.

E 2.1.1.8 — The stereo’s triangulation (constructive, transfer probe). Write the disparity’s depth’s formula.

S 2.1.1.8. The Z = f · b / d, with the f the focal (the pixels), the b the baseline, the d the disparity (u_L - u_R). The smaller d (the farther) is the larger Z. The lesson: the disparity’s inverse’s relation, and the larger’s b gives the larger’s d (the more sensitive).

§L.9 — Exit test

You are ready for Lesson 2.1.2 (Object Detection and Pose Estimation) when you can, without notes:

1. State the pinhole model and the projection equation s x = K [R | t] X.
2. State the intrinsic K’s components and the f_x = f/dx’s formula.
3. State the extrinsic [R | t]’s direction (the world-to-camera) and its inverse (the pose).
4. Verify the direction by the camera’s origin’s projection (the NaN’s) and the front point’s (the center’s).
5. State the projection’s depth’s loss and the back-projection’s ray.
6. State the FOV’s formula and the f’s effect.
7. State the radial and the tangential distortion’s formulas.
8. Compute the undistorted point for a specific (x, y) and k_1.
9. Diagnose the wrong intrinsics, the wrong direction, the Z = 0, and the no distortion from their symptoms.
10. State the stereo’s triangulation’s disparity’s Z = f·b/d.
11. State the back-projection’s rotation’s application (the R x_n), and the no rotation’s failure.
12. State the K’s and the distortion’s calibration’s role, and the no calibration’s symptom.

If any item is not answerable without notes, return to the section that introduced it: 1 to §L.1, 2 to §L.1, 3 to §L.2, 4 to §L.2.1, 5 to §L.3, 6 to §L.3, 7 to §L.4, 8 to WE-L.4.a, 9 to §L.5, 10 to WE-L.3.c, 11 to §L.3.1, 12 to §L.4.1.

End of Lesson 2.1.1. The learner has now completed the camera model’s foundation: the pinhole’s projection, the intrinsics, the extrinsics, the FOV, the back-projection, and the distortion. The next lesson, 2.1.2 — Object Detection and Pose Estimation, uses the camera model to detect the objects in the image and estimate their 6D poses from the 2D observations — the perception’s first half, and the 1.1.4’s pick-and-place’s input.