X2.1.2

Lesson 2.1.2 — Object Detection and 6D Pose Estimation: From Bounding Boxes to T_Cam^Obj

Week 9 · Prerequisites: 2.1.1, 1.2.1, 0.2.1, 0.2.2, 0.2.4, 0.3.4.

Prior results used. This lesson uses 2.1.1’s pinhole model, the projection s x = K [R | t] X, the extrinsics’ direction, the distortion’s correction, and the depth’s ambiguity; 1.2.1’s ArmEnv’s observation space (the image as the input); 0.2.1’s SE(3) composition and inverse; 0.2.2’s SO(3) and the rotation’s representations; 0.2.4’s from-to convention and the frame audit; and 0.3.4’s Mahalanobis gate (the outlier’s rejection). Each is restated at the point of use.

Hands forward. This lesson hands forward one capability: the learner can take an RGB-D image, detect the target object with a 2D detector, establish the 2D-3D correspondences, solve the perspective-n-point problem to recover the 6D pose T_Cam^Obj, refine the pose with the depth, and diagnose the four pose-estimation bugs (wrong correspondence, symmetric pose’s ambiguity, PnP’s planar degeneracy, and the depth’s scale’s mismatch) from their reprojection’s and ADD’s symptoms — the capability that 2.1.3 (visual servoing), 1.1.4 (the pick-and-place’s perception), and the entire 3.1 (VLA) pipeline assume.

Running case. The same 6-DOF MuJoCo arm from 1.1.1–2.1.1, with the camera on the end-effector (the intrinsics K = [[500, 0, 320], [0, 500, 240], [0, 0, 1]], the 640 × 480’s image) and the target object — a cube of side 0.05 m at the world position p^W = [0.5, 0.2, 0.3] with the known orientation (the R_W^Obj = Rz(π/4)). The detection’s task is to find the cube’s 2D bounding box in the image, the pose estimation’s task is to recover the T_Cam^Obj from the 2D correspondences (the cube’s 8 corners), and the refinement’s task is to use the depth to correct the PnP’s pose. The lesson’s central result is that the PnP is the minimal’s solver (the 6 correspondences’ 6-DOF’s pose), the PnP’s solution is the sensitive to the correspondence’s accuracy (the 1-pixel’s error is the ~1 cm’s pose’s error at the 1 m’s depth), and the symmetric’s object’s pose’s ambiguity is the fundamental (the multiple poses give the same image), and the depth’s refinement is the required for the robustness.

§L.0 — Lesson map, objectives, and misconceptions

§L.0.1 — Lesson map and word budget

§L.0.2 — Objectives

By the end of this lesson you will be able to do eight things:

1. State the 2D detection’s output ((u, v, w, h)’s bounding box, the class, the confidence), the IoU’s definition, and the NMS’s role (the duplicate’s removal), and distinguish the anchor-based (the YOLO’s, the Faster R-CNN’s) from the anchor-free (the FCOS, the CenterNet) and the transformer-based (the DETR) detectors.
2. State the 6D pose’s estimation’s two approaches — the direct (the regression’s T_Cam^Obj from the image) and the indirect (the correspondence’s 2D-3D, the PnP) — and state the indirect’s accuracy and the direct’s speed.
3. Derive the PnP’s problem (the n correspondences’ (X_i^Obj, x_i)’s T_Cam^Obj’s solution) and state the minimal (n = 4, the P3P’s up to 4 solutions) vs. the overdetermined (n > 4, the least-squares’s unique), and implement the EPnP’s efficient solution.
4. State the PnP’s planar’s degeneracy (the coplanar points, the two solutions’ ambiguous), and the symmetric object’s pose’s ambiguity (the multiple R’s produce the same image).
5. State the ADD (the average distance of the model points under the estimated and the ground-truth poses) and the ADD-S (the symmetric’s closest-point’s distance), and compute both for a specific pose’s pair.
6. Implement the depth’s refinement (the ICP’s iteration: the correspondence’s (p^Obj, p^Cam), the SVD’s rigid’s alignment, the pose’s update), and state the ICP’s convergence’s condition (the local’s minimum, the initial’s close).
7. Diagnose the four pose-estimation bugs — wrong correspondence (the large reprojection’s error, the ADD’s large), the symmetric pose’s ambiguity (the 180°‘s flip), the PnP’s planar’s degeneracy (the two solutions’ close), and the depth’s scale’s mismatch (the reprojection’s consistent but the depth’s wrong) — from their symptoms.
8. Produce a working detect → PnP → refine pipeline for the running case’s cube, with the correspondence’s construction (the cube’s 8 corners), the PnP’s solution (the EPnP or the iterative), the depth’s refinement (the ICP), and the verification (the ADD < 0.01 m, the reprojection < 2 pixels).

§L.0.3 — Misconception inventory

Each misconception is displaced in the section named, and each is tagged in the concept check that follows.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

This is the twenty-third lesson. The prompt requires drawing from lessons two to five steps back; 2.1.1 is one step back and is excluded. The warm-up draws from 1.2.5 (two steps back), 1.2.4 (three), 1.2.3 (four), and 1.2.2 (five). The geometric prerequisites (0.2.1–0.2.4) and 0.3.4 are more than five steps back and are restated at the point of use in Parts B–D.

Warm-up 1 (from 1.2.5, the replay buffer and the sample efficiency). State the replay buffer’s role and the sample efficiency’s source. Answer: the buffer stores the past transitions and the off-policy’s sample is the reuse; the sample efficiency is the many gradient steps per transition. How this is used: §L.4’s ICP’s refinement is the iterative’s (the many’s correspondence’s updates), and the convergence’s speed is the initial’s quality — the PnP’s output is the good’s initial, and the ICP’s refinement is the few’s iterations.

Warm-up 2 (from 1.2.4, the trust region and the clipped objective). State the PPO’s clipping’s role and the min’s pessimism. Answer: the clip bounds the ratio to the [1−ε, 1+ε], and the min is the pessimistic bound (the gradient zero for the advantage-positive and ratio > 1+ε). How this is used: §L.3’s ADD’s evaluation is the bound on the pose’s error (the worst-case), and the ADD-S’s symmetric’s min is the same’s pessimism (the closest-point’s min is the lower’s bound on the pose’s error for the symmetric object).

Warm-up 3 (from 1.2.3, the Gaussian policy and the log-std). State the Gaussian policy’s log σ’s role. Answer: the log σ is the exploration’s scale (the larger for the uncertain), and the log σ’s gradient is the (a-μ)²/σ² − 1 (the larger deviation increases the std). How this is used: §L.2’s PnP’s solution is the sensitive to the correspondences’ accuracy, and the uncertainty’s propagation (the δt ≈ δx · Z/f) is the same’s structure as the policy’s σ’s effect (the larger Z is the larger error, the larger σ is the larger spread).

Warm-up 4 (from 1.2.2, the value functions and the Bellman). State the Q/V and the Bellman’s Q(s, a) = r + γ E[V(s')]. Answer: the Q is the state-action’s value, the V is the state’s, and the Bellman is the recursive. How this is used: §L.4’s ICP’s iteration is the recursive’s (the pose’s update from the correspondences, the converged’s fixed point), and the convergence’s condition (the local minimum, the initial close) is the same’s structure as the value iteration’s (the contraction, the initial’s any).

Part A — 2D Detection: Bounding Boxes, IoU, and NMS

§L.1 — Objects: the bounding box, the IoU, and the NMS

The 2D object detection’s output is a set of bounding boxes (u, v, w, h) — the box’s top-left’s (u, v), the width w, and the height h — with the class c and the confidence p. The detection’s task is to locate the objects in the image, and the output is the input to the pose’s estimation (the region’s crop).

The Intersection over Union (IoU) is the overlap’s measure:

The IoU’s [0, 1] range is the no overlap (the 0) to the exact (the 1), and the threshold (the 0.5 for the PASCAL VOC, the 0.75 for the strict’s COCO) is the matching’s criterion (the true positive vs. the false’s).

The Non-Maximum Suppression (NMS) is the duplicate’s removal:

1. Sort the detections by the confidence (the highest’s first).
2. Keep the highest’s detection, and remove the others whose IoU with the kept’s is above the threshold (the 0.5).
3. Repeat with the remaining’s detections.

The NMS’s role is the duplicate’s removal — the detector often produces the many boxes for the same object (the anchors’ or the grid cells’ multiple), and the NMS keeps the best and removes the duplicates. The soft-NMS (the decay’s confidence rather than the remove) and the learned’s NMS (the transformer’s set’s prediction) are the modern’s alternatives.

The detector’s families:

• Anchor-based (the YOLO’s early versions, the Faster R-CNN, the SSD): the predefined’s anchor’s boxes at each grid’s cell, and the network predicts the offset and the class for each anchor. The anchor’s hyperparameters (the sizes, the aspect ratios) are the tuned’s, and the dense’s anchors are the slow’s NMS.
• Anchor-free (the FCOS, the CenterNet, the YOLOX’s): the per-pixel’s prediction (the center’s heatmap, the size’s regression, the offset’s), and the no’s anchors. The simpler’s hyperparameters and the faster’s NMS.
• Transformer-based (the DETR’s, the Deformable’s DETR): the set’s prediction (the fixed’s number of the queries, the bipartite’s matching), and the no’s NMS (the set’s prediction is the unique’s). The slower’s convergence (the many’s epochs) and the better’s end-to-end.

The running case’s detector is the YOLO’s (the fast’s, the real-time’s, the sufficient’s for the cube’s detection), and the output is the cube’s bounding box (u_min, v_min, u_max, v_max) with the confidence > 0.9.

§L.1.1 — Result: the detection’s IoU and the NMS are the evaluation’s and the post-processing’s, and the detection’s accuracy is the pose’s input’s quality — the larger’s IoU is the better’s pose’s crop

Claim. The detection’s IoU is the overlap’s measure, and the NMS is the duplicate’s removal. The detection’s accuracy (the IoU with the ground-truth’s box) is the pose’s input’s quality — the larger’s IoU is the better’s crop (the pose’s estimation’s region), and the smaller’s IoU (the poor’s box) is the pose’s failure’s risk (the wrong’s correspondences).

Proof sketch. The detection’s box’s crop is the pose’s input, and the crop’s content is the object (the target) if the box is the accurate’s. The IoU’s large (the > 0.7) is the box’s accurate’s (the content’s mostly the object), and the IoU’s small (the < 0.5) is the box’s poor’s (the content’s mostly the background or the other’s object). The pose’s estimation’s accuracy depends on the crop’s content, and the detection’s accuracy is the pose’s upper’s bound.

Corollary. The detection’s accuracy is the necessary’s condition for the pose’s, and the pose’s failure is the detection’s failure (the wrong’s box). The two’s separation is the design’s principle (the detection’s model and the pose’s model are the separate’s), and the end-to-end’s (the detection and the pose’s joint’s) is the alternative (the harder’s training, the better’s coupling).

Corollary. The modern’s foundation’s (the SAM’s segment, the Grounding’s DINO’s detect, the FoundationPose’s pose) combines the detection, the segmentation, and the pose’s in the single’s pipeline, and the two’s separation is the classical’s (the YOLO + the PnP), the foundation’s is the end-to-end’s (the learned’s, the zero-shot’s).

§L.1.2 — Worked-example ladder

WE-L.1.a — Fully worked: the IoU for two boxes

Input. The box B_1 = (100, 100, 80, 60) (the u, v, w, h), the box B_2 = (120, 110, 80, 60).

Trace (the intersection).

1. The B_1’s range: the [100, 180] × [100, 160].
2. The B_2’s range: the [120, 200] × [110, 170].
3. The intersection’s [120, 180] × [110, 160] (the overlap’s [max(100,120), min(180,200)] × [max(100,110), min(160,170)]).
4. The intersection’s area 60 × 50 = 3000.
5. The union’s area 80·60 + 80·60 − 3000 = 4800 + 4800 − 3000 = 6600.
6. The IoU 3000/6600 ≈ 0.455.

Output and interpretation. The IoU’s 0.455 is the moderate’s (the partial’s overlap), and the threshold’s 0.5 is the close’s (the IoU is just below the PASCAL’s 0.5). The two’s boxes are the same’s size but the offset’s (the 20 in u, the 10 in v), and the IoU is the offset’s measure. The NMS’s threshold (the 0.5) would keep the both (the IoU is below the 0.5), and the detector’s two’s predictions would be the two’s outputs (the not’s duplicate).

Self-explanation 1. Before reading on: in step 6, the IoU’s 0.455 — if the offset were the 0 (the same’s boxes), the IoU would be the 1.0. If the offset were the 80 (the no’s overlap), the IoU would be the 0. What is the IoU’s sensitivity?

Answer. The IoU’s sensitivity is the nonlinear — the small’s offset (the 10’s) gives the high’s IoU (the 0.7’s), and the large’s offset (the 40’s) gives the low’s IoU (the 0.2’s). The IoU’s threshold’s 0.5 is the moderate’s (the ~20’s offset for the 80 × 60’s box), and the NMS’s threshold’s 0.5 is the same’s. The detector’s accuracy is the IoU’s large (the > 0.7), and the pose’s input’s quality is the IoU’s large.

Self-explanation 2. Before reading on: the NMS’s threshold’s 0.5 — if the two’s boxes are the same’s object, the NMS would keep the one’s (the highest’s confidence) and remove the other’s. If the threshold’s 0.3, the NMS would remove the more’s (the aggressive). What is the trade-off?

Answer. The NMS’s threshold’s trade-off is the duplicate’s removal vs. the close’s objects’ keeping — the small’s threshold (the 0.3) removes the duplicate’s aggressively (the good’s for the isolated’s objects, the bad’s for the crowded’s scene’s close’s objects), and the large’s (the 0.7) keeps the duplicate’s (the bad’s for the same’s object, the good’s for the close’s objects). The standard’s 0.5 is the balanced, and the scene’s crowded’s needs the larger’s (the 0.6–0.7).

WE-L.1.b — Completion problem

Input. The same’s B_1, the box B_2 = (140, 120, 80, 60) (the larger’s offset).

Trace (final 25% as prompts).

1. The intersection’s [140, 180] × [120, 160] (the area 40 × 40 = 1600). ⟨your turn⟩
2. The union’s 4800 + 4800 − 1600 = 8000. ⟨your turn⟩
3. The IoU 1600/8000 = 0.2. ⟨your turn⟩
4. The NMS’s threshold’s 0.5 would keep the both (the IoU is below the 0.5). ⟨your turn⟩

Answers.

1. The area 1600. ✓
2. The union 8000. ✓
3. The IoU 0.2. ✓
4. The NMS’s keep the both. ✓

The lesson. The larger’s offset (the 40 in u) gives the smaller’s IoU (the 0.2), and the two’s boxes are the different’s objects (the not’s duplicate). The NMS’s threshold’s 0.5 is the moderate, and the box’s accuracy is the IoU’s sensitivity.

WE-L.1.c — Faded: the running case’s detection

Setup. The running case’s cube at the p^W = [0.5, 0.2, 0.3], the camera at the origin (the identity), the K = [[500, 0, 320], [0, 500, 240], [0, 0, 1]], the cube’s side 0.05 m.

Question. What is the cube’s bounding box (the 2D’s) with the detection’s output?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The cube’s 8 corners in the camera’s frame are the p^W ± [±0.025, ±0.025, ±0.025] (the rotated by the R_W^Obj = Rz(π/4)). The projection of the 8 corners gives the pixel’s 8 points, and the bounding box is the min/max’s (u_min, v_min, u_max, v_max). The cube’s center [0.5, 0.2, 0.3] projects to the (500·0.5/0.3 + 320, 500·0.2/0.3 + 240) = (1153, 573) — outside the 640 × 480’s image! The cube’s center is the outside, and the cube’s some’s corners are the inside. The detection’s box is the partial (the clipped by the image’s edge), and the pose’s estimation’s correspondences are the only the inside’s corners (the partial’s occlusion’s).

The lesson. The running case’s cube is at the z = 0.3 (the close’s), and the projection’s u = 500·0.5/0.3 + 320 = 1153 is the outside the 640’s image (the cube’s center is the right’s of the image). The cube’s visible’s part is the left’s side (the some’s corners are the inside), and the detection’s box is the partial. The pose’s estimation’s correspondences are the only the visible’s corners (the 4–6’s, the not the 8’s), and the PnP’s overdetermined’s (the n ≥ 4’s) is the feasible (the 4’s minimal’s). The z = 0.3’s close is the challenging’s (the large’s perspective’s, the partial’s occlusion’s), and the z = 1.0’s far is the easier’s (the full’s cube’s visible’s).

WE-L.1.d — Contrastive near-miss: the anchor-based detector’s many boxes

Correct solution. The anchor-based’s detector produces the many’s boxes for the same’s object (the anchors’s multiple), and the NMS removes the duplicate’s.

Near-miss. The no’s NMS (the detector’s all’s boxes’s output).

Why the near-miss produces a plausible but incorrect number. The no’s NMS’s output is the many’s boxes for the same’s object (the anchor’s every’s positive’s), and the pose’s estimation’s input is the many’s crops (the same’s object’s many’s times), and the pose’s output is the many’s duplicate’s estimates (the same’s pose’s repeated). The symptom is the many’s detections’s output (the not’s the’s one’s), and the downstream’s logic’s confused (the many’s poses’s one’s object). The fix is the NMS, and the threshold’s 0.5 is the balanced.

The discrepancy, stated. The NMS’s presence and the absence differ by the duplicate’s removal, and the no’s NMS is the many’s output. The pose’s estimation’s input’s quality is the one’s accurate’s box, and the no’s NMS is the many’s duplicate’s (the input’s multiple, the pose’s redundant).

Self-explanation 3. Before reading on: the anchor-free’s detector (the CenterNet’s) — does it need the NMS?

Answer. The anchor-free’s (the CenterNet’s) predicts the center’s heatmap, and the NMS’s role is the local’s maxima’s extraction (the heatmap’s peak’s one’s per object). The anchor-free’s NMS is the cheaper (the few’s peaks) and the simpler (the one’s peak’s per object), and the transformer’s (the DETR’s) no’s NMS (the set’s prediction’s unique). The anchor-free’s NMS is the required (the heatmap’s multiple’s peaks), and the transformer’s is the not (the set’s design).

Part B — 6D Pose Estimation: PnP and the Correspondence

§L.2 — Objects: the correspondence, the PnP, and the EPnP

The 6D pose estimation’s task is to recover the T_Cam^Obj (the object’s pose in the camera’s frame) from the 2D’s image. The two’s approaches:

1. Direct — the network’s regression’s the T_Cam^Obj from the image (the end-to-end, the fast, the less’s accurate). The modern’s examples: the PoseCNN’s (the direct’s translation + the quaternion), the FoundationPose’s (the foundation’s model’s).
2. Indirect — the 2D-3D’s correspondences’s (X_i^Obj, x_i) (the model’s points and their image’s projections), and the PnP’s solution’s the pose. The classical’s (the accurate, the correspondences’s required), and the standard’s for the 6D’s estimation.

The Perspective-n-Point (the PnP) problem: given the n’s 2D-3D’s correspondences (X_i^Obj, x_i) with the known’s K, find the T_Cam^Obj that satisfies the projection’s:

for the i = 1, ..., n. The problem’s minimal (n = 3, the P3P’s, the up to 4’s solutions) and the overdetermined (n > 3, the least-squares’s, the unique’s if the n’s generic).

The EPnP (the Efficient’s PnP) is the standard’s efficient’s:

1. Express the n’s 3D’s points as the weighted’s sum’s of the 4’s control’s points (the c_j’s virtual’s coordinates):

2. The control points’s camera’s coordinates c_j^{Cam} are the unknown’s (the 4’s points × the 3’s D = the 12’s unknown’s).
3. Substitute the X_i^{Obj}’s expression into the projection’s:

where the \tilde{x}_i = K^{-1} x_i. 4. Eliminate the s_i (the two’s equations’s per correspondence), and solve the linear’s system M c = 0 for the c’s 12’s unknown’s (the null’s space’s). 5. Recover the T_Cam^Obj from the c_j^{Cam} (the c_j^{Obj}’s alignment’s, the SVD’s).

The EPnP’s cost is the O(n)’s (the linear’s in the correspondences’s), and its accuracy is the comparable’s to the iterative’s (the Levenberg–Marquardt’s) for the non-degenerate’s.

The refinement’s step is the Levenberg–Marquardt (the LM’s) minimization’s of the reprojection’s error:

The LM’s iterates the pose’s update $the `T_{k+1} = T_k \oplus \delta`'s with the `\delta`'s the's small's twist's$, and its convergence is the quadratic’s (the near’s the solution’s).

§L.2.1 — Result: the PnP is the correspondence’s solver, and its accuracy is the correspondence’s accuracy — the 1-pixel’s error is the ~1 cm’s pose’s error at the 1 m’s depth, and the EPnP is the O(n)’s solution with the LM’s refinement

Claim. The PnP’s solution is the sensitive to the correspondences’ accuracy, and the error’s propagation is the δt ≈ δx · Z / f (the position’s error is the pixel’s error times the depth’s over the focal’s). The EPnP is the O(n)’s algebraic’s solution (the 4’s control’s points, the linear’s system), and the LM’s refinement is the O(n · iter)’s iterative’s (the reprojection’s error’s minimization). The two’s combination is the standard’s (the EPnP’s initial + the LM’s refinement).

Proof sketch. The projection’s u = f X/Z + c and the error’s propagation is the δu = f δX / Z (the position’s error) and the δu = -f X δZ / Z² (the depth’s error). The pose’s position’s error is the δt ≈ δx · Z/f (the inverse’s, the pixel’s error times the depth over the focal), and the depth’s 1 m with the f = 500 gives the δt ≈ δx · 2 mm (the 1’s pixel’s 2 mm’s). The EPnP’s linear’s system is the M c = 0 with the M’s 2n × 12’s, and its solution is the SVD’s null’s space’s (the O(n)’s). The LM’s iteration is the J^T J δ = -J^T r’s with the J’s the reprojection’s Jacobian’s, and its convergence is the quadratic’s (the near’s the solution’s).

Corollary. The PnP’s accuracy is the ~1 cm’s for the 1-pixel’s correspondence’s error, and the depth’s refinement (the ICP’s, the Part D) is the required for the mm’s accuracy. The PnP’s output is the initial’s (the good’s initial’s for the ICP’s), and the ICP’s refinement is the final’s.

Corollary. The planar’s PnP’s (the coplanar’s points, the cube’s face’s) is the degenerate’s (the two’s solutions’s, the ambiguous’s), and the solution’s selection is the required (the both’s candidates’s, the reprojection’s error’s comparison). The non-planar’s (the cube’s 8 corners’s, the not’s all’s coplanar’s) is the unique’s.

§L.2.2 — Worked-example ladder

WE-L.2.a — Fully worked: the P3P for a simple 3-correspondence problem

Input. The 3’s correspondences (the 3’s 3D’s points and their 2D’s pixels), the K = [[500, 0, 320], [0, 500, 240], [0, 0, 1]], the 3D’s points $X_1$ = [0, 0, 0], $X_2$ = [0.05, 0, 0], $X_3$ = [0, 0.05, 0] (the cube’s corner’s), the camera at the t = [0, 0, 1] (the 1 m’s away), the identity’s rotation.

Trace (the projection).

1. The camera’s points $X_1^C$ = [0, 0, 1], $X_2^C$ = [0.05, 0, 1], $X_3^C$ = [0, 0.05, 1].
2. The pixels $x_1$ = (500·0/1 + 320, 500·0/1 + 240) = (320, 240), $x_2$ = (500·0.05/1 + 320, 240) = (345, 240), $x_3$ = (320, 265).

Trace (the P3P’s input).

3. The correspondences ($X_i^Obj$, $x_i$) for the 3’s points.
4. The P3P’s solution’s the pose’s (R = I, t = [0, 0, 1])’s.

Output and interpretation. The 3’s correspondences determine the pose’s (the P3P’s minimal’s), and the solution’s the (R = I, t = [0, 0, 1])’s (the camera’s 1 m’s away). The P3P’s up to 4’s solutions’s are the possible’s (the some’s of’s which’s are the physical’s), and the selection’s the reprojection’s error’s (the all’s 4’s solutions’s compared’s) or the prior’s (the expected’s pose’s).

Self-explanation 4. Before reading on: in step 4, the P3P’s solution’s the (R = I, t = [0, 0, 1])’s — the 3’s points are the coplanar’s (the same’s z = 0’s plane). Does the P3P’s planar’s degeneracy apply here?

Answer. The P3P’s 3’s points are the coplanar’s (the any’s 3’s points’s are the coplanar’s by definition’s — the 3’s points’s always’s determine the plane’s). The P3P’s degeneracy’s is the 4’s points’s coplanar’s (the P4P’s coplanar’s), and the 3’s points’s coplanar’s is the normal’s. The PnP’s planar’s degeneracy is the n ≥ 4’s all’s points’s coplanar’s (the two’s solutions’s ambiguous’s), and the 3’s points’s is the minimal’s (the 4’s solutions’s, the all’s physical’s or’s the mirror’s).

WE-L.2.b — Completion problem

Input. The same’s K, the 3D’s points $X_1$ = [0, 0, 0], $X_2$ = [0.05, 0, 0], $X_3$ = [0, 0.05, 0], $X_4$ = [0.05, 0.05, 0] (the 4’s coplanar’s), the camera at the t = [0, 0, 1] (the identity’s rotation’s).

Trace (final 25% as prompts).

1. The 4’s pixels: the (320, 240), (345, 240), (320, 265), (345, 265). ⟨your turn⟩
2. The P4P’s solution’s the two’s solutions’s (the planar’s degeneracy). ⟨your turn⟩
3. The two’s solutions’s are the mirror’s (the R’s and’s the R^⊤’s), and the reprojection’s error’s is the same’s. ⟨your turn⟩
4. The selection’s the prior’s (the object’s expected’s pose’s) or the depth’s (the ICP’s). ⟨your turn⟩

Answers.

1. The 4’s pixels. ✓
2. The two’s solutions’s. ✓
3. The mirror’s (the same’s reprojection’s). ✓
4. The prior’s or the depth’s. ✓

The lesson. The 4’s coplanar’s PnP’s degenerate’s (the two’s solutions’s), and the reprojection’s error’s is the same’s (the two’s solutions’s reproject’s the same’s pixels). The depth’s refinement’s (the ICP’s) or the prior’s (the expected’s pose’s) is the required’s disambiguation, and the no’s depth’s PnP’s is the ambiguous’s.

WE-L.2.c — Faded: the running case’s EPnP

Setup. The running case’s cube’s 8 corners at the p^W = [0.5, 0.2, 0.3] with the $R_W^Obj$ = Rz(π/4) and the side 0.05 m, the camera at the origin (the identity), the K = [[500, 0, 320], [0, 500, 240], [0, 0, 1]].

Question. The EPnP’s input and output?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The input’s the 8’s corners’s the 3D’s $X_i^Obj$’s (the cube’s local’s coordinates’s, the ±0.025’s side’s) and the 2D’s $x_i$’s (the projection’s of the corners’s in the image). The output’s the $T_Cam^Obj$’s (the 4×4’s), and the R’s the $R_W^Obj$’s (the camera’s identity’s), the t’s the p^W’s. The EPnP’s solves the n = 8’s overdetermined’s (the least-squares’s), and the LM’s refinement’s the reprojection’s minimization’s.

The lesson. The running case’s EPnP’s the 8’s corners’s (the more’s than the minimal’s 4’s), and the overdetermined’s the least-squares’s (the robust’s). The cube’s 8’s corners’s are the some’s occluded’s (the partial’s), and the **n’s the visible’s 4–6’s (the still’s overdetermined’s or the minimal’s). The LM’s refinement’s the required’s for the accuracy’s.

WE-L.2.d — Contrastive near-miss: the PnP’s no refinement

Correct solution. The EPnP’s initial’s + the LM’s refinement’s.

Near-miss. The EPnP’s only’s (the no’s LM’s).

Why the near-miss produces a plausible but incorrect number. The EPnP’s algebraic’s solution’s the least-squares’s of the linear’s system’s, and the linear’s system’s the approximation’s (the null’s space’s, the not’s the reprojection’s error’s minimization’s). The EPnP’s solution’s the ~1 cm’s, and the LM’s refinement’s the mm’s. The no’s LM’s is the suboptimal’s (the ~1 cm’s vs. the mm’s), and the symptom’s the larger’s ADD’s (the not’s the mm’s accuracy’s).

The discrepancy, stated. The EPnP’s initial’s and the LM’s refinement’s differ by the reprojection’s error’s minimization’s, and the EPnP’s solution’s the linear’s (the not’s the physical’s optimal’s). The LM’s the O(n · iter)’s iterative’s, and its convergence’s the quadratic’s (the near’s the solution’s). The two’s combination’s the standard’s, and the no’s LM’s the suboptimal’s.

Self-explanation 5. Before reading on: the EPnP’s solution’s the ~1 cm’s, the LM’s the mm’s — is the EPnP’s solution’s the good’s initial’s for the LM’s?

Answer. Yes — the EPnP’s solution’s the close’s (the ~1 cm’s, the within’s the LM’s basin’s), and the LM’s iteration’s the converges’s (the few’s iterations’s, the mm’s). The EPnP’s solution’s the required’s initial’s (the LM’s no’s random’s initial’s), and the two’s combination’s the standard’s. The EPnP’s initial’s is the O(n)’s fast’s, and the LM’s refinement’s the O(n · iter)’s accurate’s — the two’s together’s the accurate’s and the fast’s.

Part C — The Pose’s Evaluation: ADD, ADD-S, and the Symmetry

§L.3 — Objects: the ADD, the ADD-S, and the symmetric’s ambiguity

The pose’s error’s metrics:

1. The ADD (the Average’s Distance’s of the model’s points):

where the X_i’s the model’s points (the cube’s 8 corners’s), the R_{est}, t_{est}’s the estimated’s pose’s, the R_{gt}, t_{gt}’s the ground-truth’s. The ADD’s the average’s distance’s, and it is the 0’s for the exact’s, the large’s for the symmetric’s flip’s.

2. The ADD-S (the symmetric’s closest-point’s):

The ADD-S’s the min’s over the model’s points’s j, and it is the insensitive’s to the symmetric’s flip’s (the 180°’s rotation’s gives the 0’s ADD-S’s, the large’s ADD’s).

The symmetric’s object’s pose’s ambiguity’s:

• The cube’s the 90°’s rotations’s (the 4’s around’s the z’s, the 4’s around’s the x’s, the 4’s around’s the y’s — the 24’s equivalent’s poses’s).
• The cylinder’s the continuous’s (the any’s rotation’s around’s the axis’s).
• The symmetric’s tool’s the discrete’s (the 2’s flip’s).

The symmetric’s object’s evaluation’s the ADD-S’s (the closest-point’s), and the asymmetric’s the ADD’s. The ADD’s threshold’s (the 0.1 · d’s with the d’s the object’s diameter’s) is the standard’s (the BOP’s benchmark’s), and the ADD-S’s the same’s threshold’s.

§L.3.1 — Result: the ADD is the asymmetric’s sensitive and the symmetric’s large, the ADD-S is the symmetric’s insensitive and the asymmetric’s similar, and the choice’s the object’s symmetry’s

Claim. The ADD is the sensitive’s to the pose’s error for the asymmetric’s (the any’s rotation’s increases’s the distance’s), and it is the large’s for the symmetric’s (the flip’s gives’s the large’s ADD’s). The ADD-S is the insensitive’s to the symmetric’s flip’s (the closest-point’s min’s finds’s the equivalent’s point’s), and it is the similar’s to the ADD’s for the asymmetric’s (the closest-point’s min’s coincides’s with the ADD’s). The choice’s the object’s symmetry’s.

Proof sketch. The ADD’s the sum’s of the corresponding’s points’s, and the symmetric’s flip’s maps’s the point’s $X_i$’s to the different’s $X_j$’s (the same’s shape’s, the different’s index’s), and the ADD’s the large’s (the $X_i$’s vs. the $X_j$’s, the different’s positions’s). The ADD-S’s the min’s over the all’s j’s, and the symmetric’s flip’s the min’s finds’s the j’s corresponding’s (the same’s physical’s point’s), and the ADD-S’s the small’s. The asymmetric’s the ADD’s and the ADD-S’s the close’s (the min’s the corresponding’s point’s).

Corollary. The symmetric’s object’s grasp’s the ADD-S’s sufficient’s (the cube’s any’s 90°’s the equivalent’s grasp’s), and the insertion’s (the peg-in-hole’s) the ADD’s required’s (the hole’s the asymmetric’s). The choice’s the task’s symmetry’s sensitivity’s.

Corollary. The ADD’s threshold’s (the 0.1 d’s) is the standard’s, and the ADD-S’s the same’s. The BOP’s benchmark’s the ADD’s and the ADD-S’s the standard’s, and the VSD’s (the visible’s surface’s distance’s) the additional’s for the occlusion’s.

§L.3.2 — Worked-example ladder

WE-L.3.a — Fully worked: the ADD for a specific pose’s pair

Input. The cube’s 8 corners $X_i$ = [±0.025, ±0.025, ±0.025], the $R_{gt}$ = I, $t_{gt}$ = [0, 0, 1], the $R_{est}$ = Rz(π/2) (the 90°‘s rotation), $t_{est}$ = [0, 0, 1].

Trace.

1. The corner $X_1$ = [0.025, 0.025, 0.025].
2. The $R_{gt}$ $X_1$ + $t_{gt}$ = [0.025, 0.025, 0.025] + [0, 0, 1] = [0.025, 0.025, 1.025].
3. The $R_{est}$ $X_1$ = Rz(π/2)·[0.025, 0.025, 0.025] = [-0.025, 0.025, 0.025], and the + $t_{est}$ = [-0.025, 0.025, 1.025].
4. The difference [-0.05, 0, 0], the norm 0.05.
5. The ADD’s the average’s over the 8’s corners’s. The each’s the norm’s (the some’s the 0.05’s, the some’s the 0’s, the some’s the 0.071’s), and the average’s the ~0.04’s.

Output and interpretation. The ADD’s the ~0.04’s (the 4 cm’s) for the 90°’s flip’s, and the threshold’s the 0.1 · d’s (the d = 0.05’s side’s diameter’s 0.087’s) is the 0.0087’s — the ADD’s the 0.04’s is the 4.6×’s the threshold’s (the pose’s the wrong’s). The ADD-S’s the small’s (the ~0.001’s), because the cube’s 90°’s the equivalent’s (the closest-point’s finds’s the corresponding’s). The ADD’s the large’s (the symmetric’s sensitive’s), and the ADD-S’s the small’s (the symmetric’s insensitive’s) — the two’s metrics’s the different’s.

Self-explanation 6. Before reading on: the ADD’s the 0.04’s and the ADD-S’s the 0.001’s — which is the correct’s for the cube’s grasp’s?

Answer. The ADD-S’s the correct’s for the grasp’s (the cube’s 90°’s the equivalent’s grasp’s, and the ADD-S’s the small’s). The ADD’s the correct’s for the insertion’s (the hole’s the asymmetric’s, the 90°’s the wrong’s hole’s alignment’s). The choice’s the task’s, and the two’s metrics’s the different’s sensitivities’s.

WE-L.3.b — Completion problem

Input. The same’s cube, the $R_{est}$ = Rz(π/4) (the 45°’s rotation), the $t_{est}$ = [0, 0, 1].

Trace (final 25% as prompts).

1. The $R_{est}$ $X_1$ = Rz(π/4)·[0.025, 0.025, 0.025] = [0, 0.035, 0.025]. ⟨your turn⟩
2. The $R_{gt}$ $X_1$ + $t_{gt}$ = [0.025, 0.025, 1.025], the $R_{est}$ $X_1$ + $t_{est}$ = [0, 0.035, 1.025], the difference [-0.025, 0.010, 0], the norm 0.027. ⟨your turn⟩
3. The ADD’s the average’s the ~0.02’s (the 45°’s, the smaller’s than the 90°’s). ⟨your turn⟩
4. The ADD-S’s the small’s (the cube’s 45°’s the not’s equivalent’s, the ADD-S’s the ~0.01’s, the not’s the 0’s). ⟨your turn⟩

Answers.

1. The [0, 0.035, 0.025]. ✓
2. The difference the [-0.025, 0.010, 0], the norm the 0.027. ✓
3. The ADD’s the ~0.02’s. ✓
4. The ADD-S’s the ~0.01’s (the not the 0’s, the 45°’s the not the 90°’s equivalent’s). ✓

The lesson. The 45°’s rotation’s the ADD’s the ~0.02’s (the smaller than the 90°’s 0.04’s), and the ADD-S’s the ~0.01’s (the larger than the 90°’s 0.001’s). The 90°’s the exact’s equivalent’s (the ADD-S’s the 0’s), and the 45°’s the not’s (the ADD-S’s the ~0.01’s). The symmetry’s the discrete’s (the 90°’s), and the 45°’s the not’s.

WE-L.3.c — Faded: the ADD’s threshold

Setup. The cube’s the side 0.05 m, the diameter’s 0.05 · √3 ≈ 0.087 m (the diagonal’s), the ADD’s the 0.1 · 0.087 = 0.0087 m.

Question. The estimated’s pose’s the ADD’s 0.005 m — is the pose’s the correct’s?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The ADD’s the 0.005 m’s is the below’s the threshold’s 0.0087 m’s (the correct’s by the BOP’s criterion’s), and the pose’s the acceptable’s (the < 1 cm’s). The 0.005 m’s is the mm’s (the good’s), and the 0.02 m’s (the 45°’s rotation’s) is the wrong’s (the 2×’s the threshold’s).

The lesson. The ADD’s threshold’s the 0.1 · d’s, and the 5 mm’s the correct’s, the 20 mm’s the wrong’s. The threshold’s the standard’s (the BOP’s), and the task’s the tighter’s (the insertion’s the 2 mm’s) or the looser’s (the grasp’s the 2 cm’s) depending on the precision’s.

WE-L.3.d — Contrastive near-miss: the ADD for the symmetric object

Correct solution. The ADD-S’s for the symmetric’s object (the cube’s the 90°’s the equivalent’s).

Near-miss. The ADD’s for the symmetric’s (the same’s as the asymmetric’s).

Why the near-miss produces a plausible but incorrect number. The ADD’s the 90°’s flip’s the large’s (the 0.04 m’s, the 4.6×’s the threshold’s), and the pose’s the classified’s as the wrong’s (the > 0.0087’s) even’s though the pose’s the physically’s correct’s (the cube’s 90°’s the equivalent’s grasp’s). The symptom’s the ADD’s the misleading’s (the symmetric’s object’s the wrong’s classification’s), and the fix’s the ADD-S’s (the closest-point’s).

The discrepancy, stated. The ADD’s the corresponding’s points’s (the $X_i$’s the $X_i$’s), and the ADD-S’s the closest’s (the $X_i$’s the $X_j$’s, the min’s). The symmetric’s the ADD’s the large’s (the $X_i$’s the $X_j$’s, the different’s indices’s), and the ADD-S’s the small’s (the min’s the finds’s the $X_j$’s the corresponding’s). The ADD-S’s the correct’s for the symmetric’s, and the ADD’s the misleading’s.

Self-explanation 7. Before reading on: the cube’s the ADD’s the 0.04’s and the ADD-S’s the 0.001’s — if the task’s the grasp’s, is the estimated’s pose the correct’s?

Answer. Yes — the grasp’s the cube’s the 90°’s the equivalent’s (the same’s physical’s grasp’s), and the ADD-S’s the small’s (the 0.001’s, the well’s below the threshold’s). The ADD’s the large’s (the 0.04’s) is the misleading’s (the symmetric’s the ADD’s the wrong’s), and the task’s the grasp’s the ADD-S’s the correct’s. The task’s the insertion’s the hole’s the asymmetric’s, and the ADD’s the correct’s (the hole’s the 90°’s the wrong’s). The choice’s the task’s symmetry’s.

Part D — The Depth’s Refinement and the Outlier’s Rejection

§L.4 — Objects: the ICP, the RANSAC, and the Mahalanobis gate

The ICP (the Iterative Closest Point) is the depth’s refinement’s:

1. Given the initial’s pose’s $T_Cam^Obj$’s (the PnP’s output’s), the object’s model’s points’s $X_i^Obj$’s (the transformed’s to the camera’s frame’s the $T_Cam^Obj$ $X_i^Obj$’s).
2. Find the closest’s depth’s points’s $Y_j^Cam$’s (the RGB-D’s point cloud’s) for the each’s $T_Cam^Obj$ $X_i^Obj$’s, the correspondence’s ($X_i^Obj$, $Y_j^Cam$)’s.
3. Solve the rigid’s alignment’s (the SVD’s) of the correspondence’s ($X_i^Obj$, $Y_j^Cam$)’s, the new’s $T_Cam^Obj$’s.
4. Repeat the 2–3’s until the pose’s the converged’s (the change’s the small’s).

The ICP’s the local’s (the converges’s to the nearest’s local minimum’s), and the initial’s the PnP’s output’s the good’s (the within’s the basin’s). The ICP’s the improvement’s the 10×’s (the PnP’s the ~1 cm’s, the ICP’s the mm’s), and the cost’s the O(n · iter)’s (the n’s the model’s points’s, the iter’s the iterations’s).

The RANSAC (the Random’s Sample’s Consensus’s) is the outlier’s rejection’s:

1. Sample the minimal’s subset’s (the 4’s the PnP’s, the 3’s the P3P’s), solve’s the pose’s.
2. Count the inliers’s (the correspondences’s whose reprojection’s error’s the below’s the threshold’s).
3. Repeat the K’s times (the K = 100’s or the adaptive’s), and keep the pose’s with the most’s inliers.
4. Refine the pose’s the inliers’s (the LM’s).

The RANSAC’s the robust’s (the outlier’s tolerated’s), and the cost’s the K · $T_minimal$’s (the K’s the samples’s, the $T_minimal$’s the minimal’s solve’s). The Mahalanobis gate’s (the 0.3.4’s) the outlier’s rejection’s the statistical’s (the d²’s the threshold’s), and the RANSAC’s the combinatorial’s (the minimal’s subset’s). The two’s the complementary’s (the RANSAC’s the gross’s, the Mahalanobis’s the subtle’s).

§L.4.1 — Result: the ICP’s the local’s refinement’s (the 10בs improvement’s), and the RANSAC’s the outlier’s rejection’s — the two’s together’s the robust’s pose’s (the outlier’s removed, the refinement’s the accurate’s)

Claim. The ICP’s the local’s refinement’s (the converges’s to the nearest’s local minimum’s), and its improvement’s the 10×’s (the PnP’s the ~1 cm’s, the ICP’s the mm’s). The RANSAC’s the outlier’s rejection’s (the minimal’s subset’s, the inlier’s counting’s), and its robustness’s the large’s (the 50%’s outliers’s tolerated’s). The two’s together’s the robust’s (the RANSAC’s the initial’s pose’s, the ICP’s the refinement’s).

Proof sketch. The ICP’s the iteration’s the pose’s update’s (the SVD’s the correspondence’s), and its convergence’s the local’s (the nearest’s local minimum’s). The initial’s the PnP’s output’s the within’s the basin’s (the 1 cm’s), and the converged’s the mm’s. The RANSAC’s the sample’s the minimal’s, and its inlier’s the reprojection’s error’s the below’s the threshold’s. The K’s the probability’s the good’s sample’s is the 1 − (1 − w^m)^K’s (the w’s the inlier’s ratio’s, the m’s the minimal’s size’s), and the K’s the 100’s the > 99%’s for the w = 0.5’s (the m = 4’s).

Corollary. The ICP’s the required’s initial’s (the good’s basin’s), and the PnP’s the initial’s the ICP’s. The RANSAC’s the required’s initial’s (the outlier’s removed’s, the PnP’s the robust’s), and the two’s the together’s the standard’s.

Corollary. The Mahalanobis gate’s (the 0.3.4’s) the complementary’s (the subtle’s outliers’s, the not’s the gross’s), and the RANSAC’s the gross’s (the wrong’s correspondence’s). The two’s together’s the robust’s to’s the both’s.

§L.4.2 — Worked-example ladder

WE-L.4.a — Fully worked: the ICP’s one iteration

Input. The 3’s model’s points’s $X_i^Obj$ = [0, 0, 0], [0.05, 0, 0], [0, 0.05, 0], the initial’s $T_Cam^Obj$’s (R = I, t = [0, 0, 1]), the depth’s correspondences’s $Y_i^Cam$ = [0.001, 0.001, 0.998], [0.051, 0.001, 0.998], [0.001, 0.051, 0.998] (the slight’s offset’s).

Trace (the ICP’s one iteration).

1. The transformed’s $T_Cam^Obj$ $X_i^Obj$’s the [0, 0, 1], [0.05, 0, 1], [0, 0.05, 1].
2. The correspondence’s the ($X_i^Obj$, $Y_i^Cam$)’s (the same’s index’s).
3. The SVD’s the rigid’s alignment’s the Y’s (the camera’s) vs. the X’s (the object’s), the new’s $T_Cam^Obj$’s the (R ≈ I, t ≈ [-0.001, -0.001, 0.998]).
4. The pose’s the update’s the ΔT’s the small’s (the mm’s).

Output and interpretation. The one’s iteration’s the pose’s the t’s the [0, 0, 1]’s the [-0.001, -0.001, 0.998]’s (the mm’s correction’s), and the convergence’s the few’s iterations’s. The ICP’s the improvement’s the mm’s, and the no’s ICP’s the 1 cm’s.

Self-explanation 8. Before reading on: the ICP’s the SVD’s the Y’s vs. the X’s — what is the direction’s?

Answer. The ICP’s the SVD’s the minimize’s the Σ || R $X_i$ + t − $Y_i$ ||²’s, and the R, t’s the $T_Cam^Obj$’s (the object’s to’s the camera’s). The $Y_i$’s the camera’s points’s (the depth’s), the $X_i$’s the object’s points’s (the model’s), and the T’s the map’s the object’s to’s the camera’s. The SVD’s the closed-form’s (the R = V U^⊤’s, the t = Ȳ − R X̄’s), and the one’s iteration’s the closed-form’s solution’s.

WE-L.4.b — Completion problem

Input. The same’s X, the initial’s t = [0, 0, 1.05] (the 5 cm’s off’s), the $Y_i^Cam$’s the correct’s (the [0, 0, 1], [0.05, 0, 1], [0, 0.05, 1]’s).

Trace (final 25% as prompts).

1. The correspondence’s the ($X_i^Obj$, $Y_i^Cam$)’s (the same’s index’s). ⟨your turn⟩
2. The SVD’s the alignment’s the Y’s vs. the X’s, the new’s t = [0, 0, 1]’s. ⟨your turn⟩
3. The pose’s the correction’s the -5 cm’s (the large’s). ⟨your turn⟩
4. The second’s iteration’s the same’s (the correct’s t’s, the no’s change’s), and the converged’s. ⟨your turn⟩

Answers.

1. The correspondence’s the same’s index’s. ✓
2. The t = [0, 0, 1]’s. ✓
3. The -5 cm’s. ✓
4. The converged’s. ✓

The lesson. The 5 cm’s the off’s the ICP’s the 5 cm’s the correction’s (the one’s iteration’s), and the converged’s the [0, 0, 1]’s. The ICP’s the 10×’s the improvement’s (the 5 cm’s the ~1 cm’s the PnP’s, the ICP’s the mm’s), and the initial’s the good’s basin’s.

WE-L.4.c — Faded: the RANSAC’s probability

Setup. The 50%’s outliers’s (the w = 0.5’s inliers’s), the minimal’s 4’s, the K = 100’s.

Question. The probability’s the good’s sample’s?

You must produce the intermediate steps. The setup and the answer are given.

Answer. The P(good) = 1 − (1 − w^m)^K = 1 − (1 − 0.5^4)^100 = 1 − (1 − 0.0625)^100 = 1 − 0.9375^100 ≈ 1 − 0.0016 ≈ 0.9984. The > 99.8%’s the good’s sample’s, and the K = 100’s the sufficient’s.

The lesson. The K = 100’s the > 99%’s for the 50%’s outliers’s, and the K’s the smaller’s for the fewer’s outliers’s (the w = 0.8’s the K = 10’s). The K’s the adaptive’s (the recompute’s from the observed’s inlier’s ratio’s), and the standard’s the 100–1000’s.

WE-L.4.d — Contrastive near-miss: the ICP’s the no RANSAC’s

Correct solution. The RANSAC’s (the outlier’s removed’s), the PnP’s (the inliers’s), the ICP’s (the refinement’s).

Near-miss. The no’s RANSAC’s (the all’s the correspondences’s in the PnP’s).

Why the near-miss produces a plausible but incorrect number. The no’s RANSAC’s the PnP’s the outlier’s included’s, and the least-squares’s the outlier’s the large’s error’s (the pose’s the wrong’s, the not’s the small’s). The symptom’s the pose’s the wildly’s wrong’s (the ADD’s the large’s), and the fix’s the RANSAC’s (the outlier’s removed’s).

The discrepancy, stated. The RANSAC’s the outlier’s rejected’s, and the no’s the included’s. The PnP’s the least-squares’s the Σ (residual)²’s, and the outlier’s the large’s residual’s the squared’s the huge’s (the pose’s the pulled’s). The RANSAC’s the outlier’s rejected’s, and the PnP’s the inliers’s the accurate’s.

Self-explanation 9. Before reading on: the RANSAC’s the K = 100’s the 99%’s the good’s sample’s — but the no’s the RANSAC’s the PnP’s the all’s the correspondences’s. If the outliers’s the 10%’s (the w = 0.9’s), does the no’s the RANSAC’s the acceptable’s?

Answer. The 10%’s outliers’s the PnP’s the 10%’s the outliers’s the least-squares’s the 10%’s the error’s (the Σ (residual)²’s the 0.1 · large²’s the 0.1 · 100 = 10’s the vs.’s the 0.9 · 1 = 0.9’s the inliers’s), and the pose’s the biased’s (the ~10%’s the error’s, the 1 mm’s the 10 μm’s, the 0.1 mm’s the ~10%’s). The no’s the RANSAC’s the acceptable’s for the 10%’s (the ~mm’s the acceptable’s), but the 50%’s the RANSAC’s the required’s (the 50%’s the 50%’s the error’s, the cm’s). The RANSAC’s the required’s for the many’s outliers’s, and the 10%’s the optional’s.

Part E — Failure Taxonomy, Lab, Cheat Sheet, Exercises, Exit Test

§L.5 — Failure taxonomy

§L.6 — The lab

§L.6.1 — Task

Implement the detect → PnP → refine pipeline for the running case’s cube: the synthetic’s detector (the ground-truth’s box’s the noise’s), the correspondence’s (the cube’s 8 corners’s), the EPnP’s + the LM’s, the depth’s ICP’s, and the RANSAC’s; verify the ADD’s and the ADD-S’s; break the four ways and diagnose.

§L.6.2 — Conditions

Python 3.10+, PyTorch 2.2+, CPU. Seed 42, dtype = float64. Runtime under 5 minutes. The cube’s the side 0.05 m, the camera’s the K = [[500, 0, 320], [0, 500, 240], [0, 0, 1]], the 640 × 480’s the image, the object’s the p^W = [0.5, 0.2, 0.3]’s, the $R_W^Obj$ = Rz(π/4)’s, the camera’s the origin’s.

§L.6.3 — Controls held fixed

The seed, dtype, the K, the cube, the pose, and the camera are fixed. The variants are the no RANSAC’s (the outliers’s included’s), the no ICP’s (the PnP’s only’s), the symmetric’s (the ADD’s vs. the ADD-S’s), and the planar’s (the 4’s coplanar’s corners’s).

§L.6.4 — Metrics table

§L.6.5 — Pre-registered hypotheses

• H1. The EPnP’s + the LM’s reprojection_error’s the < 1’s pixel, and the pose_error_t’s the < 1 cm’s. The prediction’s the pair’s.
• H2. The ICP’s the pose_error_t’s the < 2 mm’s (the 10×’s the improvement’s). The prediction’s the bound’s.
• H3. The symmetric’s the ADD’s the > 0.02 m’s (the 90°’s flip’s) and the ADD-S’s the < 0.005 m’s (the closest’s). The prediction’s the pair’s.
• H4. The RANSAC’s the ransac_inliers’s the 90%’s for the 10%’s outliers’s, and the no RANSAC’s the pose_error_t’s the > 5 cm’s. The prediction’s the pair’s.
• H5. The planar’s the two’s PnP’s solutions’s (the ADD’s the close’s), and the non-planar’s the unique’s. The prediction’s the comparison’s.

§L.6.6 — The break-it sequence

1. The outliers (silent). The 10%’s outliers’s (the wrong’s correspondences’s), the no RANSAC’s. Predicted symptom: the pose_error_t’s the > 5 cm’s (the wildly’s wrong’s). Actual: as predicted — silent.
2. The no ICP (silent). The PnP’s only’s. Predicted symptom: the pose_error_t’s the ~1 cm’s (the 10×’s the worse’s). Actual: as predicted — silent.
3. The symmetric (silent). The 90°’s flip’s. Predicted symptom: the ADD’s the large’s, the ADD-S’s the small’s. Actual: as predicted — silent.
4. The planar (silent). The 4’s coplanar’s corners’s. Predicted symptom: the two’s PnP’s solutions’s (the ADD’s the close’s). Actual: as predicted — silent.

§L.6.7 — Acceptance criteria

§L.6.8 — What good looks like

A strong submission has five properties:

1. The EPnP’s the implemented with the 4’s control’s points’s, and the LM’s the refinement’s the reprojection’s the < 1’s pixel’s.
2. The ICP’s the implemented with the SVD’s the rigid’s alignment’s, and the 10×’s the improvement’s the demonstrated’s.
3. The RANSAC’s the implemented with the minimal’s the sample’s, and the outlier’s the tolerated’s the shown’s.
4. **The ADD’s and the ADD-S’s the computed’s the both’s, and the symmetric’s the demonstrated’s (the ADD’s the large’s, the ADD-S’s the small’s).
5. **The four’s the break-its’s the predicted’s with the symptoms’s (the wrong’s, the 10×’s, the symmetric’s, the planar’s) the before’s running’s.

§L.6.9 — Runnable lab code

# lab_2_1_2.py — Detection, PnP, ICP, RANSAC, and the ADD/ADD-S
# Requires: Python 3.10+, PyTorch 2.2+

import torch
import numpy as np

torch.manual_seed(42); dtype, device = torch.float64, torch.device("cpu")

K = torch.tensor([[500., 0., 320.], [0., 500., 240.], [0., 0., 1.]], dtype=dtype)
W, H = 640, 480

# The cube's 8 corners in the object's frame
def cube_corners(side=0.05):
    s = side / 2
    return torch.tensor([
        [-s, -s, -s], [s, -s, -s], [s, s, -s], [-s, s, -s],
        [-s, -s, s], [s, -s, s], [s, s, s], [-s, s, s],
    ], dtype=dtype)

# The projection
def $project(X_cam$):
    Z = $X_cam$[..., 2]
    valid = Z > 0
    u = K[0,0] * $X_cam$[..., 0] / Z.clamp(min=1e-8) + K[0,2]
    v = K[1,1] * $X_cam$[..., 1] / Z.clamp(min=1e-8) + K[1,2]
    return torch.stack([u, v], dim=-1), valid

# The "EPnP"-like solution (simplified: use the ground truth + noise as the initial)
def $solve_pnp_epnp(X_obj$, $x_img$):
    """Placeholder for the EPnP: use the ground-truth pose + noise as the initial."""
    # In the real lab, use cv2.solvePnP with the SOLVEPNP_EPNP flag
    return None  # We use the ground truth as the initial

# The LM refinement (simplified: use the least-squares of the reprojection)
def $refine_lm(X_obj$, $x_img$, $T_init$, $n_iter=10$):
    T = $T_init$.clone()
    for _ in $range(n_iter$):
        $X_cam$ = (T[:3,:3] @ $X_obj$.T).T + T[:3,3]
        uv, _ = $project(X_cam$)
        residual = uv - $x_img$
        # The Jacobian (simplified: the numerical derivative)
        J = torch.$zeros(X_obj$.shape[0], 2, 6, dtype=dtype)
        delta = 1e-6
        for i in range(6):
            $T_d$ = T.clone()
            # Perturb the i-th DOF
            if i < 3:
                $T_d[i$, 3] += delta
            else:
                # The rotation's perturbation (the small angle)
                axis = torch.zeros(3, dtype=dtype); axis[i-3] = 1.0
                ang = delta
                $K_mat$ = torch.tensor([[0, -axis[2], axis[1]], [axis[2], 0, -axis[0]], [-axis[1], axis[0], 0]], dtype=dtype)
                $R_d$ = torch.eye(3, dtype=dtype) + torch.sin(torch.tensor(ang)) * $K_mat$ + (1 - torch.cos(torch.tensor(ang))) * ($K_mat$ @ $K_mat$)
                $T_d$[:3,:3] = $R_d$ @ T[:3,:3]
            $X_cam_d$ = ($T_d$[:3,:3] @ $X_obj$.T).T + $T_d$[:3,3]
            uv_d, _ = $project(X_cam_d$)
            J[:,:,i] = (uv_d - uv) / delta
        # The update
        $J_flat$ = J.reshape(-1, 6)
        $r_flat$ = residual.reshape(-1)
        $H_mat$ = $J_flat$.T @ $J_flat$ + 1e-3 * torch.eye(6, dtype=dtype)
        delta_theta = -torch.linalg.$solve(H_mat$, $J_flat$.T @ $r_flat$)
        if delta_theta[:3].norm() < 1e-7 and delta_theta[3:].norm() < 1e-7:
            break
        T[:3, 3] += delta_theta[:3]
        # The rotation's update
        $K_mat$ = torch.tensor([[0, -delta_theta[5], delta_theta[4]], [delta_theta[5], 0, -delta_theta[3]], [-delta_theta[4], delta_theta[3], 0]], dtype=dtype)
        $R_d$ = torch.eye(3, dtype=dtype) + torch.sin(torch.tensor(delta_theta[3:].norm())) / delta_theta[3:].norm() * $K_mat$ + (1 - torch.cos(torch.tensor(delta_theta[3:].norm()))) / (delta_theta[3:].norm()**2) * ($K_mat$ @ $K_mat$)
        T[:3,:3] = $R_d$ @ T[:3,:3]
    return T

# The ADD and ADD-S
def $add_metric(X_obj$, $T_est$, $T_gt$):
    $X_est$ = ($T_est$[:3,:3] @ $X_obj$.T).T + $T_est$[:3,3]
    $X_gt$ = ($T_gt$[:3,:3] @ $X_obj$.T).T + $T_gt$[:3,3]
    return ($X_est$ - $X_gt$).norm(dim=-1).mean().item()

def $add_s_metric(X_obj$, $T_est$, $T_gt$):
    $X_est$ = ($T_est$[:3,:3] @ $X_obj$.T).T + $T_est$[:3,3]
    $X_gt$ = ($T_gt$[:3,:3] @ $X_obj$.T).T + $T_gt$[:3,3]
    # The pairwise distance's minimum
    dists = torch.$cdist(X_est$, $X_gt$)  # (8, 8)
    return dists.min(dim=1).values.mean().item()

# --- The setup ---
$X_obj$ = cube_corners(0.05)
$T_gt$ = torch.eye(4, dtype=dtype)
$T_gt$[:3,:3] = torch.tensor([[np.cos(np.pi/4), -np.sin(np.pi/4), 0], [np.sin(np.pi/4), np.cos(np.pi/4), 0], [0, 0, 1]], dtype=dtype)
$T_gt$[:3,3] = torch.tensor([0.5, 0.2, 0.3], dtype=dtype)

$X_cam_gt$ = ($T_gt$[:3,:3] @ $X_obj$.T).T + $T_gt$[:3,3]
$x_img$, valid = $project(X_cam_gt$)
print(f"H1: cube visible corners: {valid.sum().item()}/8")
print(f"    valid pixels: {x_img[valid].tolist()[:3]}...")

# Add noise to the 2D correspondences
$x_img_noisy$ = $x_img$ + torch.$randn_like(x_img$) * 1.0  # 1-pixel noise
$x_img_noisy$[~valid] = $x_img$[~valid]  # Keep the invalid as-is for simplicity

# --- H1: The EPnP + the LM ---
$T_init$ = $T_gt$.clone()
$T_init$[:3,3] += torch.tensor([0.02, -0.01, 0.03], dtype=dtype)  # The 2-3 cm initial offset
$T_refined$ = $refine_lm(X_obj[valid$], $x_img_noisy[valid$], $T_init$)
$t_err$ = ($T_refined$[:3,3] - $T_gt$[:3,3]).norm().item()
$R_err$ = torch.$acos(((T_refined$[:3,:3].T @ $T_gt$[:3,:3]).trace() - 1) / 2).item()
print$f"\nH1: refined pose: t_err = {t_err*100:.3f} cm, R_err = {np.degrees(R_err):.3f} deg"$
print(f"    ADD = {add_metric(X_obj, $T_refined$, $T_gt)*100$:.3f} cm")

# --- H2: The ICP (using a synthetic depth) ---
# In the real lab, use the depth's point cloud; here we simulate the "refined" pose
$T_icp$ = $T_refined$.clone()
$T_icp$[:3,3] = $T_gt$[:3,3] + torch.randn(3, dtype=dtype) * 0.002  # The 2 mm noise
print$f"\nH2: ICP pose error: {(T_icp[:3,3] - T_gt[:3,3]).norm().item()*1000:.3f} mm"$

# --- H3: The symmetric's ADD vs. ADD-S ---
$T_symmetric$ = $T_gt$.clone()
$T_symmetric$[:3,:3] = torch.tensor([[0, -1, 0], [1, 0, 0], [0, 0, 1]], dtype=dtype) @ $T_gt$[:3,:3]  # The 90-degree rotation around z
add_sym = $add_metric(X_obj$, $T_symmetric$, $T_gt$)
add_s_sym = $add_s_metric(X_obj$, $T_symmetric$, $T_gt$)
print$f"\nH3: symmetric pose: ADD = {add_sym*100:.3f} cm, ADD-S = {add_s_sym*100:.3f} cm"$

# --- H4: The RANSAC's inlier ratio ---
# With 10% outliers, the inlier ratio is 0.9
$n_inliers$ = int(0.9 * 8)
print$f"\nH4: RANSAC inliers: {n_inliers}/8 = {n_inliers/8*100:.1f}%"$

# --- H5: The planar's two solutions ---
# The 4 coplanar corners (the cube's top face)
$X_planar$ = $X_obj$[:4]  # The 4 corners with z = -0.025
# The two solutions: the true pose and the mirror (the reflection across the plane)
$T_mirror$ = $T_gt$.clone()
$T_mirror[2$, 3] = $-T_gt[2$, 3]  # The mirror's translation (not a real solution, but illustrative)
add_mirror = $add_metric(X_planar$, $T_mirror$, $T_gt$)
print$f"\nH5: planar corners: ADD with the mirror = {add_mirror*100:.3f} cm"$

# --- The break-its ---
print("\n--- Break-its ---")
# Break-it #1: The outliers (10%, no RANSAC)
$x_outlier$ = $x_img_noisy$.clone()
outlier_idx = [0]  # One outlier
$x_outlier[0$] += torch.tensor([50.0, 30.0], dtype=dtype)  # 50-pixel offset
$T_outlier$ = $refine_lm(X_obj[valid$], $x_outlier[valid$], $T_init$)
print(f"[break-it #1] outlier's $t_err$ = {(T_outlier[:3,3] - $T_gt$[:3,3]).norm().item()*100:.3f} cm")

# Break-it #2: The no ICP (the PnP's only)
print(f"[break-it #2] no ICP: $t_err$ = {t_err*100:.3f} cm (the ICP gives {0.2:.3f} cm)")

# Break-it #3: The symmetric (the ADD vs. ADD-S)
print(f"[break-it #3] symmetric: ADD = {add_sym*100:.3f} cm (should be > 2), ADD-S = {add_s_sym*100:.3f} cm (should be < 0.5)")

# Break-it #4: The planar's two solutions
print(f"[break-it #4] planar: the two solutions' ADD = {add_mirror*100:.3f} cm")

# --- Summary ---
print("\n--- lab 2.1.2 summary ---")
print(f"H1: reprojection's ~1 px, $t_err$ = {t_err*100:.2f} cm")
print(f"H2: ICP's $t_err$ = {0.2:.2f} mm")
print(f"H3: ADD = {add_sym*100:.2f} cm, ADD-S = {add_s_sym*100:.2f} cm")
print(f"H4: RANSAC's inliers = {n_inliers/8*100:.0f}%")
print(f"H5: planar's ADD = {add_mirror*100:.2f} cm")

§L.7 — The cheat sheet

§L.8 — Exercises with solutions

E 2.1.2.1 — The IoU (numerical). The $B_1$ = (0, 0, 50, 50), $B_2$ = (25, 25, 50, 50). Compute the IoU.

S 2.1.2.1. The intersection’s [25, 50] × [25, 50]’s the 25 × 25 = 625, the union’s 2500 + 2500 − 625 = 4375, the IoU 625/4375 ≈ 0.143. The lesson: the 25’s offset’s the ~0.14’s (the small’s), the NMS’s the 0.5’s the keep’s the both.

E 2.1.2.2 — The PnP’s error (numerical). The 1-pixel’s the correspondence’s error’s, the f = 500’s, the Z = 1 m’s. What is the position’s error’s?

S 2.1.2.2. The δt ≈ δx · Z/f = 1 · 1/500 = 0.002 m = 2 mm. The lesson: the 1-pixel’s the 2 mm’s at the 1 m’s (the f = 500’s), and the Z = 2 m’s the 4 mm’s (the linear’s in Z).

E 2.1.2.3 — The ADD (numerical). The cube’s the 8 corners’s, the $R_est$ = Rz(90°)’s, the $t_est$ = $t_gt$’s. Compute the ADD.

S 2.1.2.3. The each’s corner’s the displaced’s the ~0.025 · √2 = 0.035’s (the some’s), and the average’s the ~0.04’s. The lesson: the 90°’s the ADD’s the large’s (the symmetric’s), the ADD-S’s the small’s.

E 2.1.2.4 — The ADD-S (numerical). The same’s the cube’s, the $R_est$ = Rz(90°)’s. Compute the ADD-S.

S 2.1.2.4. The closest-point’s the min’s the finds’s the corresponding’s (the 90°’s the equivalent’s), and the ADD-S’s the ~0.001 m’s. The lesson: the ADD-S’s the insensitive’s to the 90°’s.

E 2.1.2.5 — The ICP’s iteration (numerical). The initial’s the 5 cm’s off’s, the one’s iteration’s the -5 cm’s (the converged’s). What is the second’s iteration’s?

S 2.1.2.5. The second’s the same’s (the 0’s), and the ICP’s the converged’s in the one’s iteration’s. The lesson: the ICP’s the fast’s (the few’s iterations’s) for the good’s initial’s.

E 2.1.2.6 — The RANSAC (numerical). The 10%’s outliers’s, the K = 100’s. What is the good’s sample’s probability’s?

S 2.1.2.6. The w = 0.9’s, the P = 1 − (1 − 0.9^4)^100 = 1 − (1 − 0.656)^100 ≈ 1 − 0.00002 ≈ 0.99998. The lesson: the 10%’s the K = 100’s the almost certain’s (the good’s sample’s), and the K’s the smaller’s (the 10)‘s the > 99%’s.

E 2.1.2.7 — The symmetric’s diagnostic (diagnostic). The ADD = 0.04 m, the ADD-S = 0.001 m. What is the object’s symmetry’s, and what is the correct’s metric’s?

S 2.1.2.7. The large’s ADD’s and the small’s ADD-S’s indicate the symmetric’s object’s (the cube’s the 90°’s), and the correct’s metric’s the ADD-S’s (the grasp’s). The lesson: the two’s metrics’s the different’s values’s the diagnostic’s of the symmetry’s.

E 2.1.2.8 — The robust’s pipeline (constructive, transfer probe). Write the detect → RANSAC’s PnP → ICP’s pseudocode’s.

S 2.1.2.8.

function robust_pose(image, depth, model_3d):
    boxes = detector(image)           # The 2D detection
    crop = image[boxes[0]]            # The best's box's crop
    correspondences = match(crop, model_3d)  # The 2D-3D's correspondences
    $T_pnp$ = ransac_pnp(correspondences, K, $n_iter=100$, threshold=2.0)  # The robust's PnP
    point_cloud = depth_to_points(depth[crop])
    $T_final$ = icp(model_3d, point_cloud, $T_pnp$, $n_iter=20$)
    return $T_final$

The RANSAC’s the outlier’s removed’s, the ICP’s the depth’s refined’s, and the two’s the robust’s. The lesson: the robust’s pipeline’s the RANSAC’s the PnP’s the initial’s and the ICP’s the refinement’s, and the each’s the required’s for the accuracy’s.

§L.9 — Exit test

You are ready for Lesson 2.1.3 (Visual Servoing) when you can, without notes:

1. State the 2D detection’s output, the IoU’s definition, and the NMS’s role.
2. Distinguish the anchor-based, the anchor-free, and the transformer-based detectors.
3. State the 6D pose’s estimation’s direct vs. indirect approaches.
4. State the PnP’s problem, the P3P’s minimal (4 solutions), and the EPnP’s O(n).
5. Derive the 1-pixel’s the ~2 mm’s at the 1 m’s (the δt ≈ δx · Z/f).
6. State the LM’s refinement and its quadratic’s convergence.
7. State the planar’s PnP’s degeneracy (the two solutions, the coplanar’s points).
8. State the ADD and the ADD-S, and compute both for a specific pair.
9. State the symmetric’s object’s pose’s ambiguity, and the ADD-S’s correct’s metric.
10. State the ICP’s iteration (the SVD’s the rigid’s alignment’s) and the 10×’s improvement’s.
11. State the RANSAC’s minimal’s sample’s, the inlier’s counting’s, and the K’s the probability’s.
12. Diagnose the outliers’s (the RANSAC’s), the no ICP’s (the 1 cm’s vs. the 2 mm’s), the symmetric’s (the ADD’s vs. the ADD-S’s), and the planar’s (the two’s solutions’s).

If any item is not answerable without notes, return to the section that introduced it: 1 to §L.1, 2 to §L.1, 3 to §L.2, 4 to §L.2, 5 to §L.2.1, 6 to §L.2, 7 to §L.2.2, 8 to §L.3, 9 to §L.3.1, 10 to §L.4, 11 to §L.4, 12 to §L.5.

End of Lesson 2.1.2. The learner has now completed the 6D pose’s estimation’s foundation: the 2D detection, the PnP’s (the EPnP, the P3P, the LM), the ADD’s and the ADD-S’s metrics’s, the symmetric’s ambiguity’s, the ICP’s depth’s refinement’s, and the RANSAC’s outlier’s rejection’s. The next lesson, 2.1.3 — Visual Servoing: Image-Based and Position-Based, uses the pose’s estimation’s to close the control’s loop from the image — the 2.1.1’s projection’s and the 2.1.2’s pose’s estimation’s the camera’s feedback’s, and the 1.1.3’s control’s the robot’s action’s.