Lesson 3.3.3 — Grounding Language to Affordances

Week 25

Prerequisite lessons: 3.3.1 (LLM task planning and decomposition), 3.3.2 (code-as-policies), 2.3.1 (grasp synthesis and quality), 2.3.3 (contact-rich manipulation), 2.1.2 (object detection and 6D pose estimation), 1.1.2 (inverse kinematics).

Prior results used in this lesson. From 3.3.1: the predicate-computation division (the LLM names a predicate, the verifier computes it) and the four-layer grounding stack. From 3.3.2: the primitive contract and the four acceptance criteria. From 2.3.1: force closure and the Ferrari–Canny $\varepsilon$ quality metric. From 2.3.3: the friction cone and the grasp-force budget. From 2.1.2: the 6D pose estimator and its covariance. Each is restated at the point of use.

What this lesson hands forward. The affordance predicate as the formal object that bridges a verb phrase to a geometric feasibility check, the analytic/learned/hybrid triad, and the semantic-versus-geometric verification split. Lesson 3.3.4 (safety and constraints) uses the affordance predicates as the constraint set that an LLM’s output is checked against. Module 4.2 uses the affordance layer as the reasoning layer’s interface to the policy layer.

The running case in one sentence. The running arm, gripper, eye-in-hand camera, table, cube, bin, and divider are unchanged; the instruction is the same conditional as 3.3.1, and this lesson asks how the verb “put” is turned into a set of geometric predicates that the robot’s capability set can evaluate, and what happens when an affordance is geometrically true but semantically wrong.

§L.0 — Lesson map, objectives, and misconceptions

Observable outcomes. By the end of this lesson you can, without notes: (1) write the affordance predicate with its four arguments and explain why it is relational; (2) distinguish an object label from an affordance and give the many-to-many mapping; (3) compute the $\varepsilon$-based graspability of the cube at a candidate pose; (4) state the three grounding families and their trade-offs; (5) write the affordance-map representation and its post-processing; (6) map a verb phrase to an affordance set and name the polysemy; (7) state the argument-role problem and its verification; (8) diagnose the four affordance-grounding failure modes from their signatures; (9) apply the semantic-versus-geometric split; (10) assemble the running case’s predicate set with the table as the semantically wrong case.

Misconception inventory.

Everything is self-contained. Where a prerequisite lesson is used, the result is restated at the point of use.

§L.0.5 — Retrieval warm-up

Q1. In 3.3.1, the four-layer grounding stack was reference resolution, predicate evaluation, pose lookup, and action-space mapping. Which layer does an affordance belong to, and why is the answer not obvious?

Answer. Affordances sit between predicate evaluation and pose lookup: they are predicates, but their computation requires the pose and the capability. The stack’s layers are not disjoint for affordances, which is why they need their own treatment. Used in §L.A.1 to define the affordance’s arguments.

Q2. In 3.3.2, the primitive grasp(name) had precondition is_graspable(name) and at_grasp_pose(name). Which of the two is an affordance, and which is a state check?

Answer. is_graspable is the affordance — a property of the object relative to the gripper — while at_grasp_pose is the robot’s current state. Used in §L.D.1 to distinguish the affordance’s truth from the robot’s readiness.

Q3. In 2.3.1, a grasp is force-closed when every external wrench can be resisted by contact forces inside the friction cones, and the $\varepsilon$ metric is the radius of the largest wrench-space ball. What does $\varepsilon = 0$ mean geometrically?

Answer. It means the wrench space’s ball is degenerate: the grasp can resist some wrenches and not others, and the direction it cannot resist is the one that will fail first. Used in §L.A.2 and WE-L.B.a.

Q4. In 2.3.3, the force budget for the rotated cube grasp required $F \ge 1.39$ N at rest and $1.67$ N under a 2 m/s² acceleration. What does that budget say about the pose of the grasp?

Answer. The budget depends on the misalignment angle $\theta = 45°$, which is a property of the pose; a different grasp pose would have a different $\theta$ and a different budget. The affordance is pose-dependent. Used in §L.A.1’s relational definition.

Part A — What an affordance is

§L.A.1 — The relational definition

An affordance is a property that is defined relative to an agent. The term was introduced by Gibson (1979, tier A, S) to describe what an environment offers an animal: a surface is “climbable” for a squirrel and not for a horse, and the difference is not in the surface alone but in the relation between the surface’s geometry and the animal’s capability.

For a robot, the definition is formal. An affordance is a predicate

where $o$ is the object (its geometry and material), $g$ is the gripper’s capability (opening, force, DOF), $q$ is a candidate pose (the approach direction and the contact points), and $\pi$ is the primitive the robot intends to apply. The predicate is true when the primitive $\pi$ applied to $o$ at pose $q$ is feasible for $g$.

This is not the same as a label. A label is a function of $o$ alone: object_type("cube") = "block". An affordance is a function of four arguments, and changing any one of them can flip the truth value. The distinction is the lesson’s foundation, and it is the first misconception in formal form.

Consider the running case’s cube. It is 5 cm across, the gripper opens 0–8 cm, the friction coefficient is $\mu = 0.5$, the mass is 0.1 kg, and the cube’s orientation is $R_z(\pi/4)$. Is the cube graspable?

The question is underspecified. Graspable by this gripper — yes, because the opening exceeds the width. Graspable at a face-centre pose — yes, because the fingers can span the face. Graspable at the top face with a downward approach — depends on the wrist’s clearance. Graspable with the standard 5 N force at $45°$ misalignment — yes, with a margin of 3.6 from Eq. (B.5) of 2.3.3. Each answer is a different instantiation of the four arguments, and each is the correct answer to a different question.

The LLM’s verb names the affordance; the verifier instantiates it. The division restated from 3.3.1 §L.E.2 is the same division: the LLM cannot compute the predicate, and the verifier cannot name it from the verb.

Self-explanation 1. Why is “graspable” not a property of the cube in the way “5 cm” is?

Answer. Because “5 cm” is true of the cube regardless of any gripper, while “graspable” is false for a gripper with a 4 cm opening and true for one with an 8 cm opening; the affordance’s truth depends on the second argument.

§L.A.2 — The predicate’s four arguments in the running case

The four arguments are worth instantiating one at a time, because each is a place where the grounding can fail.

The object $o$. The cube’s geometry is known from the 2.1.2 pose estimate and the object’s model; its friction coefficient is the calibrated $\mu = 0.5$. The table’s geometry is known but its friction is not calibrated in the same way, and its surface is not graspable in the intended sense — which is the semantic problem of §L.D.2.

The gripper $g$. The parallel-jaw gripper’s opening is 0–8 cm, its closing velocity 0.05 m/s, its force up to 40 N, and its standard grip 5 N. These are the capability parameters, and they appear in every affordance’s computation.

The pose $q$. The candidate grasp pose is a 6-DOF pose; for a parallel-jaw gripper the relevant subspace is the approach direction and the two contact points. The 2.3.1 planner samples this space and returns the poses whose $\varepsilon$ is above threshold.

The primitive $\pi$. The primitive is grasp, place_at, push, or release. Each has its own feasibility condition, and the same $(o, g, q)$ can be feasible for one and infeasible for another. A pose that is a valid grasp is not necessarily a valid placement, because the placement requires the object to be released, which requires the fingers to open without colliding.

The four arguments make the affordance a testable object. The verifier’s job is to instantiate the four and evaluate the predicate. The LLM’s job is to name the predicate and bind its object argument from the instruction’s reference.

Worked-example ladder.

WE-L.A.a (fully worked). Compute the graspability of the cube at a face-centre pose with the standard 5 N grip, using the $\varepsilon$ metric and the force budget of 2.3.3.

Solution. From 2.3.3 Eq. (B.5), the minimum finger force at rest for the $45°$-rotated grasp is 1.39 N, and the standard grip is 5 N, so the force budget has a margin of 3.6. From 2.3.1, the $\varepsilon$ metric for a two-contact planar grasp against the cube’s faces is positive when the line joining the contacts is inside the friction cones; at $45°$ misalignment the half-angle is $\arctan(0.5) = 26.6°$, which is less than $45°$, so the grasp is not force-closed against arbitrary wrenches. The affordance is true for the gravity direction (which the force budget confirms) and false for arbitrary wrenches. The predicate is direction-relative.

WE-L.A.b (completion). Compute the affordance’s truth for the same grasp with the gripper opening reduced to 4 cm. Completion. The opening is smaller than the cube’s 5 cm width, so the fingers cannot span the face; the affordance is false at every face-centre pose, and only an edge or corner grasp with a different contact geometry might succeed. The completion shows the gripper-argument’s effect.

WE-L.A.c (faded). Write the affordance predicate for place_at with the same four arguments, and state the condition that the grasp’s affordance does not encode. Faded step. placeable(o, g, q, place) = true iff the pose is reachable, the path is clear, and the release does not collide with the container. Completion. The condition the grasp’s affordance does not encode is the container’s opening, which the placement’s affordance adds; the two predicates share the object and gripper but differ in the pose and the primitive.

WE-L.A.d (contrastive near-miss). A learner labels the cube “graspable” in the scene serialization and the program calls grasp("cube") at a pose the 2.1.2 estimator returned with a 3 cm covariance. The grasp misses. Why it fails: the label was computed at a pose, and the pose was wrong; the affordance is pose-relative, and the pose’s uncertainty was not in the label. The fix: carry the pose covariance into the affordance check, so that a pose whose covariance is above the tolerance makes the affordance false regardless of the geometry.

Comparative table: label versus affordance.

Design principle P44.1. An affordance is a predicate over object, gripper, pose, and primitive; never collapse it to a label, because the label is true where the predicate is false.

Consolidation checkpoint 1

(a) Three capabilities. You can write the affordance predicate with its four arguments. You can compute the cube’s graspability at a face-centre pose. You can state the label-versus-affordance distinction with an example.

(b) Transfer probe. The cube’s mass doubles to 0.2 kg. Does the graspability at the face-centre pose change, and which argument changed? Answer: the object’s mass changed, which enters the force budget of 2.3.3; the affordance’s truth can flip because the required force exceeds the available grip. The argument that changed is the object’s.

(c) Remediation rule. If you cannot state why reducing the gripper’s opening flips the affordance, re-read §L.A.1 before Part B.

Part B — Analytic, learned, and hybrid grounding

§L.B.1 — The analytic route

The analytic route computes the affordance from geometry and the capability model. For grasping, it is the pipeline of 2.3.1: sample candidate poses, compute the contact points, form the grasp matrix, test force closure, compute the $\varepsilon$ metric, and keep the poses above threshold. The result is a set of feasible grasps, and the affordance predicate is “there exists a pose in the set”.

The analytic route’s strengths are its guarantees. A pose returned by 2.3.1 is force-closed by construction (or by the $\varepsilon$ threshold), and the force budget of 2.3.3 is a check on the same geometry. The computation is deterministic, the result is reproducible, and the failure is explainable: a pose is rejected because the friction cone is violated, which is a sentence the operator can act on.

Its weaknesses are two. It requires an object model — the geometry and the friction coefficient — and the running case’s cube is modelled but the cup, screwdriver, and sponge are not. It also samples the pose space coarsely, so it can miss a feasible grasp that is off the sample grid, and the missed grasp is the one the affordance should have found.

§L.B.2 — The learned route

The learned route predicts the affordance from data. Affordance maps (Zeng et al. 2018, tier A, S) predict, per pixel, the probability that a grasp centred at that pixel succeeds; the GG-CNN (Morrison et al. 2018, tier B, S) predicts the grasp pose and quality per pixel in one pass. AffordanceNet (Do, Nguyen, Reid 2018, tier A, S) predicts object masks and their affordance labels jointly. The learned route’s input is a single image, its output is a dense map, and its inference is a single forward pass.

The learned route’s strengths are its coverage and its speed. It does not need an object model, it generalizes to objects it has seen in training, and its inference is tens of milliseconds. It is the mechanism that makes a cell with five objects and no CAD models workable.

Its weaknesses are the ones 3.1.4 §L.C.2 named for any learned function: its predictions at unseen objects and unseen poses are uncalibrated, and its failure is silent because the map’s output is a probability that looks like a confidence. A learned affordance map that predicts 0.9 on a grasp that will crush the cube is not a map with a 10% error; it is a map whose confidence is miscalibrated at that pose.

The second weakness is representation. The map’s output is a grasp pose per pixel, which is a 4-DoF parameterization (image position, angle, opening) — it does not cover the full 6-DOF grasp space, and the running case’s rotated cube needs the 6-DOF parameterization that 2.3.1 supplies.

Self-explanation 1. Why is a learned affordance map’s confidence not a probability of success?

Answer. Because the map is trained on a dataset whose distribution is not the deployment’s, and its softmax or regression output is a score whose calibration holds on the training distribution and not outside it.

§L.B.3 — The hybrid route

The hybrid route uses the learned map as a proposal and the analytic check as a verification. The learned map produces a small set of candidate poses; the analytic check evaluates each against the object’s geometry and the gripper’s capability; the surviving poses are the affordance.

The hybrid’s logic is the 3.3.2 primitive contract applied to the affordance: the learned map is a fast, unverified proposal, and the analytic check is the deterministic verifier that the deployment relies on. The proposal’s coverage supplies the objects the analytic route could not model; the verification’s guarantee supplies the safety that the learned route cannot.

The hybrid’s cost is the analytic check’s latency, which is the 2.3.1 computation’s cost per candidate. The proposal must be small — five to twenty candidates, not the full map — or the check’s cost dominates.

For the running case, the hybrid is the right choice, and §L.B.4 makes it the default. The cube has a model, so the analytic check is exact; the cup and the screwdriver do not, but the learned map proposes poses on them, and a crude analytic check using a primitive geometric model (a cylinder for the cup) verifies the candidates.

Worked-example ladder.

WE-L.B.a (fully worked). A learned affordance map predicts 0.92 at pixel (320, 240) for a grasp at the cube’s face centre. The analytic check computes $\varepsilon = 0.03$ and the force budget margin at 5 N as 3.6. Compute the hybrid’s decision.

Solution. The map’s 0.92 is above the proposal threshold (say 0.8), so the candidate enters the verification. The analytic check’s $\varepsilon = 0.03$ is above the minimum (say 0.01) and the force margin is positive, so the candidate is accepted. The affordance is true at that pose.

WE-L.B.b (completion). The map predicts 0.9 at a pixel corresponding to a grasp on the table’s edge. The analytic check’s $\varepsilon$ is positive (a table edge is graspable) and the force margin is large. What does the hybrid decide, and what is missing? Completion. The hybrid decides the table edge is graspable — which is geometrically true. What is missing is the semantic argument: the primitive was grasp("table"), and the table is not the intended object. The affordance check passes and the grounding is wrong, which is the semantic gap of §L.D.2.

WE-L.B.c (faded). Derive the proposal threshold from the analytic check’s false-negative rate: if the analytic check rejects 30% of the map’s top candidates, what proposal threshold keeps the true feasible grasps? Faded step. Sweep the threshold and measure the accepted-affordance recall on a held-out set. Completion. The threshold is chosen empirically; the principle is that the proposal’s job is recall, not precision, and the verification supplies precision.

WE-L.B.d (contrastive near-miss). A learner uses the learned map alone, with a 0.9 threshold, on the running case’s cup. The map predicts 0.95 at a grasp on the cup’s handle, and the cup’s handle is 6 mm thick while the gripper’s minimum opening is 8 mm. Why it fails: the map’s training distribution did not include the gripper’s minimum opening constraint, and the pose is infeasible for that capability. The fix: the analytic check’s gripper-argument test rejects the pose, which is exactly the hybrid.

Comparative table: the three grounding families.

Design principle P44.2. Learn the proposal, verify the affordance; the learned map supplies coverage and the analytic check supplies the guarantee, and neither alone is deployable.

Consolidation checkpoint 2

(a) Three capabilities. You can state each family’s strengths and weaknesses. You can compute the hybrid decision for a proposal-and-check pair. You can explain why the learned map’s confidence is not calibrated.

(b) Transfer probe. A cell has one object with an exact CAD model and a 10 ms budget. Which family? Answer: analytic, because the model is available and the budget admits it; the learned map’s coverage advantage is unused on a single modelled object.

(c) Remediation rule. If you cannot state why the hybrid’s proposal threshold is chosen for recall, re-read §L.B.3 before Part C.

Part C — The verb lexicon

§L.C.1 — Verbs as bundles

The LLM’s instruction’s verbs are the linguistic interface to the affordances, and the mapping is not one-to-one. Three patterns account for most of the mismatch.

A verb can be a bundle. “Pick up” is grasp plus lift; “put” is transport plus release; “open” is grasp plus rotate. The bundle’s members are the actual predicates, and the program that implements the verb is a sequence of primitive calls. The 3.3.2 program for “put the cube in the bin” had five calls for one verb.

A verb can be polysemous. “Pick up” can mean grasp-and-lift for a cube, scoop for a sponge, or pinch for a coin. The disambiguation is geometric: the object’s shape and the gripper’s capability select the member. The LLM’s prior over the verb is a distribution over the members; the affordance check is what selects.

A verb’s argument can be implicit. “Put it in the bin” has an implicit “it” that refers to the previously grasped object. The reference is resolved by the program’s state, not by the sentence, and the LLM must track it.

The lexicon table is the operational artifact: for each verb, its member predicates, its primitive sequence, and its argument roles.

The table’s role is the same as the 3.3.2 primitive contract’s: it tells the LLM what the verb’s implementation is, and it tells the verifier what predicates to check.

Concept check. The instruction says “open the bin”. The bin has a lid with a handle. Which bundle members apply, and what does the affordance check verify?

Answer. The bundle is grasp-the-handle plus rotate; the affordance check verifies the handle is graspable by the gripper’s opening and the lid is rotatable within the hinge’s range. [displaces M3]

§L.C.2 — The argument-role problem

A verb’s arguments have roles: the object acted on, the target, the instrument, the direction. The roles are what the affordance’s instantiation uses, and the LLM’s job is to bind each role to a scene entity.

The binding has two verification conditions.

Existence. The bound entity exists in the scene; the 3.3.1 reference check restated.

Type compatibility. The bound entity’s geometry is compatible with the role. An object bound to the “target” role of “put” must be a container — its affordance is placeable-into — and an object bound to the “object” role must be graspable.

The type compatibility is where the semantic check enters. The LLM’s prior over roles is semantic (“bins are targets”), and the affordance check is geometric (“the target is a container”). A target that is not a container fails the geometric check; an object that is a container but bound to the object role passes the geometric check and fails the semantic one.

§L.C.3 — The lexicon’s coverage

The lexicon’s coverage determines the reasoning layer’s vocabulary. A verb not in the lexicon cannot be grounded, and the LLM must either be re-prompted with the available verbs or the lexicon must be extended.

The coverage’s design is a trade. A large lexicon covers more instructions and is harder to verify; a small lexicon is verifiable and limits the instruction space. The running case’s lexicon is five verbs, chosen to cover the cell’s tasks and no more, and the 3.3.4 safety layer uses the lexicon’s size as a bound on the LLM’s output’s surface.

The coverage’s failure mode is a silent fallback: the LLM, faced with an uncovered verb, maps it to the nearest covered verb and produces a program that looks reasonable and is semantically wrong. The signature is a verb whose mapping the operator did not intend, and the fix is to reject uncovered verbs explicitly rather than fall back.

Worked-example ladder.

WE-L.C.a (fully worked). The instruction “put the cube in the bin” has one verb and two arguments. Instantiate the roles and the predicates.

Solution. Verb: put. Roles: object = cube, target = bin1 or bin2. Predicates: graspable(cube), placeable(cube, target), blocked(cube, target) for the conditional. Primitive sequence: grasp, transport, place_at, release. The program of 3.3.2 §L.E.1 is the instantiation.

WE-L.C.b (completion). Add the conditional’s second target and bind the roles. Completion. The conditional’s branch selects bin2 when blocked(cube, bin1) is true; the roles are re-bound at execution time, which is why the program’s conditional is on the live scene. The completion shows the roles’ binding is dynamic.

WE-L.C.c (faded). Extend the lexicon with “stack” and write its member predicates and primitive sequence. Faded step. Stack: graspable(top), placeable-on(top, base), alignable. Primitives: grasp, transport, place-on. Completion. “Stack” adds placeable-on, which is a different predicate from placeable-into; the container’s opening is replaced by the base’s support surface.

WE-L.C.d (contrastive near-miss). The instruction says “clean the table”. The lexicon has no clean verb. The LLM maps it to push and produces a program that pushes the cube off the table. Why it fails: the silent fallback; clean is not push, and the mapping was chosen because no covered verb fit. The fix: reject uncovered verbs explicitly and ask the operator to restate the instruction in the lexicon’s terms.

Comparative table: argument roles and their checks.

Design principle P44.3. A verb is a bundle and its arguments are roles; ground both, and reject an uncovered verb rather than mapping it to the nearest one.

Consolidation checkpoint 3

(a) Three capabilities. You can expand a verb into its member predicates and primitive sequence. You can bind the argument roles and check their types. You can state the silent-fallback failure.

(b) Transfer probe. “Place the cup on the plate” has verb “place” and roles object = cup, target = plate. Which predicate differs from “put the cup in the bin”? Answer: placeable-on replaces placeable-into; the cup rests on the plate rather than being contained by it, and the plate’s geometry is a support surface, not a container.

(c) Remediation rule. If you cannot expand a verb of the lexicon into its predicates, re-read §L.C.1 before Part D.

Part D — Verification and the semantic gap

§L.D.1 — The predicate is a precondition

The affordance predicate is the precondition of the primitive, restated from 3.3.2 §L.B.1. The grasp primitive’s precondition is_graspable(name) is the affordance’s instantiation, and the executor’s precondition guard is the affordance check’s enforcement.

The restatement has an important consequence: the affordance must be evaluated at execution time, not at planning time. The pose changes, the object can move, and the gripper’s state can change between the plan and the call. The 3.3.2 semantic race is the failure of a planning-time affordance, and the precondition guard is the execution-time fix.

The second consequence: the affordance’s evaluation cost is on the execution’s critical path. A 2.3.1 grasp-quality computation at 20 candidate poses is a few milliseconds; a learned map’s inference is tens of milliseconds. The cost is acceptable for the grasp primitive and not for the place_at primitive, whose feasibility check is cheaper and should not be padded.

§L.D.2 — The semantic gap

The semantic gap is the difference between a geometric affordance and the intended action. The table is graspable; the operator did not intend to grasp it. The cube is graspable; so is the sponge, and the instruction said “the cube”.

The gap’s structure is worth stating precisely. The affordance predicate $A(o, g, q, \pi)$ is a function of geometry and capability. The instruction’s reference $o^\star$ is a function of the operator’s intent. The grounding is correct when $o = o^\star$ and $A(o, g, q, \pi)$ is true. The first condition is semantic; the second is geometric.

The two conditions are checked by different mechanisms.

The semantic condition is checked by the 3.3.1 reference check: the instruction’s reference must resolve to exactly one object, and the resolution must be consistent with the scene’s labels. The check is a name-and-description match, and it is vulnerable to perceptual aliasing (two cubes) and to description mismatch (the operator says “the box” and the scene has “cube”).

The geometric condition is checked by the affordance predicate. It is robust to the semantic errors (it does not care whether the object is called a cube) and vulnerable to the geometric ones (a pose whose covariance is large makes the predicate false even when the object is right).

The two checks are complementary, and the deployment needs both. A system with only the semantic check grasps the wrong object with a feasible grasp. A system with only the geometric check grasps the right object with an infeasible one — or grasps the table.

Self-explanation 2. Why is the semantic check not reducible to a geometric check?

Answer. Because intention is not a geometric property; the table’s graspability is a fact about the table, and the operator’s not intending to grasp it is a fact about the instruction.

Concept check. The instruction says “grasp the cube”, the scene has two cubes, and the LLM picks the nearer one. Which check catches the ambiguity, and what does the verifier do?

Answer. The 3.3.1 reference check, because “the cube” is not unique; the verifier rejects and the LLM is re-prompted with a disambiguation. The geometric check would pass for either cube, which is why it cannot catch the ambiguity. [displaces M4]

§L.D.3 — The four grounding failure modes

Each failure has a signature, and each signature maps to a check.

Semantic mis-binding. The object is bound to the wrong entity, or the roles are swapped. Signature: the program’s first call’s argument is the wrong name. Check: reference uniqueness and description match.

Geometric false positive. The affordance is true where it should be false — a pose that passes the check and fails at execution. Signature: the primitive’s postcondition fails. Check: the analytic verification of the learned proposal, and the postcondition.

Geometric false negative. The affordance is false where it should be true — a feasible pose the check rejected. Signature: the program’s recovery loops, or the planner reports infeasible for a reachable target. Check: the coverage of the proposal and the tightness of the analytic threshold.

Semantic-geometric mismatch. The object is the right one and the pose is feasible, and the verb’s bundle is wrong — the operator said “push” and the LLM implemented “grasp”. Signature: the primitive sequence does not match the verb’s lexicon entry. Check: the lexicon’s expansion at parse time.

The four modes are ordered by how early the check runs: the semantic mis-binding and the mismatch at parse, the false positive at execution, the false negative at recovery. The earlier the catch, the cheaper the fix.

Worked-example ladder.

WE-L.D.a (fully worked). A program calls grasp("table") at a table-edge pose. The reference check passes (the table is in the scene), the geometric check passes (the edge is force-closed), and the program grasps the table. Diagnose the failure and propose the fix.

Solution. The failure is a semantic mis-binding: the instruction was “grasp the cube”, and the program bound the object to “table”. The fix is the 3.3.1 reference check with a description match — “the cube” must resolve to an object whose label is “cube” — and the LLM’s re-prompt with the mismatch. The geometric check cannot catch this, which is the semantic gap.

WE-L.D.b (completion). The reference check passes and the table is still grasped, because the instruction was “grasp it” with “it” referring to the cube, and the LLM bound “it” to the table. Completion. The reference check catches the pronoun’s binding only if the program’s state tracks the previous reference; the fix is the program’s state, not the check. The completion shows the semantic check’s dependence on the program’s history.

WE-L.D.c (faded). Write the lexicon’s parse-time check for the verb’s bundle and give an example of a mismatch. Faded step. The check expands the verb and compares the program’s first primitive to the expansion’s first primitive. Completion. The example is “push the cube” implemented as grasp("cube"); the check rejects because push’s expansion begins with contact, not grasp.

WE-L.D.d (contrastive near-miss). A learner’s verifier checks only the geometric affordance and finds that the cell frequently grasps the sponge when the instruction says “cube”. Why it fails: both objects are graspable, and the geometric check cannot distinguish them. The fix: the reference check’s description match, plus the scene’s labels.

Comparative table: failure modes and their checks.

Design principle P44.4. Check the semantic condition and the geometric condition separately; each catches a failure the other cannot see.

Consolidation checkpoint 4

(a) Three capabilities. You can state why the affordance must be evaluated at execution time. You can diagnose the four failure modes from their signatures. You can apply the semantic-geometric split to a deployment.

(b) Transfer probe. The program grasps the right cube at a pose whose covariance is 2 cm. Which failure mode? Answer: a geometric false positive — the pose passed the check and the actual pose is within 2 cm of the intended; the postcondition catches it, and the fix is the covariance check of §L.A.2.

(c) Remediation rule. If you cannot state why the geometric check cannot catch the table grasp, re-read §L.D.2 before Part E.

Part E — The running case

§L.E.1 — The predicate set

The instruction’s predicates, with the lexicon of §L.C.1:

where $c$ is the cube, $b_1$ the primary bin, and $b_2$ the second bin. The predicates are instantiated from the verb “put” and its roles, with the conditional’s blocked predicate selecting the target.

The predicates’ computations:

• graspable(c): the 2.3.1 $\varepsilon$ at the 2.1.2 pose, with the covariance check of §L.A.2.
• placeable(c, b): the 2.3.2 feasibility of the target pose and the path’s clearance.
• blocked(c, $b_1$): the 2.3.2 clearance check from the cube’s pose to the primary bin’s opening.

The three predicates are the preconditions of the grasp, place_at, and is_blocked primitives, and the 3.3.2 harness enforces them at execution time.

§L.E.2 — The table and the semantic gap

The running case has a table spanning $x \in [0, 1]$ and $y \in [-0.5, 0.5]$ at $z = 0$. The table is graspable at its edge: a parallel-jaw gripper can span the edge, the friction coefficient is adequate, and the $\varepsilon$ check passes. The table is not the intended object, and the semantic check is what excludes it.

The instruction “put the cube in the bin” has the cube as the object role and the bin as the target role. The table is neither, so the reference check rejects a program that binds the table to either role. The affordance check would have accepted the table’s edge, which is why the semantic check is not optional.

The lesson’s operational statement for the running case: every grasp call’s argument is checked twice — once against the instruction’s references and once against the affordance — and both checks must pass.

§L.E.3 — The acceptance criteria

The affordance layer’s acceptance is the 3.3.2 §L.E.3 conjunction with one addition.

Correctness. The program achieves the instruction on the nominal scene; the predicates’ instantiations are correct.

Robustness. The program achieves the instruction on 80% of the scene variations; the predicates’ false negatives do not dominate.

Safety. No call’s precondition is violated, including the semantic check; the table is never grasped.

Traceability. Every predicate’s evaluation is logged with its arguments and its result, so a failure is attributable to a predicate and an argument.

The addition is the predicate coverage: the fraction of the instruction’s verbs and roles that the lexicon grounds. A cell whose instruction uses a verb outside the lexicon is not deployable, regardless of its success rate, because the silent fallback’s success is not the intended success.

Self-explanation 3. Why is a silent-fallback success not a success?

Answer. Because the operator’s intent was not the executed action; the cell achieved a different task and the operator’s instruction was not followed, which is the definition of a reasoning failure regardless of the outcome’s plausibility.

Worked-example ladder.

WE-L.E.a (fully worked). The affordance layer grounds 4 of the instruction’s 5 verbs. The 5th is “stack”, which the lexicon does not cover. Compute the coverage and the deployment decision.

Solution. Coverage is 0.8, below the 1.0 threshold; the deployment is blocked, and the fix is to add “stack” to the lexicon with its predicates and primitive sequence, or to re-prompt the operator to restate in covered verbs.

WE-L.E.b (completion). The lexicon is extended with “stack” and the coverage is 1.0. The program succeeds on 85% of the variations, with 0 semantic or geometric violations. Compute the decision. Completion. Coverage 1.0, correctness met, robustness 85% > 80%, safety 0 violations, traceability in place; the deployment is accepted.

WE-L.E.c (faded). The table is bound to the object role by an LLM program and the reference check rejects it. Write the rejection reason and the re-prompt. Faded step. Reason: “reference: table is not the object role’s entity; instruction names cube”. Re-prompt: original instruction, scene’s labels, reason. Completion. The re-prompt’s directive is at the end, as in 3.3.1 §L.D.3.

WE-L.E.d (contrastive near-miss). A learner adds the table to the scene’s “graspable objects” list to allow the clean verb, and finds that the cell now grasps the table when the instruction is “grasp the cube”. Why it fails: the affordance’s object list was conflated with the instruction’s roles, and the semantic check was bypassed. The fix: the scene’s list is a fact about the table, not a permission for the program; the reference check uses the instruction’s roles, not the scene’s list.

Comparative table: the running case’s predicates.

Design principle P44.5. Every grasp call is checked twice — against the instruction’s roles and against the affordance — and the two checks are both necessary and neither is sufficient.

Consolidation checkpoint 5

(a) Three capabilities. You can instantiate the running case’s predicate set. You can state the table’s semantic gap. You can apply the four acceptance criteria with predicate coverage.

(b) Transfer probe. The scene adds a second cube at a different pose. What changes in the predicate set and the checks? Answer: the reference check’s uniqueness fails and the LLM is re-prompted for disambiguation; the geometric predicates are unchanged because each cube is separately graspable.

(c) Remediation rule. If you cannot state why the silent fallback is a reasoning failure, re-read §L.E.3 before the lab.

Failure taxonomy

The table covers the three lab corruptions plus three failures not exercised in the lab: stale scene, cell refusal, and slow check.

The lab

Task. Compute the analytic graspability predicate for the cube at a candidate pose using the 2.3.1 $\varepsilon$ metric and the 2.3.3 force budget, compare it with a stub learned-affordance score at the same pose, and run the hybrid decision on a set of poses that includes the table’s edge as the semantically wrong case.

Conditions. Python 3.10+, NumPy only. Cube 5 cm, mass 0.1 kg, $\mu = 0.5$, gripper opening 0-8 cm, standard grip 5 N. The stub learned map returns a score per pose from a pre-written table. Seed all RNGs.

Metrics.

Pre-registered hypotheses.

H1. The cube’s face-centre pose has positive epsilon and a force margin above 3, and the hybrid accepts it.

H2. The table edge’s pose has positive epsilon and a force margin above 3, and the hybrid also accepts it — the semantic gap.

H3. The reference check rejects the table edge’s program, and the deployment’s two checks together accept the cube and reject the table.

H4. The analytic check’s latency is at least an order of magnitude above the stub learned map’s, motivating the hybrid’s small proposal set.

Break-it sequence.

1. Remove the semantic check silently. The table edge is grasped. Silent: the geometric check passes and the program is valid.
2. Remove the covariance check silently. The cube is grasped at a high-covariance pose and the grasp misses. Silent: the pose’s point estimate is correct.
3. Add the table to the graspable list silently. The reference check’s role binding is bypassed. Silent: the program is syntactically valid.

Acceptance criteria.

What good looks like. A good artefact has five properties. (1) The epsilon and the force margin are reported per pose, not aggregated. (2) The table edge is shown to pass the geometric check, which is the semantic gap made visible. (3) The two checks are separate functions, so the deployment’s conjunction is explicit. (4) The break-it results are a table. (5) The covariance check is a parameter, not a constant.

A runnable core, seeded and annotated:

# lab_3_3_3.py — analytic affordance and the semantic gap
import numpy as np
np.random.seed(0)

CUBE_HALF, CUBE_MASS, MU, G = 0.025, 0.1, 0.5, 9.81
GRIP_OPEN, STANDARD_GRIP = 0.08, 5.0

def epsilon_two_contact(p1, p2, normal, mu=MU):           # Ferrari-Canny, planar
    line = (p2 - p1); line /= np.linalg.norm(line)
    angle = np.arccos(np.clip(abs(line @ normal), -1, 1))  # misalignment
    cone = np.arctan(mu)                                   # half-angle
    return max(0.0, cone - angle)                          # 0 if outside the cone

def force_margin(mass=CUBE_MASS, mu=MU, F=STANDARD_GRIP, theta=np.pi/4):
    required = mass * G / (2 * mu * np.cos(theta))         # 2.3.3 Eq. (B.5)
    return F / required                                    # margin factor

def graspable_analytic(pose, cov, obj_half=CUBE_HALF):
    if 2 * obj_half > GRIP_OPEN:                           # gripper argument
        return False, "opening too small"
    if cov > 0.01:                                         # pose covariance
        return False, f"covariance {cov:.3f} too large"
    p1, p2 = pose - np.array([obj_half, 0, 0]), pose + np.array([obj_half, 0, 0])
    eps = epsilon_two_contact(p1, p2, np.array([1.0, 0, 0]))
    if eps <= 0.01:                                        # epsilon threshold
        return False, f"epsilon {eps:.3f} below threshold"
    if force_margin() < 1.5:                               # force margin
        return False, "force margin below 1.5"
    return True, "ok"

def stub_learned_map(pose):                                # stand-in for a CNN
    return 0.92 if abs(pose[0] - 0.5) < 0.1 else 0.3       # cube region

def hybrid(pose, cov, semantic_role):
    ok, reason = graspable_analytic(pose, cov)
    if not ok:
        return False, reason
    if semantic_role != "object":                          # semantic check
        return False, f"semantic: role is {semantic_role}"
    if stub_learned_map(pose) < 0.8:                       # proposal threshold
        return False, "learned score below threshold"
    return True, "ok"

poses = [
    ("cube face-centre", np.array([0.5, 0.2, 0.3]), 0.005, "object"),
    ("cube face-centre, high cov", np.array([0.5, 0.2, 0.3]), 0.03, "object"),
    ("table edge", np.array([0.5, 0.5, 0.0]), 0.005, "object"),
    ("bin rim", np.array([0.4, -0.3, 0.15]), 0.005, "target"),
]
for name, pose, cov, role in poses:
    ok, reason = hybrid(pose, cov, role)
    print(f"{name:28s} {'ACCEPT' if ok else 'reject':7s}  {reason}")

Line 9 is the epsilon computation: the misalignment between the contact line and the face’s normal, compared to the friction cone’s half-angle. Line 13 is the force budget of 2.3.3 at the $45°$ misalignment. Lines 16–25 are the analytic affordance: the four arguments are checked in order, and each rejection names the argument. Lines 27–28 are the stub learned map, a stand-in for the CNN proposal. Lines 30–38 are the hybrid: the analytic check, then the semantic role, then the proposal threshold. The four poses show the cube’s correct acceptance, the covariance rejection, the table’s geometric acceptance (the semantic gap made visible), and the bin’s rejection on role.

Cheat sheet

Exercises with solutions

E 3.3.3.1. Write the affordance predicate with its four arguments and instantiate each for the cube’s grasp.

S 3.3.3.1. $A(o, g, q, \pi)$. Object: the cube’s geometry and $\mu = 0.5$. Gripper: opening 8 cm, standard grip 5 N. Pose: the face-centre pose with the $45°$ misalignment. Primitive: grasp. The lesson: the predicate is only meaningful when all four arguments are bound.

E 3.3.3.2 (transfer). The gripper’s opening is reduced to 4 cm. Which argument changed, and does the affordance flip at the face-centre pose?

S 3.3.3.2. The gripper argument changed; the opening is smaller than the cube’s 5 cm, so the affordance is false at every face-centre pose. The lesson: the affordance is relational, and the gripper’s capability is one of the relations.

E 3.3.3.3. Compute the force margin for the cube’s grasp at $45°$ with a 5 N grip and a 2 m/s² acceleration.

S 3.3.3.3. Required force $= 0.1 \times 11.81 / (2 \times 0.5 \times 0.7071) = 1.67$ N; margin $= 5/1.67 = 3.0$. The lesson: the margin depends on the pose’s misalignment and the acceleration, both of which are arguments of the affordance.

E 3.3.3.4 (diagnostic). A learned affordance map predicts 0.95 at a table edge and the program grasps it. Diagnose and propose two fixes.

S 3.3.3.4. The semantic gap: the table edge is geometrically graspable and semantically wrong. Fixes: (1) the reference check binds the program’s argument to the instruction’s role, rejecting the table; (2) the scene’s label set excludes the table from the object role. The lesson: the geometric check cannot catch the semantic error, and the semantic check is the fix.

E 3.3.3.5. Expand the verb “put” into its member predicates and primitive sequence.

S 3.3.3.5. Predicates: graspable(object), placeable(object, target), blocked(object, target). Primitives: grasp, transport, place_at, release. The lesson: a verb is a bundle, and the bundle’s members are the predicates the verifier checks.

E 3.3.3.6 (transfer). The instruction says “stack the cube on the second cube”. Which predicates differ from “put the cube in the bin”?

S 3.3.3.6. placeable-on replaces placeable-into, and blocked is replaced by aligned and stable. The target role is a support surface rather than a container. The lesson: the verb’s bundle determines the predicates, and a different verb’s bundle is a different predicate set.

E 3.3.3.7 (diagnostic). The program’s grasp postcondition fails at a pose whose covariance is 2 cm. Which failure mode and which check?

S 3.3.3.7. A geometric false positive: the pose passed the check at its point estimate and failed at execution because the covariance was within the check’s blind spot. The postcondition catches the execution failure; the covariance check is the fix. The lesson: the covariance is an argument of the affordance, and ignoring it is a false-positive mechanism.

E 3.3.3.8. Write the lexicon expansion for “open” and state which predicate’s evaluation requires the hinge’s geometry.

S 3.3.3.8. Predicates: graspable(handle), rotatable(lid); primitives: grasp, rotate. The rotatable predicate requires the hinge’s axis and range. The lesson: the predicates can require object-specific geometry that the scene’s model must supply.

E 3.3.3.9 (transfer). A cell’s instruction uses a verb outside the lexicon and the LLM silently falls back to a covered verb. Diagnose and state the fix.

S 3.3.3.9. The silent fallback; the executed task is not the operator’s intent. The fix is to reject uncovered verbs explicitly and ask the operator to restate in covered verbs, or to extend the lexicon. The lesson: coverage is a deployment criterion, and a fallback’s success is not a success.

Exit test

You are ready for Lesson 3.3.4 when you can, without notes:

1. Write the affordance predicate with its four arguments and explain why it is relational.
2. Distinguish an object label from an affordance with an example.
3. Compute the cube’s epsilon and force margin at a candidate pose.
4. State the three grounding families and their trade-offs.
5. Explain why a learned map’s confidence is not calibrated.
6. Write the hybrid decision and state its proposal and verification roles.
7. Expand a verb into its member predicates and primitive sequence.
8. Bind the argument roles and check their existence and type.
9. State the silent-fallback failure and its fix.
10. Explain why the affordance must be evaluated at execution time.
11. State the semantic gap and give the table as its instance.
12. Diagnose the four failure modes from their signatures.
13. Apply the semantic and geometric checks as a conjunction.
14. Instantiate the running case’s predicate set.
15. Apply the acceptance criteria with predicate coverage.

Cross-references. 3.3.1 supplies the predicate-computation division and the four-layer grounding stack restated in §L.A.1 and §L.D.2. 3.3.2 supplies the primitive contract and the acceptance criteria restated in §L.D.1 and §L.E.3. 2.3.1 supplies the epsilon metric restated in §L.A.2. 2.3.3 supplies the force budget restated in WE-L.A.a. 2.1.2 supplies the pose and its covariance restated in §L.A.2. 1.1.2 supplies the inverse kinematics invoked in the reachability check. Lesson 3.3.4 uses the affordance predicates as the constraint set for the safety layer. Module 4.2 uses the affordance layer as the reasoning layer’s interface to the policy layer.

Ready. The next lesson to write is 3.3.4 — Safety and Constraints on LLM Outputs. Say “next” to generate it.