Week 36. Prerequisite: Modules 1–20, especially Module 3 (context, attention), Module 5 (evaluation), Module 7 (verifiers), Module 11 (trace, cost), Module 15 (datasets), Module 17b (verifier calibration), and Part 0, §§0.5, 0.9.

21.0 Module map and objectives

Modules 1–20 built the harness for the coding agent and the tool-using agent — the domains where the action space is a set of typed functions and the verifier is a test suite. This module extends the discipline to the three domains where the harness’s assumptions break: computer use (the action space is a click and a keystroke, not a function; the observation is a screen, not a text result), autonomous research (the verifier is not a test suite, and the verification is often the bottleneck rather than the capability), and the high-stakes verticals (medicine, law, finance, where the verifier is a human expert and the failure’s cost is severe).

The module’s central claim is that the harness’s pattern is the same — the four necessary elements (Module 1), the context engineering (Module 3), the verifier calibration (Module 17b), the autonomy threshold (Module 17c) — and the domain’s difference is in the verifier’s construction and the autonomy’s economics. The module’s discipline is therefore the verification-first design: specify the verifier before designing the agent, and let the verifier’s operating point drive the agent’s architecture, the autonomy’s level, and the deployment’s economics.

The module has five parts and one lab.

By the end of the module you will be able to do nine things:

1. Formalise the GUI agent’s perception–action grounding and identify its error sources (the observation’s ambiguity, the state’s ambiguity, the action’s precision).
2. Characterise OSWorld and WebArena by their triples (the task distribution, the verifier, the protocol) and state what each does and does not measure.
3. Derive the persistence’s optimum for a browsing task and explain why the browsing benchmarks reward it.
4. State the GUI agent’s four failure modes and the verification’s difficulty (the app-state’s verifier’s coverage).
5. Describe the research agent’s anatomy and state the verification gap’s formal structure.
6. Name the research verification’s options (the reproduction, the citation’s check, the expert’s review, the internal consistency) and rank them by the TPR, the FPR, and the cost.
7. State the survey’s findings (the trace’s release’s rate, the absence of the externally validated in-loop oracle) and their status (P).
8. Apply the domain adaptation’s dimensions to a high-stakes vertical and design the human–expert integration (the expert as the verifier, the collaborator, the overseer).
9. Produce the verification-first specification for a domain, with the verifier’s operating point, the autonomy’s threshold, the residual risk, and the revision condition.

Everything is self-contained. Where Part 0 or a prior module is used, the result is restated.

PART A — COMPUTER-USE AND GUI/BROWSER AGENTS

21.1 What a GUI agent is

21.1.1 The definition

Definition 21.1.1 (GUI agent). A GUI agent is an agent whose action space is the screen’s interaction: the pointer’s (the click, the drag, the scroll), the keyboard’s (the typing, the shortcuts), and the observation’s is the screen’s (the pixels, the accessibility tree, or the DOM). The environment is the application (the browser, the desktop, the mobile app), and the task is the user’s goal (the booking, the form’s submission, the settings’ change).

The GUI agent is the harness’s special case where the tool is the computer’s interface itself rather than a typed function. The four necessary elements (Module 1, §1.1.2) are still present: the loop, the “tool interface” (the screen’s interaction), the context manager (the screen’s history), and the control mechanisms (the permissions, the bounds). The difference is in the action space’s form and the observation’s form.

21.1.2 The action space’s form

The typed tool call (Module 2) is a discrete, structured action: the function’s name and the arguments. The GUI action is the low-level, unstructured (or partially structured):

• The pixel’s coordinates: the click at $(x, y)$. The action is continuous, and the model must produce the coordinates.
• The element’s reference: the click on the element with the accessibility’s ID, the role, or the label. The action is discrete but requires the observation’s grounding.
• The keyboard’s: the typing’s string, the shortcut’s keys. The action is the string or the key’s combination.

The three forms are the abstraction’s levels, and the choice is the harness’s design: the pixel’s is the most general and the most error-prone; the element’s is the most reliable and the least general; the keyboard’s is the middle.

21.1.3 The observation’s form

The GUI agent’s observation is the screen, in one of three modalities:

• The screenshot (the pixels): the raw image, the highest fidelity, the highest token cost (the image’s tokens), the least structured.
• The accessibility tree: the elements’ roles, names, and bounding boxes, the structured, the medium cost, the incomplete (some elements are not exposed).
• The DOM (for the web): the HTML’s structure, the most structured, the medium cost, the web-specific.

The modality’s choice is the harness’s design, and it determines the grounding’s difficulty: the screenshot’s requires the visual grounding, the accessibility’s requires the element’s selection, the DOM’s requires the selector’s.

21.2 The perception–action grounding

21.2.1 The formal statement

Definition 21.2.1 (The grounding’s function). The GUI agent’s grounding is the function

where $\mathcal{O}$ is the observation’s space (the screen’s modality), $\mathcal{T}$ is the task’s description, and $\mathcal{A}$ is the action’s space (the coordinates, the element’s reference, the keyboard’s). The grounding is the model’s mapping from “what I see and what I want” to “what I do.”

The grounding’s error is the mismatch between the intended action and the executed action: the agent intended to click the “Submit” button, and the executed action clicked the “Cancel” button (the observation’s ambiguity) or the wrong coordinates (the precision’s error) or the stale element (the state’s ambiguity).

21.2.2 The three error sources

Source 1 — The observation’s ambiguity. Two elements look the same (the identical buttons, the small text, the colour’s contrast). The grounding’s function cannot distinguish them from the observation alone. The measurement: the element’s identification’s accuracy on the ambiguous’s cases.

Source 2 — The state’s ambiguity. The screen’s is the transitional (the page’s loading, the modal’s overlay, the animation’s frame). The observation is the stale’s or the incomplete’s, and the grounding’s function acts on the wrong state. The measurement: the action’s failure’s rate on the tasks with the asynchronous state.

Source 3 — The action’s precision. The coordinates are the off-by-a-few-pixels (the click lands on the edge, the element’s target’s is small), or the element’s is the partially obscured. The grounding’s function is the right’s (the element is the correct), and the execution is the wrong’s. The measurement: the action’s success’s rate for the small targets.

21.2.3 The grounding’s accuracy

The grounding’s accuracy is the composite:

where the three are the element’s identification’s, the action’s execution’s, and the state’s correctness’s (the screen is the current’s). The product’s is the step’s success’s rate, and the task’s success’s rate is the product over the steps (the compounding, Module 14, §14.7.3).

Worked example. A 20-step task with the per-step’s 0.95’s. The task’s success’s $0.95^{20}=0.358$. A per-step’s 0.99’s gives $0.99^{20}=0.818$. The GUI’s compounding’s is the same as the tiered’s architecture’s (Module 14), and the per-step’s accuracy’s is the binding’s constraint.

21.2.4 The grounding’s improvement

The grounding’s improvement’s options:

• The element’s reference’s (the accessibility’s ID) instead of the pixels’s: the discrete’s action’s removes the precision’s error.
• The observation’s enrichment’s: the screenshot’s + the accessibility’s tree’s (the multi-modal’s), the ambiguity’s reduction.
• The state’s check’s: the wait’s for the page’s load’s, the modal’s dismissal’s, the explicit’s state’s verification’s before the action’s.
• The action’s verification’s: the post-action’s check’s (the screen’s changed’s as expected’s), the retry’s on the mismatch’s.

21.3 OSWorld

21.3.1 The triple

The task distribution $\mathcal{T}$: 369 real computer tasks across the OS applications (the file manager, the browser, the code editor, the spreadsheet). The tasks are the multi-step (the minutes’s), and the environments are the real’s (the Ubuntu’s VM’s, the applications’s).

The verifier $\mathcal{V}$: the final state’s of the filesystem or the application, checked by the script (the state’s comparison’s, the file’s content’s, the application’s setting’s).

The protocol $\mathcal{P}$: the agent controls the mouse and the keyboard (or the accessibility’s API), the one’s episode’s per task, the no’s human’s help’s.

21.3.2 What it measures

The OSWorld measures the GUI control’s capability: the perception’s (the screen’s), the planning’s (the multi-step’s), the grounding’s (the element’s selection’s), and the execution’s (the click’s, the typing’s). It is the closest’s widely’s-used’s proxy’s for the computer-use’s agent’s.

21.3.3 What it does not measure

• The reliability’s: the one’s episode’s per task, the pass’s@1’s, the no’s pass’s^k’s (Module 5, §5.5). The variance’s across the runs is’s not’s the benchmark’s focus’s.
• The long-horizon’s: the tasks’s are the minutes’s, not the hours’s.
• The recovery’s: the agent’s failure’s on a step’s is’s the task’s failure’s, and the benchmark’s does not’s measure’s the recovery’s ability’s.
• The security’s: the tasks’s are the benign’s, and the injection’s (Module 19) is’s not’s the benchmark’s.
• The cost’s: the tokens’s, the time’s, the tool’s calls’s are’s not’s the primary’s metric’s.

21.4 WebArena

21.4.1 The triple

The task distribution $\mathcal{T}$: 812 tasks across the four self-hosted web applications (the shopping’s site’s, the forum’s, the GitLab’s instance’s, the CMS’s).

The verifier $\mathcal{V}$: the final state’s of the application (the item’s in the cart’s, the post’s created’s, the code’s committed’s).

The protocol $\mathcal{P}$: the agent controls the browser’s and navigates’s the applications’s.

21.4.2 What it measures

The WebArena measures the web navigation’s and the task’s completion’s in the realistic’s applications’s. It is the closest’s proxy’s for the browser’s agent’s, and it is the lower’s contamination’s (the applications’s are the self-hosted’s, the task’s instances’s are the synthetic’s).

21.4.3 What it does not measure

• The open web’s: the applications’s are the self-hosted’s and the stable’s; the real’s web is’s not’s.
• The adversarial’s: the tasks’s do not’s include’s the adversarial’s content’s (the Module 19’s injection’s).
• The long-horizon’s and the cost’s: the same’s as the OSWorld’s.

21.5 Why browsing benchmarks reward persistence

21.5.1 The mechanism

The browsing’s benchmarks (the BrowseComp’s, the long-horizon’s web’s suites’s) have the tasks’s where the answer’s is the hard-to-find’s (the specific’s fact’s, the obscure’s page’s), and the verifier’s is the exact’s answer’s. The agent’s that’s keeps’s searching’s (the more’s queries’s, the more’s links’s) is’s the more’s likely’s to’s find’s the answer’s.

Proposition 21.5.1 (The persistence’s bound). If each query’s finds’s the answer’s with the independent’s probability’s $p$, the success’s probability’s after’s $n$‘s queries’s is’s $1-(1-p)^n$ (the Module 4, §4.6’s oracle’s bound’s). The success’s is’s the increasing’s in’s $n$, and’s the benchmark’s (the exact’s answer’s) rewards’s the $n$‘s increase’s.

Argument. The independence’s is’s the assumption’s; the queries’s are’s the different’s, and’s the answer’s is’s the found’s if’s at least’s one’s query’s finds’s it’s. The $1-(1-p)^n$‘s is’s the standard’s result’s (Module 4, §4.6.2’s). $\square$

21.5.2 The optimum

The persistence’s optimum’s is’s the marginal’s benefit’s equals’s the marginal’s cost’s:

where’s $V$‘s is’s the value’s of the answer’s and’s $c_{\text{query}}$‘s is’s the query’s cost’s (the tokens’s, the latency’s). The solution’s:

Worked example. $p=0.05$, $V=1.0$, $c_{\text{query}}=0.01$. The $n^* = \log(0.01/(0.05\times1.0))/\log(0.95) = \log(0.2)/\log(0.95) = (-1.609)/(-0.0513) = 31.4$. The optimum’s is’s the $\approx31$‘s queries’s. The lesson’s: the low’s $p$‘s (the hard-to-find’s) needs’s the many’s queries’s, and’s the benchmark’s rewards’s the persistence’s.

21.5.3 The risk

The persistence’s risk’s: the no’s convergence’s. The agent’s that’s keeps’s searching’s without’s the finding’s is’s the loop’s (Module 7, §7.21’s), and’s the benchmark’s (the exact’s answer’s) does’s not’s reward’s the stopping’s when’s the answer’s is’s not’s findable’s. The real’s browsing’s task’s need’s the calibrated’s stopping’s (the agent’s confidence’s that’s the answer’s is’s not’s findable’s), which’s the benchmark’s does’s not’s measure’s.

21.5.4 The concept’s check

Why does the browsing benchmark’s reward for persistence not transfer to the deployment’s reliability?

Answer. Because the benchmark’s verifier is the exact answer, and the persistence’s is the optimal’s strategy’s for the finding’s. The deployment’s needs the calibrated’s stopping’s (the agent’s knows’s when’s the answer’s is’s not’s findable’s, and’s stops’s), and the reliability’s (the agent’s does not’s fabricate’s the answer’s when’s the search’s fails’s). The benchmark’s rewards’s the $1-(1-p)^n$‘s, and the deployment’s needs’s the precision’s (the answer’s is’s the correct’s when’s the agent’s says’s it’s found’s). The two’s are’s the different’s metrics’s, and the persistence’s optimum’s is’s not’s the reliability’s optimum’s.

21.6 The GUI agent’s failure modes

21.6.1 The four modes

Mode 1 — The grounding’s error. The agent clicks the wrong element (the observation’s ambiguity, the precision’s). The detection: the post-action’s screen’s check’s.

Mode 2 — The state’s error. The agent acts on the stale’s screen (the loading’s, the transition’s). The detection: the explicit’s wait’s, the state’s verification’s.

Mode 3 — The plan’s error. The agent’s sequence’s is’s the wrong’s (the task’s decomposition’s is’s the incorrect’s, the wrong’s page’s, the wrong’s form’s). The detection: the task’s verifier’s (the final state’s).

Mode 4 — The loop’s. The agent’s repeats’s the action’s without’s the progress’s (the retry’s on the same’s failure’s, the navigation’s cycle’s). The detection: the Module 7, §7.21’s loop detector’s.

21.6.2 The failure’s distribution

The GUI agent’s failure’s distribution’s is’s the diagnostic’s (Module 11, §11.9’s): the grounding’s and the state’s are’s the perception’s (the agent’s sees’s wrong’s), the plan’s is’s the reasoning’s, and’s the loop’s is’s the control’s. The fix’s differs’s by the mode’s: the perception’s needs’s the observation’s improvement’s (the accessibility’s, the wait’s), the reasoning’s needs’s the plan’s check’s, the control’s needs’s the loop’s detector’s.

21.7 The verification of a GUI agent

21.7.1 The difficulty

The GUI agent’s verification’s is’s the final state’s (the application’s, the filesystem’s), and’s the state’s check’s is’s the script’s (the OSWorld’s, the WebArena’s). The difficulty’s:

• The state’s observability’s: the application’s internal’s state’s is’s not’s the screen’s; the verifier’s must’s inspect’s the application’s API’s or the storage’s, not’s the pixels’s.
• The state’s coverage’s: the task’s success’s is’s the specific’s state’s (the item’s in the cart’s, the correct’s address’s), and’s the verifier’s checks’s the specific’s fields’s; the fields’s the verifier’s misses’s are’s the coverage’s gap’s (Module 9, §9.7’s).
• The timing’s: the state’s is’s the async’s (the network’s, the backend’s), and’s the verifier’s must’s wait’s for the state’s convergence’s.

21.7.2 The verifier’s calibration

The GUI’s verifier’s is’s the deterministic’s (the script’s), so’s the TPR’s and’s the FPR’s are’s the high’s and’s the low’s (the Module 17b’s), conditional’s on the script’s correctness’s and’s the coverage’s. The calibration’s is’s the coverage’s measurement’s: the fraction’s of the task’s success’s modes’s the script’s checks’s, and’s the failure’s modes’s the script’s misses’s (the false’s accepts’s). The coverage’s is’s the GUI’s verifier’s TPR’s measure’s, and’s the missing’s coverage’s is’s the verification’s gap’s.

21.7.3 The concept’s check

Why is a GUI agent’s verifier’s TPR harder to measure than a coding agent’s?

Answer. Because the coding’s verifier’s (the test’s suite’s) is’s the behaviour’s (the function’s output’s), and’s the tests’s are’s the runnable’s on the arbitrary’s inputs’s. The GUI’s verifier’s is’s the final state’s (the application’s state’s), and’s the states’s are’s the specific’s (the one’s task’s state’s); the verifier’s cannot’s run’s the state’s on’s the arbitrary’s inputs’s, and’s the coverage’s (the success’s modes’s the state’s check’s captures’s) must’s be’s argued’s, not’s measured’s by’s the execution’s. The TPR’s is’s the coverage’s, and’s the coverage’s is’s the harder’s to’s measure’s. The consequence’s: the GUI’s verifier’s is’s more’s likely’s to’s have’s the false’s accepts’s (the state’s check’s passes’s but’s the task’s is’s not’s really’s done’s), and’s the calibration’s must’s include’s the held-out’s state’s checks’s and’s the human’s sample’s.

PART B — AUTONOMOUS RESEARCH AGENTS

21.8 The research agent’s anatomy

21.8.1 The pipeline

The research agent’s anatomy is the scientific’s pipeline:

Stage 1 — The literature’s search. The agent reads the papers, the prior’s work’s, and’s identifies’s the gap’s.

Stage 2 — The hypothesis’s. The agent proposes’s the hypothesis’s (the claim’s, the prediction’s).

Stage 3 — The experiment’s design. The agent designs’s the experiment’s (the method’s, the data’s, the analysis’s plan’s).

Stage 4 — The execution’s. The agent runs’s the experiment’s (the code’s, the data’s, the measurement’s).

Stage 5 — The analysis’s. The agent interprets’s the result’s (the statistical’s, the qualitative’s).

Stage 6 — The write-up’s. The agent produces’s the paper’s (the claim’s, the evidence’s, the citation’s).

The pipeline’s the long-horizon’s (the days’s, the many’s context’s windows’s), and’s the harness’s needs’s the Module 8’s multi-context’s execution’s, the Module 15’s data’s, and’s the Module 17’s environment’s.

21.8.2 The verifier’s at each stage

Each stage’s has’s its’s verifier’s, and’s the verifier’s strength’s varies’s:

21.8.3 The verification’s chain

The research’s verification’s is’s the chain’s: the claim’s is’s only’s as’s good’s as’s the weakest’s link’s (the Module 14, §14.7.3’s compounding’s). The literature’s check’s is’s the weakest’s (the low’s), and’s the execution’s reproduction’s is’s the strongest’s (the high’s). The chain’s the verification’s gap’s measure’s: the claim’s the agent’s produces’s is’s the verified’s at’s the chain’s weakest’s link’s.

21.9 The verification gap

21.9.1 The statement

Definition 21.9.1 (The verification gap). The verification gap is the condition in which the verification of the agent’s output is the limiting’s obstacle’s to the deployment’s, rather than the agent’s raw’s capability. The gap’s is’s the difference between the agent’s output’s rate’s and’s the verification’s rate’s:

where’s $r_{\text{out}}$‘s is’s the claims’s the agent’s produces’s per unit time’s, and’s $r_{\text{ver}}$‘s is’s the claims’s the verification’s can’s check’s per unit time’s.

The gap’s positive’s means’s the verification’s backlog’s grows’s, and’s the deployment’s is’s the verification’s bottleneck’s. The gap’s negative’s means’s the verification’s keeps’s up’s, and’s the agent’s is’s the bottleneck’s.

21.9.2 The survey’s framing

The autonomous research survey (arXiv:2608.05179, tier P: preprint, single-lab, survey-derived) reports’s:

• Only’s the 38%‘s of the runnable’s AI-scientist’s systems’s release’s the execution’s traces’s.
• No’s LLM-era’s system’s demonstrates’s the externally validated’s in-loop’s oracle’s (the verifier’s that’s is’s both’s reliable’s and’s used’s during’s the agent’s operation’s, not’s just’s at’s the end’s).
• The verification’s gap’s is’s the limiting’s obstacle’s for’s the research’s agents’s, more’s than’s the raw’s capability’s.

The status’s: P (the survey’s, the single-lab’s, the claims’s are’s the survey’s synthesis’s). The curriculum’s posture’s: the gap’s is’s the well-founded’s hypothesis’s, and’s the deployment’s must’s measure’s its’s own’s $r_{\text{out}}$‘s and’s $r_{\text{ver}}$‘s rather’s than’s assume’s the survey’s numbers’s.

21.9.3 The gap’s consequences

The gap’s consequences’s:

• The backlog’s: the unverified’s claims’s accumulate’s, and’s the deployment’s cannot’s use’s them’s.
• The quality’s degradation’s: the agent’s, facing’s the backlog’s, may’s be’s incentivized’s (by the reward’s) to’s produce’s the easy-to-verify’s claims’s, not’s the important’s ones’s (the Module 20, §20.10’s specification’s gaming’s).
• The trust’s erosion’s: the users’s cannot’s tell’s the verified’s from’s the unverified’s, and’s the trust’s in’s the system’s is’s the unverified’s.

21.10 The verification’s options

21.10.1 The four options

Option 1 — The reproduction’s. The experiment’s is’s re-run’s, and’s the result’s is’s compared’s. The TPR: high (the deterministic’s, if the experiment’s is’s the reproducible’s). The FPR: low (the reproduction’s of’s a correct’s result’s fails’s only’s if the environment’s differs’s). The cost: high (the experiment’s re-run’s, the days’s).

Option 2 — The citation’s check’s. The claim’s is’s checked’s against’s the source’s. The TPR: low (the citation’s is’s the necessary’s, not’s the sufficient’s; the misquotation’s is’s the common’s). The FPR: low. The cost: low (the search’s, the read’s).

Option 3 — The expert’s review’s. The domain’s expert’s reviews’s the claim’s and’s the evidence’s. The TPR: moderate’s to’s high (the expert’s judgment’s). The FPR: moderate’s (the expert’s bias’s, the fatigue’s). The cost: high (the expert’s time’s).

Option 4 — The internal’s consistency’s. The claim’s is’s checked’s against’s the agent’s own’s other’s claims’s and’s the data’s. The TPR: low (the consistency’s is’s not’s the correctness’s). The FPR: low. The cost: low (the automated’s).

21.10.2 The ranking

The ranking’s by the cost’s per correct’s verification’s (the Module 17c’s):

where’s the $\pi$‘s is’s the base’s rate’s (the fraction’s of the claims’s that’s are’s correct’s). The ranking’s depends’s on the $\pi$‘s and’s the $C_f$‘s:

• The high’s $C_f$‘s (the safety’s-critical’s): the reproduction’s and’s the expert’s review’s (the high’s TPR’s) are’s the worth’s, and’s the citation’s check’s is’s the supplementary’s.
• The low’s $C_f$‘s (the exploratory’s): the citation’s check’s and’s the internal’s consistency’s (the cheap’s) are’s the first’s, and’s the reproduction’s is’s the sample’s.

21.10.3 The in-loop’s vs the post-hoc’s

The survey’s key’s finding’s: the verifier’s is’s often’s the post-hoc’s (the verification’s at the end’s), not’s the in-loop’s (the verification’s at’s each’s step’s). The in-loop’s is’s the Module 16, §16.9’s VPR’s (the dense’s, the per-step’s), and’s the post-hoc’s is’s the outcome’s only’s. The in-loop’s is’s the harder’s (the per-step’s verifier’s must’s be’s the reliable’s), and’s it’s is’s the better’s (the credit’s assignment’s, the early’s stopping’s, the agent’s adaptation’s).

The research’s in-loop’s verifier’s: the the execution’s reproduction’s at’s each’s experiment’s (the code’s runs’s, the result’s matches’s), and’s the the literature’s check’s at’s each’s claim’s (the citation’s is’s the real’s). The two’s are’s the candidates’s for’s the in-loop’s oracle’s, and’s the survey’s says’s the no’s LLM-era’s system’s has’s demonstrated’s the externally validated’s one’s.

21.11 The research agent’s evaluation

21.11.1 The evaluation’s design

The research agent’s evaluation’s is’s the Module 5’s, with the research’s specifics’s:

• The task’s set’s: the research’s questions’s (the benchmark’s, the real’s).
• The verifier’s: the reproduction’s, the expert’s, the citation’s (the options’s).
• The metric’s: the claim’s correctness’s (the verifier’s verdict’s), the the novelty’s (the claim’s is’s not’s the known’s), the the significance’s (the claim’s matters’s).
• The cost’s: the experiment’s, the expert’s, the tokens’s.

21.11.2 The novelty and significance

The novelty’s and’s the significance’s are’s the hard’s to’s verify’s: the novelty’s requires’s the literature’s knowledge’s (the claim’s is’s not’s in’s the prior’s work’s), and’s the significance’s requires’s the domain’s judgment’s. The two’s are’s the expert’s (the Option 3’s), and’s the automated’s verifier’s is’s the weak’s.

21.11.3 The evaluation’s limits

The evaluation’s limits’s:

• The base’s rate’s: the research’s claims’s are’s the mostly’s wrong’s (the base’s rate’s is’s the low’s), and’s the verifier’s precision’s at’s the low’s base’s rate’s is’s the poor’s (Part 0, §0.5’s). The 99%‘s TPR’s and’s the 1%‘s FPR’s at’s the 5%‘s base’s rate’s gives’s the 0.84’s precision’s; at’s the 1%‘s base’s rate’s, the 0.50’s. The research’s verifier’s needs’s the very’s high’s TPR’s to’s be’s useful’s.
• The adversarial’s: the agent’s may’s produce’s the plausible’s claims’s that’s pass’s the citation’s check’s (the citation’s exists’s) but’s are’s not’s supported’s (the misquotation’s). The verifier’s must’s check’s the support’s, not’s just’s the existence’s.

21.12 The research agent’s harness

21.12.1 The harness’s requirements

The research agent’s harness’s needs’s:

• The multi-context’s execution’s (Module 8): the days’s-long’s, the many’s windows’s.
• The reproducibility’s (Module 7, §7.7): the experiment’s environment’s, the code’s, the data’s.
• The trace’s (Module 11): the execution’s trace’s, the claim’s evidence’s chain’s.
• The verifier’s integration’s (Module 17b): the in-loop’s reproduction’s, the post-hoc’s expert’s review’s.
• The cost’s control’s (Module 11, §11.13): the experiments’s are’s the expensive’s.

21.12.2 The harness’s differences from the coding’s

The research’s harness’s differs’s from the coding’s:

• The verifier’s is’s the partial’s: the coding’s has’s the test’s suite’s; the research’s has’s the reproduction’s (the high’s for the execution’s, the low’s for the claim’s).
• The horizon’s is’s the longer’s: the research’s days’s vs the coding’s minutes’s.
• The novelty’s: the research’s output’s is’s the new’s, and’s the verifier’s cannot’s compare’s it’s to’s the gold’s patch’s (there’s is’s none’s).
• The expert’s: the research’s verifier’s is’s the expert’s, and’s the expert’s availability’s is’s the bottleneck’s.

21.13 The autonomous research’s evidence

21.13.1 The systems

The autonomous research’s systems’s (the AI-scientist’s, the research’s agents’s) are’s the research’s prototypes’s (tier P: the preprints’s, the small’s evaluations’s). The systems’s produce’s the papers’s (the claims’s, the experiments’s), and’s the evaluations’s are’s the small’s (the few’s tasks’s, the narrow’s domains’s).

21.13.2 The evidence’s status

The evidence’s status’s: P (the single-lab’s, the unreplicated’s, the survey’s synthesis’s). The claims’s (the agent’s produces’s the novel’s research’s) are’s the hypotheses’s, and’s the deployment’s must’s measure’s its’s own’s.

21.14 The research agent’s risks

21.14.1 The risks

• The fabrication’s: the agent’s cites’s the non-existent’s papers’s, the invented’s results’s.
• The misquotation’s: the agent’s cites’s the real’s papers’s, the wrong’s claim’s.
• The p-hacking’s: the agent’s finds’s the significant’s result’s by’s the multiple’s comparisons’s (the Module 5, §5.6’s).
• The reproducibility’s failure’s: the experiment’s does’s not’s reproduce’s, and’s the claim’s is’s the unsupported’s.
• The specification’s gaming’s: the agent’s optimizes’s the metric’s (the number’s of’s papers’s, the citations’s) rather’s than’s the research’s quality’s (the Module 20, §20.10’s).

21.14.2 The defences

The defences’s:

• The reproduction’s (the Option 1’s): the experiment’s re-run’s, the result’s matches’s.
• The citation’s verification’s: the citation’s exists’s and’s the claim’s is’s in’s the source’s (the support’s, not’s the existence’s).
• The pre-registration’s: the hypothesis’s and’s the analysis’s plan’s are’s fixed’s before’s the data’s (the Module 5, §5.10’s), the p-hacking’s’s defence’s.
• The expert’s review’s (the Option 3’s): the domain’s expert’s.
• The trace’s release’s: the execution’s trace’s is’s public’s, and’s the reproduction’s is’s the external’s (the survey’s 38%‘s is’s the signal’s that’s the trace’s release’s is’s the exception’s).

21.15 The concept’s check

Why is the research agent’s verification gap the capability’s problem, not just the evaluation’s problem?

Answer. Because the gap’s changes’s the agent’s incentive’s: if the verification’s is’s the bottleneck’s, the agent’s (optimized’s by the RL’s, Module 16’s) learns’s to produce’s the easy-to-verify’s claims’s (the citation’s check’s, the internal’s consistency’s) rather’s than’s the important’s ones’s (the novel’s, the significant’s, which’s need’s the reproduction’s and’s the expert’s). The gap’s is’s the Module 20, §20.10’s specification’s gaming’s at’s the research’s level’s: the agent’s optimizes’s the verifiable’s, not’s the valuable’s. The capability’s (the agent’s can’s produce’s the important’s claims’s) is’s not’s the deployment’s (the agent’s does’s produce’s the important’s claims’s), and’s the gap’s is’s the reason’s the two’s differ’s. The fix’s is’s the verifier’s design’s (the in-loop’s reproduction’s, the expert’s), not’s the agent’s capability’s.

PART C — DOMAIN ADAPTATION IN HIGH-STAKES VERTICALS

21.16 The transfer’s dimensions

21.16.1 The five dimensions

The transfer’s from the coding’s or the general’s agent’s to the high-stakes’s vertical’s has’s the five’s dimensions’s:

Dimension 1 — The task’s structure. The structured’s (the form’s filling’s, the code’s editing’s) vs the unstructured’s (the diagnosis’s, the legal’s argument’s).

Dimension 2 — The verifier’s strength. The strong’s (the test’s suite’s, the deterministic’s check’s) vs the weak’s (the expert’s judgment’s, the outcome’s).

Dimension 3 — The data’s sensitivity. The public’s vs the private’s, the personal’s, the regulated’s.

Dimension 4 — The regulation. The low’s (the internal’s tools’s) vs the high’s (the EU’s AI’s Act’s high-risk’s, the sectoral’s).

Dimension 5 — The human’s expertise. The available’s (the expert’s in the loop’s) vs the unavailable’s (the expert’s is’s the bottleneck’s).

21.16.2 The high-stakes’s profile

The high-stakes’s vertical’s (the medicine’s, the law’s, the finance’s) profile’s:

• The unstructured’s task’s (the diagnosis’s, the argument’s).
• The weak’s verifier’s (the expert’s, the outcome’s).
• The sensitive’s data’s (the patient’s, the client’s).
• The high’s regulation’s (the MDR’s, the FDA’s, the legal’s ethics’s, the financial’s conduct’s).
• The required’s expert’s (the doctor’s, the lawyer’s, the analyst’s).

The profile’s the hardest’s transfer’s: the verifier’s is’s the weak’s, the data’s is’s the sensitive’s, the regulation’s is’s the high’s, and’s the expert’s is’s the requirement’s.

21.17 The human–expert integration

21.17.1 The three roles

The expert’s integration’s has’s the three’s roles’s:

Role 1 — The expert as the verifier. The expert’s checks’s the agent’s output’s (the diagnosis’s, the legal’s memo’s). The expert’s TPR’s and’s FPR’s are’s the verifier’s operating’s point’s (the Module 17b’s), and’s the autonomy’s threshold’s (the Module 17c’s) uses’s them’s.

Role 2 — The expert as the collaborator. The expert’s works’s with’s the agent’s (the agent’s drafts’s, the expert’s refines’s). The expert’s is’s the co-author’s, not’s the checker’s, and’s the division’s of’s labor’s is’s the design’s.

Role 3 — The expert as the overseer. The expert’s supervises’s the agent’s process’s (the plan’s, the steps’s), not’s just’s the output’s. The expert’s catches’s the process’s errors’s (the wrong’s plan’s, the missed’s step’s), and’s the expert’s is’s the the control’s (the Module 20, §20.6’s) human’s.

21.17.2 The integration’s design

The integration’s design’s:

• The role’s clarity’s: which’s role’s the expert’s plays’s (the verifier’s, the collaborator’s, the overseer’s), and’s the scope’s.
• The interface’s: what’s the expert’s sees’s (the output’s, the process’s, the evidence’s), and’s the time’s.
• The authority’s: the expert’s can’s approve’s, reject’s, modify’s, stop’s (the Module 7, §7.15’s gate’s).
• The measurement’s: the expert’s TPR’s and’s FPR’s (the audit’s sample’s, the Module 20, §20.23’s).

21.17.3 The expert’s bottleneck

The expert’s is’s the bottleneck’s: the expert’s time’s is’s the scarce’s resource’s, and’s the agent’s output’s rate’s is’s the high’s. The integration’s design’s must’s allocate’s the expert’s time’s by the risk’s (the high’s risk’s cases’s get’s the expert’s, the low’s risk’s get’s the agent’s + the sample’s). The allocation’s is’s the Module 17c’s verification’s budget’s, with’s the expert’s as’s the verifier’s.

21.18 The high-stakes verifiers

21.18.1 The medical

The medical’s verifier’s:

• The clinical’s outcome’s: the patient’s result’s (the delayed’s, the noisy’s).
• The guideline’s adherence’s: the deviation’s from’s the clinical’s guidelines’s (the checkable’s, the incomplete’s).
• The expert’s review’s: the physician’s (the TPR’s, the FPR’s’s, the Module 20, §20.23’s).
• The formal’s: the drug’s interaction’s checker’s, the dosing’s calculator’s (the deterministic’s, the high’s TPR’s for the specific’s checks’s).

21.18.2 The legal

The legal’s verifier’s:

• The statute’s check’s: the citation’s is’s the current’s law’s.
• The precedent’s check’s: the case’s is’s the good’s law’s, the jurisdiction’s.
• The expert’s review’s: the lawyer’s (the TPR’s, the FPR’s).
• The consistency’s: the memo’s is’s consistent’s with’s the facts’s and’s the law’s.

21.18.3 The financial

The financial’s verifier’s:

• The audit’s: the reconciliation’s, the ledger’s balance’s (the deterministic’s).
• The regulatory’s: the compliance’s check’s (the AML’s, the KYC’s, the disclosure’s).
• The expert’s review’s: the analyst’s, the compliance’s officer’s.
• The model’s risk’s: the value-at-risk’s, the stress’s test’s (the statistical’s, the model’s-dependent’s).

21.18.4 The common pattern

The common’s pattern’s: the deterministic’s checks’s (the guideline’s, the statute’s, the reconciliation’s) are’s the high’s TPR’s, the expert’s review’s is’s the moderate’s, and’s the outcome’s is’s the delayed’s and’s the noisy’s. The verifier’s design’s: the deterministic’s first’s, the expert’s second’s, the outcome’s last’s (the monitoring’s).

21.19 The regulatory overlay

21.19.1 The tiers

The high-stakes’s verticals’s are’s the high-risk’s under’s the EU’s AI’s Act’s (the Annex III’s: the essential’s services’s, the employment’s, the education’s, the health’s, the law’s enforcement’s, the migration’s, the justice’s). The obligations’s (the Module 20, §20.14.1’s): the risk’s management’s, the data’s governance’s, the technical’s documentation’s, the logging’s, the transparency’s, the human’s oversight’s, the accuracy’s and’s the robustness’s.

21.19.2 The human’s oversight’s requirement

The Article 14’s human’s oversight’s is’s the explicit’s: the human’s must’s be’s able’s to intervene’s and’s stop’s the system’s, and’s the oversight’s must’s be’s the designed’s and’s the documented’s. The high-stakes’s vertical’s is’s the case’s where’s the Article 14’s is’s the mandatory’s, not’s the optional’s, and’s the Module 20, §20.24’s oversight’s design’s is’s the compliance’s artefact’s.

21.19.3 The sectoral’s

The sectoral’s (the MDR’s, the FDA’s, the legal’s ethics’s, the financial’s conduct’s) apply’s on top, and’s they’re’s the stricter’s. The deployment’s must’s meet’s the highest’s (the Module 20, §20.17’s posture’s).

21.20 The evaluation’s difficulty

21.20.1 The difficulty

The high-stakes’s evaluation’s is’s the hardest’s:

• The base’s rate’s: the rare’s events’s (the adverse’s drug’s reaction’s, the fraud’s) are’s the low’s, and’s the verifier’s precision’s at’s the low’s base’s rate’s is’s the poor’s.
• The cost’s: the failure’s cost’s is’s the high’s (the patient’s harm’s, the legal’s liability’s), and’s the autonomy’s threshold’s is’s the strict’s.
• The expert’s availability’s: the expert’s is’s the bottleneck’s.
• The ground’s truth’s: the outcome’s is’s the delayed’s and’s the confounded’s.

21.20.2 The evaluation’s design

The evaluation’s design’s:

• The task’s set’s: the real’s cases’s (the retrospective’s), the expert’s labeled’s.
• The verifier’s: the deterministic’s + the expert’s (the composed’s, Module 7, §7.8.2’s), with’s the measured’s operating’s point’s.
• The metric’s: the TPR’s, the FPR’s, the precision’s at’s the deployment’s base’s rate’s, the cost’s per correct’s.
• The autonomy’s threshold’s: the strict’s (the high’s $C_f$‘s, the low’s $C_h$‘s relative’s).
• The monitoring’s: the outcome’s (the delayed’s), the expert’s sample’s.

21.20.3 The concept’s check

Why is the high-stakes domain’s evaluation’s the “verification-first” design’s, not the “capability-first’s”?

Answer. Because the high-stakes’s verifier’s is’s the weak’s (the expert’s, the outcome’s), and’s the capability’s (the agent’s accuracy’s) is’s the necessary’s, not’s the sufficient’s: the agent’s that’s is’s 95%‘s accurate’s on’s the expert’s benchmark’s may’s have’s the 5%‘s error’s that’s the expert’s misses’s (the false’s accepts’s), and’s in’s the high-stakes’s, the 5%‘s is’s the harm’s. The verification-first’s design’s specifies’s the verifier’s (the deterministic’s + the expert’s, the measured’s TPR’s/FPR’s) and’s the autonomy’s threshold’s (the strict’s) before’s the agent’s, so’s the agent’s design’s is’s the verifier’s-constrained’s (the human’s oversight’s, the conservative’s autonomy’s). The capability-first’s design’s builds’s the agent’s and’s then’s asks’s the verifier’s to’s catch’s it’s, which’s in’s the high-stakes’s is’s the backwards’s.

PART D — VERIFICATION-FIRST DESIGN

21.23 The principle

21.23.1 The statement

The verification-first’s principle. For a deployment’s in’s a new’s domain’s, specify’s the verifier’s — its kind’s, its operating’s point’s, its cost’s, and’s its coverage’s — before’s designing’s the agent’s architecture’s, the harness’s, or the training’s.

The principle’s the module’s synthesis: the GUI’s domain’s, the research’s domain’s, and’s the high-stakes’s vertical’s all’s illustrate’s that’s the verifier’s is’s the constraint’s (the GUI’s coverage’s, the research’s gap’s, the high-stakes’s expert’s). The agent’s design’s is’s the downstream’s of’s the verifier’s specification’s.

21.23.2 The rationale

The rationale’s:

• The verifier’s determines’s the autonomy’s (the Part 0, §0.5’s threshold’s), and’s the autonomy’s determines’s the agent’s architecture’s (the human’s oversight’s, the permission’s, the monitoring’s).
• The verifier’s determines’s the economics’s (the Module 17c’s), and’s the economics’s determines’s the deployment’s viability’s.
• The verifier’s determines’s the failure’s modes’s the deployment’s can’s detect’s, and’s the undetected’s modes’s are’s the residual’s risk’s.
• The verifier’s is’s the domain’s transfer’s’s hardest’s part’s, and’s the capability’s (the model’s) is’s the transferable’s (the general’s model’s, the same’s harness’s, the different’s verifier’s).

21.23.3 The failure’s of’s the capability-first’s

The capability-first’s design’s (the agent’s first’s, the verifier’s later’s) fails’s because’s:

• The agent’s is’s optimized’s for’s the benchmark’s, not’s the deployment’s verifier’s.
• The verifier’s is’s the afterthought’s, and’s its’s operating’s point’s is’s the unmeasured’s.
• The autonomy’s is’s the assumed’s (the “it’s works’s on’s the benchmark’s”), not’s the decided’s.
• The residual’s risk’s is’s the unknown’s, and’s the deployment’s discovers’s it’s in’s the incident’s.

21.24 The verifier’s specification’s checklist

21.24.1 The checklist

Procedure 21.24.1 (The verifier’s specification). For a deployment’s in’s a new’s domain’s:

1. The failure’s modes’s: enumerate’s the ways’s the agent’s output’s can’s be’s wrong’s (the wrong’s diagnosis’s, the fabricated’s citation’s, the missed’s step’s).
2. The verifier’s kind’s: for each’s failure’s mode’s, choose’s the verifier’s (the deterministic’s check’s, the test’s, the expert’s, the reproduction’s, the judge’s), with’s the kind’s’s V1–V5’s (Module 17b, §17b.1’s).
3. The operating’s point’s: the TPR’s, the FPR’s, the precision’s at’s the deployment’s base’s rate’s, with’s the confidence’s intervals’s (the Module 17b, §17b.6’s).
4. The cost’s: the per-check’s cost’s, the coverage’s, the throughput’s ($r_{\text{ver}}$‘s).
5. The autonomy’s threshold’s: the $(1-\mathrm{TPR})\pi C_f < C_h$‘s, the feasible’s set’s, the operating’s point’s (the Module 17c’s).
6. The residual’s risk’s: the failure’s modes’s the verifier’s misses’s (the coverage’s gap’s), and’s the expected’s loss’s.
7. The monitoring’s: the detection’s of’s the verifier’s misses’s (the outcome’s, the sample’s, the audit’s).
8. The revision’s condition’s: the triggers’s (the new’s failure’s mode’s, the distribution’s change’s, the verifier’s drift’s).

21.24.2 The checklist’s use

The checklist’s is’s the design’s gate’s: the agent’s design’s does’s not’s start’s until’s the checklist’s is’s complete’s. The gate’s the Module 12, §12.16’s gates’s applied’s to’s the domain’s: the verifier’s is’s the cheapest’s first’s check’s (the specification’s is’s the document’s, not’s the experiment’s), and’s the candidate’s that’s cannot’s specify’s the verifier’s is’s the no-go’s.

21.25 The autonomy’s decision

21.25.1 The decision’s structure

The autonomy’s decision’s is’s the Module 17c’s, with’s the domain’s parameters’s:

• The $\pi$‘s (the base’s rate’s) is’s the domain’s (the research’s low’s, the coding’s high’s).
• The $C_f$‘s is’s the domain’s (the high-stakes’s high’s).
• The $C_h$‘s is’s the expert’s time’s (the high-stakes’s expensive’s).
• The TPR’s/FPR’s are’s the domain’s verifier’s (the measured’s).

21.25.2 The decision’s

The decision’s:

• The research’s: the low’s $\pi$‘s (the claims’s are’s the mostly’s wrong’s) and’s the moderate’s $C_f$‘s, so’s the autonomy’s is’s the strict’s (the expert’s review’s is’s the default’s, and’s the agent’s is’s the drafting’s tool’s).
• The GUI’s: the moderate’s $\pi$‘s and’s the moderate’s $C_f$‘s, with’s the high’s TPR’s verifier’s (the final state’s script’s), so’s the autonomy’s is’s the possible’s for’s the well-verified’s tasks’s (the final state’s check’s is’s the strong’s).
• The high-stakes’s: the low’s $\pi$‘s and’s the high’s $C_f$‘s, so’s the autonomy’s is’s the near-zero’s (the expert’s review’s is’s the mandatory’s, and’s the agent’s is’s the assistive’s).

21.26 The worked example

21.26.1 The deployment

A radiology’s report’s drafting’s agent’s (the assistive’s, the radiologist’s reviews’s the draft’s).

21.26.2 The checklist

1. The failure’s modes’s: the missed’s finding’s (the lesion’s), the wrong’s finding’s (the false’s positive’s), the wrong’s location’s, the wrong’s recommendation’s.
2. The verifier’s kind’s: the the radiologist’s review’s (the expert’s, the V5’s-adjacent’s) for all’s the modes’s; the the deterministic’s check’s (the structured’s report’s completeness’s) for the format’s.
3. The operating’s point’s: the radiologist’s TPR’s on the missed’s findings’s (the measured’s in’s the audit’s, the Module 20, §20.23’s), the FPR’s (the false’s positives’s the radiologist’s flags’s).
4. The cost’s: the radiologist’s time’s per report’s, the $r_{\text{ver}}$‘s (the reports’s per hour’s).
5. The autonomy’s threshold’s: the $(1-\text{TPR})\pi C_f < C_h$, with’s the $C_f$‘s the missed’s diagnosis’s harm’s, and’s the $C_h$‘s the radiologist’s time’s. The high’s $C_f$‘s and’s the low’s $\pi$‘s (the findings’s are’s the rare’s) make’s the threshold’s the strict’s: the autonomy’s is’s the near-zero’s, and’s the agent’s is’s the draft’s (the radiologist’s reviews’s all’s).
6. The residual’s risk’s: the radiologist’s misses’s (the TPR’s < 1’s), and’s the monitoring’s (the outcome’s, the second’s read’s).
7. The monitoring’s: the audit’s sample’s, the second’s read’s, the incident’s reporting’s.
8. The revision’s condition’s: the new’s finding’s type’s, the radiologist’s TPR’s drift’s, the regulatory’s change’s.

21.26.3 The decision’s

The decision’s: the agent’s is’s the drafting’s assistive’s, the radiologist’s reviews’s all’s, and’s the autonomy’s is’s the none’s (the Article 14’s human’s oversight’s, the Module 20’s). The value’s is’s the radiologist’s time’s saved’s (the draft’s is’s faster’s than’s the blank’s report’s), not’s the autonomy’s. The lesson’s: the verification-first’s design’s reveals’s the right’s value’s proposition’s (the assistive’s, not’s the autonomous’s), which’s the capability-first’s would’s have’s missed’s.

21.27 The concept’s check

Why is the verification-first’s design’s the “right’s” value’s proposition’s, not’s just’s the risk’s control’s?

Answer. Because the verifier’s strength’s determines’s what’s the agent’s can’s be’s trusted’s with’s, and’s the trust’s is’s the value’s. The radiology’s agent’s with’s the expert’s verifier’s (the radiologist’s reviews’s all’s) is’s the drafting’s assistive’s (the radiologist’s time’s saved’s), and’s the same’s agent’s with’s the strong’s deterministic’s verifier’s (the final’s state’s script’s, the coding’s case’s) is’s the autonomous’s (the time’s saved’s, the scale’s). The verification-first’s design’s discovers’s which’s value’s proposition’s the domain’s supports’s, and’s the capability-first’s design’s builds’s the autonomous’s agent’s and’s discovers’s (in’s the deployment’s) that’s the verifier’s cannot’s support’s it’s. The value’s proposition’s is’s the verifier’s consequence’s, and’s the verification-first’s is’s the honest’s design’s.

PART E — THE LAB

21.28 The lab’s task specification

Select a domain from your own’s practice’s (a vertical’s, a workflow’s, a task’s family’s). Specify’s the verifier’s before’s designing’s the agent’s, using’s the Procedure 21.24.1’s checklist’s.

21.29 The deliverables

21.29.1 The specification’s document

The document’s, with’s the eight’s sections’s:

1. The domain’s and’s the task’s: the task’s family’s, the users’s, the data’s, the regulation’s.
2. The failure’s modes’s: the enumerated’s (the $\ge5$‘s), with’s the severity’s (the $C_f$‘s) each’s.
3. The verifier’s: the kind’s (the V1–V5’s, the composed’s) for each’s mode’s, the construction’s, the coverage’s.
4. The operating’s point’s: the TPR’s, the FPR’s, the precision’s at’s the base’s rate’s, with’s the intervals’s (the measured’s, if’s the data’s available’s; the estimated’s, if’s not’s).
5. The cost’s: the $C_v$‘s per check’s, the $r_{\text{ver}}$‘s, the $\pi$‘s, the $C_f$‘s, the $C_h$‘s.
6. The autonomy’s threshold’s: the computed’s, the feasible’s set’s, the recommended’s operating’s point’s.
7. The residual’s risk’s: the uncovered’s modes’s, the expected’s loss’s, the monitoring’s.
8. The revision’s condition’s: the triggers’s.

21.29.2 The design’s memo

The memo’s: the verification-first’s design’s implies’s which’s agent’s architecture’s (the autonomy’s level’s, the human’s oversight’s, the harness’s), and’s the value’s proposition’s (the assistive’s, the autonomous’s, the hybrid’s).

21.30 The acceptance criteria

21.31 The hypotheses

• H1. The domain’s verifier’s is’s the weak’s (the expert’s, the outcome’s), and’s the autonomy’s threshold’s is’s the strict’s (the autonomy’s is’s the limited’s).
• H2. The value’s proposition’s is’s the assistive’s (the expert’s time’s saved’s, the draft’s), not’s the autonomous’s.
• H3. The verifier’s coverage’s has’s the gap’s (the failure’s modes’s the verifier’s misses’s), and’s the gap’s is’s the residual’s risk’s.
• H4. The base’s rate’s is’s the low’s (the rare’s events’s), and’s the precision’s at’s the deployment’s base’s rate’s is’s the poor’s (the $\le0.5$‘s), which’s is’s the autonomy’s constraint’s.
• H5. The monitoring’s (the sample’s, the outcome’s) is’s the necessary’s, and’s the audit’s sample’s size’s is’s the Module 5’s power’s.

21.32 What good looks like

A good lab report has five properties.

The verifier comes first, in the document’s order. The specification’s document’s has’s the failure’s modes’s and’s the verifier’s before the agent’s architecture’s. A report that starts’s with’s “the agent’s uses’s the model’s X” has’s done’s the capability-first’s and’s missed’s the module’s point’s.

The precision is at the deployment’s base rate, not the calibration’s. The report’s headline’s is the precision’s at’s the $\pi$‘s, and’s the base-rate’s collapse’s is’s the finding’s. A report’s that’s says’s “the verifier’s is’s 95%‘s accurate’s” without’s the base-rate’s precision’s has’s the wrong’s metric’s.

The residual risk is the uncovered modes. The report’s lists’s the failure’s modes’s the verifier’s misses’s, and’s the residual’s risk’s is’s the expected’s loss’s of those’s modes’s, not’s the aggregate’s.

The value proposition is the verifier’s consequence. The report’s states’s whether’s the domain’s supports’s the assistive’s, the autonomous’s, or the hybrid’s value’s proposition’s, because of the verifier’s strength, not because of the model’s.

The revision condition is specific. The triggers’s (the new’s failure’s mode’s, the base’s rate’s shift’s, the expert’s TPR’s drift’s) are’s named’s, and’s the review’s cadence’s is’s stated’s.

21.33 What good’s looks’s like’s (the worked’s example’s)

For the radiology’s case’s, the good’s report’s has’s:

• The five’s failure’s modes’s, the expert’s verifier’s (the radiologist’s) as’s the primary’s, the deterministic’s format’s check’s as’s the secondary’s.
• The radiologist’s TPR’s estimated’s from’s the literature’s or’s the pilot’s, with’s the interval’s.
• The precision’s at’s the low’s base’s rate’s (the finding’s prevalence’s), showing’s the collapse’s.
• The autonomy’s threshold’s: the strict’s, the autonomous’s infeasible’s.
• The residual’s risk’s: the radiologist’s misses’s, the second’s read’s as’s the monitoring’s.
• The value’s proposition’s: the assistive’s (the draft’s), the radiologist’s time’s saved’s.
• The revision’s condition’s: the new’s finding’s type’s, the TPR’s drift’s, the regulatory’s change’s.

MODULE 21 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E21.1. A GUI agent’s per-step’s grounding’s accuracy’s is 0.92 on the element’s, 0.98 on the action’s, and 0.95 on the state’s. Compute the per-step’s and the 15-step’s task’s success’s.

S21.1. The per-step’s $=0.92\times0.98\times0.95=0.857$. The 15-step’s $=0.857^{15}=0.094$. The task’s success’s is 9.4%, which is the compounding’s. The lesson: the GUI’s per-step’s accuracy’s must be very high for the long-horizon’s tasks, and the element’s 0.92 is the dominant’s error’s. The fix’s: the element’s reference’s (the accessibility’s ID, the discrete’s) instead of the pixels’s.

E21.2. A browsing task’s has $p=0.1$ per query, $V=1.0$, and $c_{\text{query}}=0.02$. Compute the optimal number of queries.

S21.2. $n^*=\log(0.02/(0.1\times1.0))/\log(0.9)=\log(0.2)/\log(0.9)=(-1.609)/(-0.105)=15.3$. The optimum’s is $\approx15$ queries. The lesson: the harder’s the task’s (the $p=0.1$), the more’s the persistence’s is’s the optimal’s, and the benchmark’s rewards’s it’s. The deployment’s needs’s the stopping’s rule’s (the calibrated’s), not’s the infinite’s persistence’s.

E21.3. A research verification’s has $r_{\text{out}}=50$ claims per week and $r_{\text{ver}}=20$ claims per week. Compute the gap and state the consequence.

S21.3. The gap’s $=50-20=30$ claims per week. The backlog’s grows’s by 30 per week, and the verification’s is’s the bottleneck’s. The consequence’s: the unverified’s claims’s accumulate’s, and’s the agent’s (the RL’s) learns’s to produce’s the easy-to-verify’s claims’s (the specification’s gaming’s, §21.15). The fix’s: the $r_{\text{ver}}$‘s increase’s (the automated’s reproduction’s, the cheaper’s citation’s check’s) or the $r_{\text{out}}$‘s decrease’s (the agent’s scope’s). The lesson: the gap’s is’s the deployment’s constraint, and the verifier’s throughput’s is’s the design’s variable.

E21.4. A verifier’s has TPR = 0.99, FPR = 0.01, and the base rate is 0.05. Compute the precision, and state whether it is useful for a high-stakes deployment.

S21.4. The precision = $0.99\times0.05/(0.99\times0.05+0.01\times0.95)=0.0495/(0.0495+0.0095)=0.0495/0.059=0.839$. The precision’s is 0.84, which means 16% of the accepted outputs are wrong. For a high-stakes deployment with a high $C_f$, the 0.16’s false accept’s rate’s is too high: the autonomy’s threshold’s $(1-0.99)\times0.05\times C_f=0.0005C_f$ must be below the $C_h$, and for a high $C_f$, it is not. The lesson: even a 0.99/0.01 verifier is not enough at a 5% base rate for the high stakes, and the verifier’s TPR must be higher or the base rate higher or the $C_f$ lower.

E21.5. A radiology agent’s drafts reports. The radiologist’s TPR on the missed findings is 0.85, the base rate of findings is 0.10, the missed diagnosis cost is 1,000, and the radiologist’s review cost is 50. Is the autonomous release rational?

S21.5. The threshold = $(1-0.85)\times0.10\times1,000=0.15\times100=15$. The review cost is 50. Since 15 < 50, the threshold condition is satisfied — wait, that would suggest autonomy is rational. But the threshold is the expected missed-failure cost per attempt, and the comparison is to the review cost. If the missed-failure cost is 15 and the review is 50, the autonomy is cheaper. But the clinical reality: the 0.15 × 0.10 = 1.5% of the reports have a missed finding that the radiologist did not catch, and the cost of a missed diagnosis is catastrophic (not 1,000 but potentially a life). The lesson: the arithmetic says the autonomy is rational at the assumed $C_f$, and the $C_f$‘s assumption is the issue: the missed diagnosis’s cost is not the 1,000; it’s the harm plus the liability plus the trust. The verification-first design must challenge the $C_f$‘s assumption, and the clinical deployment’s $C_f$ is effectively infinite for the missed diagnosis, which makes the threshold violated. The lesson: the $C_f$‘s estimate is the decision’s most sensitive and most uncertain input in the high-stakes domain.

E21.6. A verifier’s coverage is 0.70 (it catches 70% of the failure modes). What is the residual risk’s structure, and what would you do?

S21.6. The residual is the 30% of the failure modes the verifier does not model. The structure: (1) enumerate the modes, and classify them as covered/uncovered; (2) for the uncovered, estimate the likelihood and the impact; (3) the residual’s expected loss is the sum. The action: (1) extend the verifier’s coverage (the new tests, the new properties); (2) add the monitoring (the sample’s audit, the outcome’s); (3) accept the residual if the expected loss is below the tolerance. The lesson: the coverage’s 0.70 is the TPR’s upper bound on the failure’s distribution, and the 30% is the risk’s, not the verifier’s footnote.

E21.7. A research agent’s paper cites 50 references. The citation’s check’s finds 48 exist and 2 do not. What are the failure’s modes, and what is the verification’s next step?

S21.7. The modes: (1) the fabricated citation (the 2 non-existent); (2) the misquotation (the 48 exist, but the claim’s support is not checked by the existence’s check). The next step: the support’s check (the claim’s is in the source’s, not just the source’s existence’s), which is the expert’s or the retrieval’s check. The lesson: the existence’s check’s TPR is low for the research’s claims (the existence’s is the necessary, not the sufficient), and the support’s check’s is the next option. The 2 fabricated’s of the 50 is the 4% fabrication’s rate, and the support’s misquotation’s rate is likely higher.

E21.8. A deployment’s value proposition is the “autonomous” but the verifier’s TPR is 0.80 and the base rate is 0.6, the $C_f/C_h$ is 10. Is the autonomous’s value proposition supported?

S21.8. The threshold = $(1-0.80)\times0.6\times10=0.2\times6=1.2>1$. The autonomy’s is not rational: the expected missed-failure cost is 1.2× the review cost. The autonomous’s value proposition’s is not supported by the verifier’s TPR; the deployment’s must either (1) improve the verifier’s TPR (to 0.90: $0.1\times6=0.6<1$, rational), (2) reduce the $C_f$, or (3) adopt the assistive’s value proposition (the expert’s review’s, the agent’s draft’s). The lesson: the value proposition’s is the verifier’s consequence, and the autonomous’s claim’s without’s the threshold’s is the capability-first’s error.

EXIT TEST FOR MODULE 21

You are ready for Module 22 when you can, without notes:

1. Define a GUI agent and state the action’s and observation’s forms.
2. State the grounding’s function and the three error sources.
3. Compute the per-step’s and the task’s success’s from the grounding’s accuracy’s.
4. Characterise OSWorld and WebArena by their triples and state what each does not measure.
5. Derive the persistence’s bound and the optimum, and explain the browsing benchmark’s reward.
6. State the GUI agent’s four failure modes and the verification’s difficulty.
7. Describe the research agent’s anatomy and the verifier’s per stage.
8. State the verification gap and the survey’s findings, with their status (P).
9. Name the four verification options and rank them by the cost per correct verification.
10. State the research agent’s risks and defences.
11. State the five transfer dimensions and the high-stakes profile.
12. Describe the expert’s three roles and the expert’s bottleneck.
13. State the high-stakes verifiers (the medical, the legal, the financial) and the Article 14’s oversight.
14. State the verification-first principle and the capability-first failure.
15. Apply the verifier’s specification’s checklist to a domain.
16. Compute the autonomy’s threshold for a domain and state the value proposition (the assistive, the autonomous, the hybrid).

End of Module 21. Say “next” to receive Module 22 in full.