Weeks 16–17. Prerequisite: Modules 1–10 and Part 0, §§0.8, 0.9.

11.0 Module map and objectives

Modules 1–10 built, constrained, and optimised the harness. This module addresses the two questions that determine whether the harness is operable and viable: how do you know what happened (observability), and what does it cost (economics and latency). The two are linked: a harness you cannot observe cannot be cost-optimised, because you cannot attribute the cost to a component; and a harness that is too expensive or too slow is not deployable, regardless of its success rate.

The module has five parts and two labs.

By the end of the module you will be able to do eight things:

1. Specify a trace schema sufficient for the three queries (failure clustering, regression attribution, cost per success), and justify every field.
2. Instrument a harness to emit the schema, and demonstrate the three queries on real traces.
3. State the OpenTelemetry GenAI conventions and their status, and decide which to adopt.
4. Cluster failures from a large run set and identify the largest cluster as the priority.
5. Apply the three-term cost invariant to diagnose a cost regression, and compute a Pareto frontier of accuracy vs cost.
6. Derive the routing and cascade conditions, and choose a routing policy for a deployment.
7. Compute the optimal verification spend for a given task distribution and verifier ROC.
8. Structure a postmortem for a non-deterministic failure, using the trace as the evidence, and distinguish the four failure causes.

Everything is self-contained. Where Part 0 or a prior module is used, the result is restated.

PART A — TRACE SCHEMAS

11.1 The three queries

11.1.1 Why a trace schema is a design decision

A trace is not a log. A log is a stream of messages; a trace is a structured record of a run, with the fields that make the run analysable. The fields are a design decision, and the decision determines what questions the trace can answer.

The curriculum’s position (Module 10, §10.5) is that the trace must support the three AHE observability pillars: component observability, experience observability, and decision observability. This module makes that operational by specifying the three queries the trace must support, and deriving the minimal field set from them.

11.1.2 The three queries

Query 1 — Failure clustering by tool-call pattern. Given a set of failed runs, cluster them by the sequence of tool calls, and identify the most common failure pattern. What the trace must contain: the tool calls (name, arguments, result status), the run’s outcome, and the ordering.

Query 2 — Regression attribution to a harness edit. Given a performance regression after a harness edit, identify which runs changed and which component caused the change. What the trace must contain: the harness version, the component versions, the per-step decisions, and a stable identifier that links the trace to the edit.

Query 3 — Cost per successful task. Given a set of runs, compute the total cost (tokens, tool calls, verifier calls) and the success rate, and divide. What the trace must contain: the token counts (input, output, cached), the tool calls with their durations, the verifier calls, and the outcome.

The three queries are not exhaustive; they are the minimum that the curriculum requires, and they are chosen because each corresponds to a Module 10/11 function: Query 1 is the failure-trace mining (Module 10, §10.10); Query 2 is the regression attribution (Module 10, §10.19); Query 3 is the cost model (Part 0, §0.9).

11.1.3 The minimal field set

From the three queries, the minimal field set is derived by asking, for each field, which query would fail without it.

Definition 11.1.1 (Minimal trace schema). A trace record for one step of one run contains:

Identification.

• run_id: a unique identifier for the run.
• step_id: a monotonic step index.
• trace_id: a unique identifier for the whole trace (the run).
• span_id / parent_span_id: the OpenTelemetry-style span identifiers, so the step can be nested in a tool call or a subagent (Module 8).
• timestamp: monotonic wall-clock, for ordering and latency.

Harness state.

• harness_version: the git SHA or a semantic version, so the trace is attributable to an edit (Query 2).
• component_versions: the versions of the tool schemas, the context manager’s policy, the policy engine’s rules (Query 2).
• model: the model identifier, the version, and the sampling parameters (temperature, top-p, seed) (Query 2, Query 3).

Context.

• context_tokens: the assembled context’s token count.
• context_dropped: what the context manager removed, and why (Module 3, §3.4).
• cache_hit_tokens: the cached prefix tokens (Query 3, §11.4).

Model call.

• input_tokens, output_tokens, reasoning_tokens (if separate).
• model_latency: the time from the request to the response.
• output: the raw model output (or a reference to it, for storage).

Tool calls.

• tool_name, tool_arguments (or a hash), tool_call_id.
• tool_status: ok, error, timeout, denied (Module 1, §1.2.2).
• tool_latency: the time from the dispatch to the result.
• tool_result_tokens: the result’s token count (for the cost model).

Decision (Module 10, pillar 3).

• reasoning: the model’s reasoning trace (ReAct thought, Module 4), or a reference to it.
• alternatives: the tools the model considered but did not call (if the model emits them).

Outcome.

• step_outcome: continue, terminate, error.
• run_outcome: success, failure, timeout, stalled, budget_exhausted.
• verifier_verdict: the verifier’s result (if the verifier ran at this step).

Cost (Query 3).

• tokens_in, tokens_out, tokens_cached, tokens_uncached.
• tool_calls_count, verifier_calls_count.
• cost_estimate: the computed cost of the step (from the three-term invariant, §11.14).

The field set is minimal in the sense that every field is needed by at least one query. A field that no query needs is a candidate for removal; a query that needs a field not in the set is a query the schema does not support.

11.1.4 The derivation

The derivation is worth stating as a discipline, because it is the schema’s justification.

A schema that omits component_versions cannot support Q2 (the trace cannot be attributed to a component). A schema that omits tokens_cached cannot support Q3 (the cost is wrong, because the cached tokens are priced differently, Module 3, §3.10). The derivation is the schema’s sufficiency argument, and it is the check for the schema’s completeness.

11.2 The OpenTelemetry GenAI conventions

11.2.1 What they are

OpenTelemetry (OTel) is the industry standard for traces, metrics, and logs. It defines a data model (spans, attributes, events) and a set of semantic conventions for common operations. The GenAI semantic conventions (tier B: vendor and standards publication; verify against the live specification) define the span names and attributes for generative-AI operations: model calls, token counts, tool calls, and the model’s parameters.

The conventions’ value is interoperability: a trace emitted in the OTel GenAI format can be read by any OTel-compatible backend (Jaeger, Tempo, a commercial APM), and the analysis tooling is shared. A bespoke trace format is readable only by the bespoke tooling.

11.2.2 What they standardise and what they do not

Standardised.

• Span names: gen_ai.chat, gen_ai.completion, gen_ai.tool.call, etc.
• Attributes: gen_ai.system (the provider), gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, gen_ai.tool.name.
• The span hierarchy: a model call is a span; a tool call is a child span; the run is the root span.

Not standardised.

• The harness-specific fields: the context manager’s decisions, the policy engine’s verdicts, the reasoning trace, the component versions. These are the fields that make the trace harness-observable, and they are not part of the GenAI conventions.
• The failure taxonomy (Module 18’s MAST). The taxonomy is the analysis layer, not the trace layer.

The distinction is the design implication: adopt the OTel GenAI conventions for the standard fields (interoperability), and extend them with the harness-specific fields (the three queries’ requirements) as custom attributes. The extension is the harness’s schema; the standard is the substrate.

11.2.3 The status and the caveat

The GenAI conventions are tier B and evolving: the specification is not yet stable (as of the curriculum’s date), and the attribute names may change. The curriculum’s posture is: adopt the current conventions for the standard fields, record the convention version in the trace (otel_genai_version), and be prepared to migrate. The migration is the cost of the interoperability, and the cost is worth it for a production harness.

11.3 The trace hierarchy

11.3.1 The three levels

The trace has three levels, corresponding to the run’s structure.

Level 1 — The run. The root span. It records the task, the harness version, the model, the start and end times, the outcome, and the total cost. One per run.

Level 2 — The step. A child of the run. It records the model call (the input context, the output), the tool calls (the action), and the step’s outcome. One per loop iteration.

Level 3 — The sub-operation. A child of a step or a tool call. It records the sub-operations: a subagent’s run (Module 8, §8.14), a tool’s internal call (an MCP server’s call, a database query), a verifier’s execution. One per sub-operation.

The hierarchy is the span tree, and it is the OTel model: a span has a span_id and a parent_span_id, and the tree is reconstructible from the fields. The hierarchy is what makes the drill-down (§11.7) possible: a run’s failure is attributable to a step, and a step’s failure is attributable to a sub-operation, by following the tree.

11.3.2 The subagent’s trace

A subagent (Module 8, §8.14) is a separate run, with its own run_id and its own trace. The parent’s trace records the subagent’s spawn and result (a span with the subagent’s run_id as an attribute), and the subagent’s trace is linked by the run_id. The link is the cross-trace reference, and it is what makes the multi-agent trace analysable: the orchestrator’s failure is attributable to a subagent, and the subagent’s failure is analysed in its own trace.

11.3.3 The trace’s completeness

A trace is complete if every step, every tool call, and every sub-operation is recorded. The completeness is a design requirement, not an aspiration: a trace with gaps is a trace whose analysis is biased (the missing steps are not random; they are the ones the harness did not instrument, which are often the failure paths).

The check: a run’s trace should have a record for every model call and every tool call, and the record’s count should match the harness’s own counters. A mismatch is a completeness bug, and it should be a test in the harness’s CI (Module 9, §9.9).

11.4 The fields for the three queries, revisited

11.4.1 Query 1 — Failure clustering

The clustering query needs the tool-call sequence and the outcome. The sequence is reconstructible from the step_id, tool_name, and tool_arguments fields, ordered by step_id. The outcome is the run_outcome. The clustering is a sequence similarity problem: the runs are clustered by the similarity of their tool-call sequences, and the largest cluster is the priority (Module 10, §10.10’s failure-trace mining).

The clustering method (the lab uses it):

1. Extract the sequence. For each failed run, the ordered list of (tool_name, argument_hash, status).
2. Compute the similarity. The pairwise similarity between sequences (the longest common subsequence, the Jaccard similarity of the call sets, or the edit distance).
3. Cluster. Agglomerative clustering (the threshold is the similarity), or DBSCAN (the density-based clustering), or the simple prefix-tree clustering (the shared prefix of the sequences).
4. Rank. The clusters by size (the number of runs), and the largest cluster is the priority.

The prefix-tree clustering is the simplest and often the most informative: it builds a trie of the tool-call sequences, and the largest subtree is the common failure pattern. The pattern is the failure signature, and it is the input to the harness edit.

11.4.2 Query 2 — Regression attribution

The attribution query needs the version fields and the decision fields. The version fields (harness_version, component_versions, model) link the trace to the edit; the decision fields (reasoning, context_dropped, step_outcome) show the mechanism.

The attribution method:

1. Split the traces by the harness version (before and after the edit).
2. Compare the outcomes (the success rate, the cost, the latency) between the splits, with the Module 5 discipline (paired bootstrap, confidence intervals).
3. Drill down to the steps where the splits differ: the component whose version changed, the decision that changed, the failure that appeared.
4. Attribute the regression to the component, with the evidence (the step’s trace) as the support.

The attribution is not a statistical proof (the split is observational, not randomised); it is a diagnosis that identifies the likely cause, and the confirmation is a controlled comparison (revert the edit, re-measure). The trace is the diagnostic tool; the controlled comparison is the proof.

11.4.3 Query 3 — Cost per success

The cost query needs the token fields, the latency fields, and the outcome. The cost is the three-term invariant (Part 0, §0.9):

with the cached tokens priced separately (Module 3, §3.10).

The cost per success is the total cost divided by the number of successes:

The computation is straightforward; the diagnostic is the decomposition. A cost regression can be due to the tokens, the tools, or the verifiers, and the three-term invariant’s terms are the diagnostic (Module 3, §3.12’s worked example). A cost regression that is not decomposed is a regression that cannot be fixed.

11.5 Failure taxonomies

11.5.1 The MAST taxonomy (preview)

Module 18 develops the MAST failure taxonomy (Cemri et al., arXiv:2503.13657, tier P). The preview is its structure, because the trace schema must support it.

MAST classifies multi-agent failures into three categories and fourteen failure modes:

• System design failures: the architecture, the role definitions, the coordination protocol.
• Inter-agent misalignment: the communication, the shared state, the conflict resolution.
• Task verification: the verifier’s coverage, the success criteria, the reward hacking.

The taxonomy is the analysis layer; the trace is the data layer. A trace that supports MAST must record the role of each agent, the messages between them, the shared state’s changes, and the verifier’s verdicts. The minimal field set of §11.1.3 supports the single-agent case; the multi-agent case adds the inter-agent fields (Module 18, §18.4).

11.5.2 The single-agent taxonomy

For the single-agent harness, the failure modes are the ones the curriculum has built up:

The taxonomy is the clustering’s label set: the clusters are labelled by the mode, and the label determines the remedy (Module 10, §10.10’s failure-trace mining). The trace’s fields are what make the labelling possible: a trace without context_dropped cannot be labelled “context loss”; a trace without tool_status cannot be labelled “tool failure”.

11.5.3 The taxonomy’s use

The taxonomy has three uses.

Use 1 — The clustering’s labels. The clusters are labelled by the mode, and the label determines the priority: a large cluster of “context loss” is a different priority from a large cluster of “model error”, because the former is the harness’s fix and the latter is not.

Use 2 — The harness’s coverage check. The taxonomy’s modes are the harness’s modelled failure set (Module 9, §9.7): a harness that models only “tool failure” and “loop/stall” has a coverage gap for “context loss” and “verification failure”. The gap is the harness’s blind spot, and the taxonomy is the check.

Use 3 — The evaluation’s strata. The evaluation (Module 5) can be stratified by the failure mode: the success rate on the “context loss” tasks is a different measure from the success rate on the “model error” tasks, and the two are more informative than the aggregate. The stratification is the trace’s value for the evaluation.

11.6 Storage and retention

11.6.1 The storage problem

A trace is large. A 200-step run with full context and reasoning is megabytes; a thousand runs a day is gigabytes; a year is terabytes. The storage cost is real, and the retention policy is a design decision.

11.6.2 The retention tiers

Three tiers, each with a different retention and cost.

Tier 1 — Hot (the full trace). The full trace, with the context and reasoning, retained for days to weeks. Used for the debugging and the failure analysis (the drill-down). Cost: high (the full data).

Tier 2 — Warm (the summary trace). The trace’s summary: the step sequence, the tool calls, the outcomes, the costs, without the full context and reasoning. Retained for months. Used for the aggregate analysis (the cost, the success rate, the trends). Cost: moderate (the summary is small).

Tier 3 — Cold (the aggregate). The aggregate metrics: the daily success rate, the cost, the failure-mode distribution. Retained for years. Used for the long-term trends and the governance (Module 20). Cost: low.

The tiers are the retention policy: the hot tier for the debugging, the warm for the analysis, the cold for the governance. The policy is a design decision, and the decision depends on the deployment’s needs: a high-stakes deployment needs a longer hot retention (the incident analysis); a low-stakes deployment can afford a shorter one.

11.6.3 The privacy and the security

The traces contain the model’s inputs and outputs, which may contain sensitive data (the user’s content, the credentials, the personal information). The storage must be encrypted at rest, the access audited (Module 20), and the retention bounded (the data minimisation). The trace’s privacy is a governance requirement, not a convenience, and the retention policy must be justified by the requirement.

11.7 Drill-down

11.7.1 The mechanism

Drill-down is the navigation from an aggregate to a specific trace: the success rate drops, the analyst drills into the failed runs, then into the failing step, then into the sub-operation. The drill-down is what makes the aggregate actionable: the aggregate says something changed; the drill-down says what.

11.7.2 The prerequisites

The drill-down requires three things.

The hierarchy (§11.3), so the analyst can navigate from the run to the step to the sub-operation.

The stable identifiers (run_id, step_id, trace_id), so the analyst can retrieve the specific record.

The indexed fields (the harness_version, the model, the tool_name, the run_outcome), so the analyst can filter the traces by the dimension of interest.

The three prerequisites are the schema’s design requirements: a schema without the hierarchy cannot drill down; a schema without the identifiers cannot retrieve; a schema without the indexed fields cannot filter. The schema’s sufficiency (§11.1.4) is the drill-down’s possibility.

11.7.3 The worked drill-down

The aggregate. The success rate on the coding task drops from 0.72 to 0.61 after a harness edit (the context manager’s compaction policy changed).

The drill-down, step 1. Filter the traces by harness_version: the before-edit (0.72) and after-edit (0.61). The difference is 11 points.

The drill-down, step 2. Filter the failed runs by run_outcome and step_outcome: the after-edit failures are concentrated in the runs that hit the context-trimming threshold (the context_dropped field is non-empty).

The drill-down, step 3. Inspect the failing steps: the context_dropped shows that the new policy dropped the file contents read early in the run, and the reasoning shows the model acting on incomplete information.

The attribution. The compaction policy is the cause. The confirmation: revert the policy, re-measure (the controlled comparison, §11.4.2). The trace is the diagnostic; the revert is the proof.

The worked drill-down is the pattern, and it is the reason the schema’s fields are what they are: every field in §11.1.3 is used in the drill-down, and a missing field would have stopped the navigation at some step.

PART B — FAILURE TAXONOMIES AND ERROR ANALYSIS AT SCALE

11.8 Error analysis at scale

11.8.1 The problem

At small scale, the failure analysis is manual: the analyst reads the failed traces and classifies them. At large scale (thousands of runs), the manual analysis is infeasible, and the analysis must be automated or sampled.

11.8.2 The two approaches

Automated classification. A model (or a rule-based classifier) labels each failed trace with a taxonomy mode. The classifier is a verifier (Module 7, §7.8), and its reliability is the calibration (Module 7, §7.8.3). The automated classification’s advantage is the scale; its disadvantage is the classifier’s error, which must be measured (the confusion matrix, the TPR/FPR per mode).

Stratified sampling. The traces are sampled stratified by the dimensions of interest (the task type, the model, the harness version), and the analyst reads the sample. The sampling’s advantage is the accuracy; its disadvantage is the coverage (the rare modes are under-sampled). The sampling’s size is the Module 5 power calculation, applied to the estimation of the mode distribution.

The two approaches are complementary: the automated classification is the screen (the modes’ rough distribution), and the stratified sampling is the audit (the classifier’s validation). The audit is the discipline: an automated classifier that is not audited is a classifier of unknown reliability, and its labels are the analysis’s foundation.

11.8.3 The error-analysis workflow

The workflow, developed from Module 10’s failure-trace mining:

1. Collect the failed traces. The runs with run_outcome = failure (and the timeout/stall/budget cases).
2. Cluster. The tool-call sequence clustering (§11.4.1), or the reasoning-based clustering (the embedding similarity of the reasoning traces).
3. Label. The clusters are labelled with the taxonomy mode (§11.5.2), by the automated classifier or the analyst.
4. Rank. The clusters by size, and the largest is the priority.
5. Diagnose. The largest cluster is drilled down to the mechanism (§11.7), and the mechanism is the harness’s fix.
6. Fix. The harness edit is proposed, evaluated, and accepted (Module 10’s loop).
7. Re-measure. The failure-mode distribution is re-measured after the fix, and the shift is the fix’s effect.

The workflow is the error-analysis loop, and it is the Module 10 loop applied to the failures: the clusters are the observations, the fix is the proposal, the re-measurement is the evaluation.

11.9 The failure-mode distribution

11.9.1 The distribution as a diagnostic

The failure-mode distribution — the fraction of failures in each mode — is the harness’s diagnostic, and it is more informative than the aggregate failure rate.

Worked example. Two harnesses have the same success rate (0.70), but harness A’s failures are 60% “context loss” and 40% “model error”, while harness B’s are 60% “model error” and 40% “tool failure”. Harness A’s failures are mostly the harness’s fix (the context manager); harness B’s are mostly the model’s (which the harness cannot fix) and the tool’s (which the tool team can). The distribution tells you where the effort should go, which the aggregate does not.

11.9.2 The distribution’s shift

The distribution’s shift over time is the harness’s progress measure. A harness that fixes the context loss should see the “context loss” fraction fall, and the other fractions rise (as the total failure rate falls). The shift is the evidence that the fix worked, and it is the leading indicator relative to the aggregate success rate (which is noisy and slow).

The shift’s measurement is the same as the success rate’s: the Module 5 discipline, applied to the mode fractions. The comparison is the before/after distribution, with the confidence intervals (the multinomial’s, or the per-mode binomial’s).

11.10 The failure-mode taxonomy’s limits

11.10.1 The taxonomy is not exhaustive

The taxonomy is a useful set of modes, not a complete partition. There are failures that do not fit any mode (a novel failure, a combination of modes), and the taxonomy’s “other” category is the residual. The residual’s size is the taxonomy’s coverage measure: a large “other” is a taxonomy that is missing modes, and the missing modes are the research.

11.10.2 The modes are not independent

A failure can be a combination: a context loss that causes a tool failure that causes a loop. The taxonomy’s modes are not mutually exclusive, and the classification must choose the primary mode (the one the fix addresses) or the root mode (the one that caused the others). The choice is a design decision, and it should be stated (the classification’s rule).

11.10.3 The classification is model-relative

The “model error” mode is a residual: it is the failure the harness cannot fix. As the model improves, the mode’s fraction should fall (Module 6’s compensating components), and the mode’s meaning changes: the “model error” of one model is the “model weakness” that the next model fixes. The taxonomy’s modes are stable; the distribution’s interpretation is model-relative.

11.11 The trace and the evaluation

11.11.1 The trace’s role in the evaluation

The trace is the evaluation’s evidence. The Module 5 evaluation produces the aggregate (the success rate, the confidence interval); the trace produces the why (the failure modes, the drill-down). The two are complementary: the aggregate says whether the harness improved; the trace says how.

11.11.2 The trace’s role in the regression

The trace is the regression’s diagnostic (§11.4.2). A regression is detected by the aggregate (the Module 5 comparison) and attributed by the trace (the drill-down). The attribution is the diagnosis, and the controlled comparison (the revert) is the proof.

11.11.3 The trace’s role in the loop

The trace is the loop’s input (Module 10, §10.6): the observations are the traces, the proposals are the failure clusters, and the evaluation is the trace-based measurement. The loop’s quality is the trace’s quality: a loop with a poor trace is a loop with a poor observation, and its proposals are the guesses.

11.12 The observability’s cost

11.12.1 The cost of the trace

The trace is not free. The emission’s cost is the serialisation and the storage; the context’s cost is the trace’s fields in the context (if the trace is shown to the model); the analysis’s cost is the analyst’s time.

The cost is a design trade-off: a more complete trace is more expensive, and the completeness is justified by the queries it supports. The minimal field set (§11.1.3) is the minimum that supports the three queries; a trace that omits a field is cheaper and less useful, and the trade-off is the deployment’s decision.

11.12.2 The cost’s measurement

The cost is measured in the same way as the harness’s cost (the three-term invariant): the trace’s tokens (if in the context), the storage’s bytes, the analysis’s time. The trace’s cost is part of the harness’s cost per task, and it should be included in the cost model (§11.14).

PART C — COST MODELS

11.13 The three-term invariant

11.13.1 Restating the invariant

Part 0, §0.9 gives the three-term cost invariant:

The invariant is the cost model’s decomposition, and it is the diagnostic’s basis: a cost regression is attributable to one of the three terms, and the term identifies the fix.

11.13.2 The four sub-terms of the token term

The token term is itself decomposed, by the cache’s effect:

where $\Delta_{\text{cache}}$ is the saving from the cached prefix (Module 3, §3.10, Proposition 0.8.4):

The cache’s saving is a function of the prefix’s stability (the hit rate $h$) and the prefix’s length ($L_p$). The cache-aware layout (Module 3, §3.11) is the lever, and the saving is the measurement.

11.13.3 The tool term’s decomposition

The tool term is decomposed by the tool’s latency and the retry’s effect:

where $n_t$ is the number of calls to tool $t$, and $\bar c_t$ is the tool’s cost (the latency’s cost, the external service’s fee). The retries add to $n_t$: a tool with a 10% failure rate and 3 retries has an expected 1.11 calls per success, and the retry’s cost is the 11% premium (Module 1, §1.2.5, Proposition 7.19.2’s context).

11.13.4 The verifier term’s decomposition

The verifier term is decomposed by the verifier’s placement (Module 7, §7.11.2):

The pre-execution verifier (the policy engine) is cheap (a function); the post-execution (the test suite) is moderate; the post-release (the CI, the monitor) is the deployment’s cost. The terms are the three placements, and the allocation between them is the verification budget (§11.20).

11.14 The cost regression’s diagnosis

11.14.1 The diagnosis procedure

Procedure 11.14.1 (Cost regression diagnosis).

1. Measure the cost’s change. The three terms’ change between the before and after (§11.13.1).
2. Attribute the change. The term with the largest change is the primary cause.
3. Decompose the term. The token term by the cache; the tool term by the tool; the verifier term by the placement.
4. Identify the mechanism. The decomposition’s largest sub-term is the mechanism.
5. Fix and re-measure. The fix addresses the mechanism, and the re-measurement is the confirmation.

The procedure is the cost’s analogue of the failure’s diagnosis (§11.7.3): the aggregate says whether, the decomposition says where, and the fix is the what.

11.14.2 The worked diagnosis

The observation. The cost per task rises from 0.12 to 0.15 after a harness edit (a 25% increase).

The decomposition. The token term rises from 0.06 to 0.065 (a 8% increase); the tool term from 0.04 to 0.065 (a 63% increase); the verifier term is unchanged (0.02).

The attribution. The tool term is the primary cause (a 0.025 increase, out of 0.03 total).

The tool’s decomposition. The run_tests calls rise from 4 to 7 per task (a 75% increase), and the other tools are unchanged.

The mechanism. The edit (a compaction policy change) caused the agent to re-run the tests more often, because the context loss made it unsure whether the previous test run was current.

The fix. The compaction policy is reverted (or adjusted to preserve the test’s result), and the tool calls fall back to 4 per task.

The lesson. The token term’s increase was small; the tool term’s increase dominated, and the tool term is often the dominant cost in agent workloads (Module 3, §3.12’s worked example). The diagnosis’s value is the attribution: without the decomposition, the fix would have targeted the tokens (the visible cost), and the regression would have persisted.

11.15 Latency engineering

11.15.1 The latency’s sources

An agent run’s latency has four sources.

Model latency. The time for the model’s call. It is the generation’s token count times the per-token latency, plus the prefill’s time.

Tool latency. The time for the tool’s execution. It is the tool’s service time, plus the queueing (if the tool is shared).

Queueing latency. The time a request waits for a resource (the model’s server, the tool’s service). It is Kingman’s formula (Part 0, §0.8): $\mathbb{E}[W_q]\approx\frac{\rho}{1-\rho}\frac{c_a^2+c_s^2}{2}\mathbb{E}[S]$.

Orchestration latency. The time the harness spends (the context assembly, the parsing, the dispatch). It is usually small, but it can be significant if the harness is inefficient.

11.15.2 The latency’s model

The run’s latency is the sum of the sources, with the parallelism’s effect:

The sequential steps’ latencies add; the parallel steps’ latencies overlap (the max, not the sum); the queueing is the shared resource’s delay. The three components are the levers: the sequential’s sum is reduced by fewer steps (a better context, a better model); the parallel’s max is reduced by more parallelism; the queueing is reduced by more capacity (the utilisation $\rho$) or a better service-time distribution.

11.15.3 The p95’s problem

The p95 latency (the 95th percentile) is the deployment’s relevant metric, because the slow runs are the ones the users notice. The p95 is dominated by the tail, which is the queueing (Kingman) and the rare long generation. The tail is not the mean: a harness with a low mean and a high p95 is a harness that is fast on average and slow on the runs that matter.

The tail’s levers:

• Reduce the utilisation (more capacity), which reduces Kingman’s $\rho/(1-\rho)$.
• Reduce the burstiness (the $c_a^2$), by smoothing the arrivals or by batching.
• Reduce the service-time variance (the $c_s^2$), by limiting the generation’s length (the max tokens) or by a better routing (§11.17).

The p95’s measurement is the trace’s model_latency and tool_latency percentiles, and the diagnosis is the source’s contribution (the model, the tool, the queue).

11.16 Parallelism

11.16.1 The forms

Parallelism in an agent harness has three forms.

Parallel tool calls. The model emits multiple tool calls in one step, and the harness executes them concurrently. The latency is the max, not the sum, and the cost is the same. The form is available when the tools are independent (Module 2, §2.1.3’s multi-call).

Parallel subagents. The orchestrator spawns multiple subagents, and they run concurrently (Module 8, §8.14). The latency is the max, and the cost is the sum. The form is available when the subtasks are independent (Module 8, §8.12.1’s case 3).

Parallel samples. The harness draws $n$ samples and selects by the verifier (Module 4, §4.6). The latency is the max (if concurrent), and the cost is $n$ times. The form is available when the verifier is reliable (Module 4, §4.6.3).

11.16.2 The parallelism’s trade-off

The parallelism trades latency for cost (the parallel subagents and samples) or nothing (the parallel tool calls, if the tools are truly independent). The trade-off’s decision is the latency’s value: a user-facing agent’s latency is worth the cost; a batch agent’s is not.

The parallel tool calls’ special case: the harness must detect the independence (the calls do not share state), because a parallel execution of dependent calls is a race (Module 7, §7.19’s idempotency’s context). The detection is a static analysis (the calls’ arguments’ dependencies) or a dynamic check (the calls’ read/write sets).

11.16.3 The parallel sample’s cost

The parallel sample’s cost is the $n$ samples’ cost, and its benefit is the oracle bound (Module 4, §4.6.2): $1-(1-p)^n$, limited by the verifier’s precision (§4.6.3). The decision rule is the Module 4, §4.6.5’s:

The parallel sample is the same trade-off as Module 4’s, with the latency’s benefit added: the concurrent samples reduce the latency, and the latency’s value is added to the left side.

11.17 Routing and cascades

11.17.1 The routing problem

A deployment may have multiple models (a cheap model, a strong model) and multiple harness configurations (a simple loop, a full harness). The routing problem is: for each task, which (model, harness) pair?

11.17.2 The routing’s criterion

The criterion is the same as the delegation’s (Module 8, Proposition 8.11.1), applied to the model/harness pair: route to the pair that maximises the expected value net of cost:

The success probability and the cost are the pair’s, and they are estimated from the trace (the pair’s historical performance on the task type). The estimation is a contextual bandit (the task’s features are the context, the pair is the arm), and the exploration-exploitation trade-off is the routing policy’s.

11.17.3 The cascade

The cascade is the sequential routing: the cheap model runs first, and if the verifier rejects the output, the strong model runs (Part 0, §0.9, Proposition 0.9.5). The cascade’s cost:

and the escalation condition:

The cascade’s advantage is the cost (the cheap model handles the easy tasks); the disadvantage is the latency (the easy tasks are fast, but the hard ones are the cheap model’s latency plus the strong model’s). The cascade is the right choice when the cheap model’s success rate is high (the easy tasks dominate) and the latency is acceptable.

11.17.4 The cascade’s verifier

The cascade’s verifier is the escalation’s gate (the verifier’s acceptance is the cheap model’s output’s release), and its reliability is the cascade’s accuracy: a verifier with a high false-accept rate releases the cheap model’s wrong outputs, and the cascade’s accuracy is the verifier’s precision (Part 0, §0.5’s precision formula). The cascade’s verifier is the irreducible component (Module 6), and the cascade’s design is the verifier’s design.

11.18 The Pareto frontier

11.18.1 The frontier’s definition

The Pareto frontier of accuracy vs cost is the set of (model, harness) pairs that are not dominated: no other pair has both a higher accuracy and a lower cost. The frontier is the deployment’s choice set: the pairs above the frontier are dominated (worse on both), and the pairs on the frontier are the trade-off’s options.

Worked example. Five pairs:

The frontier: A (cheapest), B, D (the best accuracy). C is dominated by B (B is cheaper and more accurate), and E is dominated by D (D is cheaper and more accurate). The frontier is {A, B, D}.

11.18.2 The frontier’s use

The frontier is the deployment’s decision: the deployment’s accuracy requirement (the $r$ in Module 4, §4.10.3’s delegable horizon) selects the cheapest pair that meets it. A deployment that needs 0.70 accuracy selects B; a deployment that needs 0.75 selects D; a deployment that needs 0.72 has no pair on the frontier (the closest is B, which is below the requirement, or D, which exceeds it) — the requirement is infeasible, and the frontier says so.

The frontier is the verification budget’s context (§11.20): the verification’s spend moves the pair along the frontier (a better verifier can move a pair up and left, if the verifier’s cost is low).

11.18.3 The frontier’s measurement

The frontier is measured from the trace: each pair’s accuracy and cost, over the task distribution. The measurement is the Module 5 discipline (the confidence intervals, the paired comparisons), and the frontier is the upper envelope of the measured points. The frontier’s estimation error is the points’ confidence intervals; a pair near the frontier’s edge is a pair whose position is uncertain, and the uncertainty should be reported.

11.19 The cost’s optimisation

11.19.1 The three levers

The cost’s optimisation has three levers, each with a different effect.

Lever 1 — The model’s choice. The routing (§11.17) selects the model, and the model’s price is the token term’s dominant factor. The routing is the largest lever.

Lever 2 — The context’s efficiency. The cache-aware layout (Module 3, §3.11) and the compaction (Module 3, §3.4) reduce the token term. The lever is the second largest.

Lever 3 — The tools’ efficiency. The tools’ latency and the retries (§11.13.3) reduce the tool term. The lever is the third largest, and it is the one most often missed (the tool term’s dominance in agent workloads, §11.14.2).

The three levers are the cost’s decomposition, and the decomposition’s terms are the levers’ targets. The optimisation is the allocation between the levers, and the allocation is the deployment’s decision (the cost’s value vs the effort).

11.19.2 The cost’s measurement

The cost’s measurement is the trace’s cost_estimate field (§11.1.3), aggregated by the run, the task type, the model, and the harness version. The measurement is the cost model’s input, and the diagnosis (§11.14) is the measurement’s use.

11.19.3 The cost’s governance

The cost’s governance (Module 20) is the budget’s enforcement: a per-run budget (the Module 1 bounds), a per-user budget (the rate limit), a per-deployment budget (the FinOps). The governance is the cost model’s constraint, and the constraint is the deployment’s policy.

PART D — THE VERIFICATION BUDGET

11.20 The cost per verified success

11.20.1 Restating the formula

Part 0, §0.5, Theorem 0.5.7 gives the cost per correctly completed task:

The formula’s terms: $C_g$ (the generation’s cost), $C_v$ (the verification’s cost), $C_f$ (the failure’s cost), $\mathrm{FPR}$ and $\mathrm{TPR}$ (the verifier’s operating point), and $\pi$ (the base rate).

11.20.2 The formula’s interpretation

The formula is the verification’s economics, and it has three properties.

Property 1 — The TPR is the denominator. A higher TPR increases the number of correct tasks per attempt, which lowers the cost per correct task. The TPR’s improvement is the verifier’s investment’s return.

Property 2 — The FPR is the numerator’s penalty. A higher FPR increases the number of false accepts, which incurs the failure’s cost $C_f$. The FPR’s reduction is the verifier’s investment’s return, too.

Property 3 — The verification’s cost is in the numerator. A higher $C_v$ increases the cost per correct task, which is the verifier’s cost. The verifier’s operating point is the trade-off between the TPR/FPR’s benefit and the $C_v$‘s cost.

11.20.3 The optimisation

The optimisation is over the verifier’s operating point (the ROC curve’s points):

The solution is the point where the marginal cost of the TPR’s increase equals the marginal benefit of the FPR’s decrease (the ROC’s slope, weighted by the failure’s cost). The solution is the optimal operating point, and it is the verification budget’s answer.

11.21 The worked optimisation

11.21.1 The setup

A coding agent’s deployment:

• $C_g = 0.05$ (the generation’s cost per attempt).
• $C_v = 0.02$ (the verification’s cost per attempt, the test suite’s run).
• $C_f = 200$ (the failure’s cost, the incident’s cost).
• $\pi = 0.7$ (the base rate of correct outputs).

The verifier’s ROC has three points:

11.21.2 The computation

P1: $(0.05 + 0.02 + 200\times0.05\times0.3)/(0.80\times0.7) = (0.07 + 3.0)/0.56 = 3.07/0.56 = 5.48$.

P2: $(0.05 + 0.03 + 200\times0.10\times0.3)/(0.90\times0.7) = (0.08 + 6.0)/0.63 = 6.08/0.63 = 9.65$.

P3: $(0.05 + 0.05 + 200\times0.20\times0.3)/(0.95\times0.7) = (0.10 + 12.0)/0.665 = 12.10/0.665 = 18.20$.

The optimum is P1 (cost per correct task 5.48), not P2 or P3. The higher TPR points have higher FPR, and the FPR’s cost (the failure’s cost $C_f=200$) dominates. The lesson: the “better” verifier (P3’s TPR 0.95) is worse in the cost per correct task, because its FPR’s penalty exceeds its TPR’s benefit.

11.21.3 The operating point’s choice

The choice is P1, and the choice’s justification is the formula. But P1’s TPR is 0.80, and the autonomy threshold (Part 0, §0.5, Theorem 0.5.6) is $(1-0.80)\times0.7\times200 = 0.20\times140 = 28 > C_h$. If the human’s review cost $C_h$ is, say, 15, the autonomy is not rational at P1 (28 > 15), and the deployment must review every output. The verification budget’s answer (P1) and the autonomy’s answer (not rational) are different, and the deployment’s decision is the combination: the cheapest verifier that supports the autonomy, or the review for the failures.

The combination is the bridge: the verification budget’s optimisation (§11.20.3) is the cost’s answer; the autonomy threshold (Part 0, §0.5) is the safety’s answer; the deployment’s choice is the point that satisfies both. If no point satisfies both, the deployment must either accept the review’s cost (the human’s) or improve the verifier (a better ROC).

11.22 The verification’s allocation

11.22.1 The three placements’ budgets

The verification’s budget is allocated across the three placements (Module 7, §7.11.2):

• Pre-execution (the policy engine): cheap, high-precision for the action’s legality.
• Post-execution (the tests): moderate, the main verifier.
• Post-release (the monitor, the CI): the deployment’s cost, the residual risk’s detection.

The allocation is the same optimisation, with the three placements’ verifiers composed (§11.22.2). The composition’s operating point is the three placements’ combination, and the budget is the combination’s cost.

11.22.2 The composition’s ROC

The composed verifier’s ROC is the intersection of the individual verifiers’ ROCs (Module 7, §7.8.2’s composition): a correct output must pass all verifiers, so the composed TPR is the product (if independent), and the composed FPR is the product. The composition’s ROC is above the individual ROCs (a higher TPR for the same FPR, if the verifiers are independent and their errors are uncorrelated), and the composition’s cost is the sum.

The composition’s benefit is the diversity: two independent verifiers with 0.9 TPR and 0.1 FPR each have a composed TPR of 0.81 and a composed FPR of 0.01, which is a much higher precision (Part 0, §0.5’s precision formula) — a 0.81/0.01 verifier is far more precise than a 0.9/0.1 verifier. The composition’s cost is the sum of the two verifiers’ costs, and the trade-off is the diversity’s benefit vs the cost’s sum.

11.22.3 The allocation’s decision

The allocation’s decision is the verification budget’s allocation: the pre-execution, the post-execution, and the post-release’s spend. The decision is the deployment’s, and the criterion is the marginal cost per correct task of each placement: the placement with the lowest marginal cost per correct task is the first to invest in, and the investment continues until the marginal costs are equal (the same logic as the guide/sensor allocation, Module 9, §9.6).

11.23 The bridge to Modules 17c and 22

11.23.1 Module 17c

Module 17c (the verification budget and the autonomy economics) develops the formula’s application: the optimal verification spend for a stated deployment, the operating point’s choice, and the autonomy’s decision. This module’s Part D is the foundation, and Module 17c is the application.

11.23.2 Module 22

Module 22 (the production operations and organisational adoption) develops the unit economics: the cost per success, the deployment’s FinOps, the team’s budget. This module’s Part C and Part D are the cost model, and Module 22 is the operationalisation.

11.23.3 The common thread

The common thread is the cost per correct task (Part 0, §0.5): it is the verification’s cost, the deployment’s economics, and the autonomy’s criterion, in one formula. The curriculum’s “cost” theme is the formula’s application, and this module is its derivation.

11.24 The cost’s uncertainty

11.24.1 The estimation’s error

The cost model’s inputs ($p_{\text{tok}}$, $\bar c_{\text{tool}}$, $C_f$, the TPR/FPR) are estimates, not exact values. The cost per correct task’s estimate has a variance, and the variance’s sources:

• The token count’s variance (the model’s output length).
• The tool’s cost’s variance (the latency, the retry’s rate).
• The verifier’s TPR/FPR’s variance (the calibration’s sampling error).
• The failure’s cost’s variance (the incident’s severity).

11.24.2 The sensitivity

The sensitivity analysis is the which-input-matters question: the cost per correct task’s derivative with respect to each input. The derivatives identify the inputs whose estimation error matters most.

The FPR’s sensitivity. The derivative with respect to FPR is $C_f(1-\pi)/(\mathrm{TPR}\pi)$, which for the worked example ($C_f=200$, $\pi=0.7$, TPR=0.8) is $200\times0.3/(0.8\times0.7)=60/0.56=107$. A 0.01 FPR error is a 1.07 cost error, which is 20% of the cost per correct task (5.48). The FPR is the most sensitive input, and its calibration is the most important.

The TPR’s sensitivity. The derivative with respect to TPR is $-(C_g+C_v+C_f\,\mathrm{FPR}(1-\pi))/(\mathrm{TPR}^2\pi)$, which for the worked example is $-(0.07+3.0)/(0.64\times0.7)=-3.07/0.448=-6.85$. A 0.01 TPR error is a 0.069 cost error, which is 1.3% of the cost. The TPR is less sensitive than the FPR, because the FPR’s penalty is larger.

11.24.3 The decision’s robustness

The sensitivity’s analysis is the decision’s robustness: a decision that flips with the inputs’ plausible errors is a decision that is not robust, and the inputs’ calibration is the fix. The robustness’s check is the sensitivity’s range: the decision is robust if it is the same across the inputs’ confidence intervals, and the calibration’s priority is the input with the largest sensitivity.

PART E — INCIDENT ANALYSIS AND POSTMORTEMS FOR NON-DETERMINISTIC SYSTEMS

11.25 The non-determinism’s problem

11.25.1 The problem

A conventional software incident is reproducible: the bug is a deterministic function of the input, and the fix is the code’s change. An agent’s incident is non-deterministic: the same input can produce different outputs (the sampling, the tool’s stochasticity, the model’s version), and the failure is a distribution’s tail, not a deterministic bug.

The non-determinism’s problem for the postmortem:

• The reproduction is not exact. The analyst cannot re-run the same failure; they can only sample from the same distribution, and the sample may not reproduce.
• The attribution is probabilistic. The failure’s cause is “the model’s output, with probability $p$”, not “the code’s line”.
• The fix is statistical. The fix is a harness change that reduces the failure’s probability, and the fix’s validation is the distribution’s shift, not the single failure’s disappearance.

11.25.2 The four causes

The non-deterministic failure’s causes fall into four categories, and the postmortem’s first task is the classification.

Cause 1 — The model’s output. The model produced a wrong output (the wrong tool call, the wrong answer). The cause is the model’s capability, and the fix is the model’s upgrade or the harness’s compensation (a guide, a verifier).

Cause 2 — The harness’s component. The harness’s component (the context manager, the tool, the policy engine) produced a wrong behaviour. The cause is the component’s bug or design, and the fix is the component’s change.

Cause 3 — The environment’s state. The environment (the tool’s service, the network, the data) was in a state that the harness did not handle. The cause is the environment’s stochasticity, and the fix is the harness’s robustness (a retry, a timeout, a fallback).

Cause 4 — The specification’s ambiguity. The task’s specification was ambiguous, and the model’s interpretation differed from the intent. The cause is the specification, and the fix is the spec’s clarification (or the verifier’s tightening).

The four causes are the classification’s first step, and the classification determines the fix’s owner: the model team (1), the harness team (2), the platform team (3), the product team (4).

11.25.3 The trace’s role

The trace is the evidence, and it is the only evidence for the non-deterministic failure: the single failure’s trace is the record of what happened, and the aggregate’s trace is the record of the distribution. The postmortem’s evidence is the trace’s fields:

• The model and the sampling parameters (the model’s cause).
• The component_versions (the harness’s cause).
• The tool_status and the environment’s fields (the environment’s cause).
• The reasoning (the specification’s interpretation).

The trace’s completeness (§11.3.3) is the postmortem’s possibility: a trace with gaps cannot be the evidence, and the postmortem is the speculation.

11.26 The postmortem’s structure

11.26.1 The blameless postmortem

The blameless postmortem is the standard practice (from the SRE literature): the postmortem focuses on the system’s failure, not the individual’s blame. The rationale: the blame suppresses the reporting, and the reporting is the learning’s source. The blameless structure is the postmortem’s culture, and it is the prerequisite for the evidence’s honesty.

11.26.2 The structure

The blameless postmortem’s sections:

1. The summary. What happened, when, the impact (the users, the duration, the cost).
2. The timeline. The events’ sequence, from the trace (the failure’s detection, the response, the resolution).
3. The impact. The failure’s cost: the users affected, the tasks failed, the revenue lost, the trust’s damage.
4. The root cause. The four causes’ classification (§11.25.2), with the trace’s evidence.
5. The contributing factors. The factors that made the failure more likely or worse (the monitoring’s gap, the rollback’s slowness, the spec’s ambiguity).
6. The action items. The fixes, each with an owner, a deadline, and a success criterion.
7. The lessons. The generalisable insights, for the broader team.

The structure is the postmortem’s content, and the trace is the timeline’s and the root cause’s evidence.

11.26.3 The action items’ discipline

The action items are the postmortem’s output, and their discipline is the Module 5/6 discipline:

• Specific. Each action is a concrete change (a component, a parameter), not a vague intention (“improve the testing”).
• Owned. Each action has an owner and a deadline.
• Measured. Each action has a success criterion (the failure’s rate’s reduction, the trace’s field’s addition).
• Verified. Each action’s effect is measured after the change (the Module 5 comparison).

The discipline is the difference between a postmortem that fixes the failure and one that documents it. The action items’ verification is the fix’s proof, and the trace is the verification’s evidence.

11.27 The non-deterministic failure’s analysis

11.27.1 The distribution’s framing

The non-deterministic failure’s analysis is the distribution’s framing: the failure is not “the run failed”; it is “the run’s outcome was a sample from a distribution, and the sample was a failure”. The analysis’s questions:

• What is the failure’s probability? (The distribution’s tail’s mass.)
• What are the conditions? (The task type, the model, the harness version, the environment’s state.)
• What is the mechanism? (The trace’s reasoning, the component’s behaviour.)
• What is the fix’s expected effect? (The distribution’s shift, the trace’s re-measurement.)

The framing is the statistical view, and it is the non-deterministic failure’s correct view: the single failure is the data point, the distribution is the object, and the fix is the distribution’s change.

11.27.2 The reproduction’s limits

The non-deterministic failure’s reproduction is statistical, not exact: the analyst re-runs the same condition (the task, the model, the harness) many times, and the failure’s rate is the reproduction’s measure. The exact failure’s re-run is not the goal (it may not reproduce); the rate’s re-run is.

The reproduction’s design:

• The same condition: the same task, the same model, the same harness, the same environment.
• The sample’s size: the Module 5 power calculation, for the rate’s estimation.
• The rate’s comparison: the before/after rate, with the confidence interval.

The statistical reproduction is the non-deterministic failure’s evidence, and the exact reproduction is the deterministic failure’s evidence. The distinction is the postmortem’s methodological core.

11.27.3 The fix’s validation

The fix’s validation is the distribution’s shift, measured by the trace:

• The failure’s rate on the affected condition, before and after.
• The failure-mode distribution’s shift, if the fix targets a mode.
• The regression’s absence on the other conditions (the Module 5 held-out validation).

The validation is the Module 10 loop’s evaluation, applied to the incident’s fix. The fix is an edit, and its evaluation is the same discipline: the pre-registered prediction, the power calculation, the held-out validation, the stability’s margin.

11.28 The incident’s learning

11.28.1 The learning’s levels

The incident’s learning has three levels.

Level 1 — The specific fix. The failure’s cause is fixed (the component’s bug, the environment’s handling). The fix is the action item, and it addresses the specific failure.

Level 2 — The class’s fix. The failure’s class is fixed: the same kind of failure (the same mode, the same component) is prevented or detected across the system. The class’s fix is the constraint’s addition (Module 9, §9.2) or the verifier’s extension (Module 7, §7.8).

Level 3 — The process’s fix. The process that allowed the failure is fixed: the testing’s gap, the review’s gap, the monitoring’s gap. The process’s fix is the organisational change, and it is the slowest and most valuable.

The three levels are the postmortem’s depth, and the deepest level is the one that prevents the future failures, not just the current one.

11.28.2 The learning’s recording

The learning is recorded in the postmortem’s database (the searchable record), the failure taxonomy’s extension (the new mode, if the failure does not fit), and the constraint layer’s rules (the new check, if the failure is preventable). The recording is the learning’s institutionalisation: an incident that is not recorded is an incident that will recur.

11.28.3 The learning’s measurement

The learning’s measurement is the recurrence’s rate: the same failure’s (or the same class’s) rate after the fix. The recurrence’s rate is the learning’s effectiveness, and it is the postmortem’s long-term measure. A postmortem that fixes the failure but does not reduce the class’s rate is a postmortem that addressed the symptom, not the cause.

11.29 The postmortem and the curriculum

11.29.1 The connections

The postmortem is the curriculum’s integration point: it uses the trace (this module), the failure taxonomy (this module, Module 18), the verification (Module 7), the constraint layer (Module 9), the harness loop (Module 10), and the governance (Module 20). It is the practice that the modules’ techniques serve, and it is the test of whether the harness is operable: a harness whose failures cannot be postmortemed is a harness that cannot be improved.

11.29.2 The mastery criterion

The mastery criterion (the curriculum’s §3.3) says “every autonomy increase is paired with a named verifier, its ROC, and the autonomy threshold”. The postmortem’s connection is the inverse: every autonomy’s failure is paired with a postmortem, a classification, and an action item. The two are the autonomy’s cycle: the increase is the investment, the postmortem is the learning, and the cycle is the improvement.

LAB 11a — FAILURE CLUSTERING AND A HARNESS CHANGE

11a.1 Task specification

Run the harness on a task set of at least 1,000 runs (or the largest set your budget allows; the analysis is the same at smaller scale, with wider confidence intervals). The task set must have a deterministic verifier and a diverse set of failures (the task set should be hard enough that a meaningful fraction of runs fail).

11a.2 The clustering

Step 1 — Extract the failed traces. The runs with run_outcome = failure, plus the timeout/stall/budget cases.

Step 2 — Extract the tool-call sequences. For each failed run, the ordered list of (tool_name, argument_hash, tool_status).

Step 3 — Cluster. Use the prefix-tree clustering (§11.4.1) or the agglomerative clustering. Report the clusters, their sizes, and their failure signatures (the common tool-call pattern).

Step 4 — Label. Label the largest cluster with the taxonomy mode (§11.5.2), and justify the label with the trace’s evidence.

11a.3 The harness change

Step 5 — Diagnose. Drill down into the largest cluster (§11.7), and identify the mechanism (the component, the decision, the trace’s field).

Step 6 — Propose. A harness change that addresses the mechanism. The change must be a minimal edit (Module 10, §10.10) and a compensating or tuning component (Module 10, §10.3.2’s guardrail).

Step 7 — Predict. The self-declared prediction (Module 10, §10.7): the direction, the magnitude, the metric.

Step 8 — Evaluate. The change’s effect on the task set, with the Module 5 discipline (paired bootstrap, confidence intervals, multiple-comparison correction if multiple changes are tested).

Step 9 — Measure. The failure-mode distribution’s shift, and the cluster’s size’s change.

11a.4 The metrics

11a.5 The hypotheses

State before running:

• H1. The largest cluster is a harness-addressable failure (not a model error), because the failures are diverse and the harness is the common factor.
• H2. The change reduces the largest cluster’s size by at least 30%.
• H3. The success rate improves by at least 3 points, with a CI that excludes zero.
• H4. The failure-mode distribution shifts: the targeted mode’s fraction falls, and the others rise (as the total falls).
• H5. The change’s cost is small (the cost per task’s change is within the noise).

11a.6 Acceptance criteria

11a.7 What good looks like

A good lab report has five properties.

It reports the clusters, not just the largest. The distribution is the diagnostic: a single dominant cluster is a different finding from a long tail.

It justifies the label with the trace. The label “context loss” is supported by the context_dropped field, not by the analyst’s intuition. A label without the trace’s evidence is a guess.

It predicts before evaluating. The prediction is the mechanism check, and its accuracy is the finding (Module 10, §10.23.9).

It reports the distribution’s shift. The success rate’s change is the aggregate; the distribution’s shift is the diagnosis. A report that says “the success rate improved by 4 points” without the distribution’s shift has missed the mechanism.

It reports the cost’s change. The change’s cost is the trade-off, and a success improvement with a cost regression is a different finding from one without.

LAB 11b — TRACE SCHEMA DESIGN AND THE THREE QUERIES

11b.1 Task specification

Design a trace schema that supports the three queries (§11.1.2), instrument the harness to emit it, and demonstrate the three queries on the emitted traces.

11b.2 The schema

The schema must:

1. Adopt the OTel GenAI conventions for the standard fields (§11.2), with the convention’s version recorded.
2. Extend with the harness-specific fields required by the three queries (§11.1.3).
3. State the field’s role for each query (the derivation table, §11.1.4).
4. State the retention policy (the hot/warm/cold tiers, §11.6.2).

11b.3 The instrumentation

Instrument the harness to emit the schema:

• The span hierarchy (the run, the step, the sub-operation).
• The standard fields (the OTel GenAI attributes).
• The harness-specific fields (the component versions, the context’s decisions, the reasoning).
• The cost fields (the tokens, the tool calls, the verifier calls).

11b.4 The three queries

Demonstrate the three queries on the emitted traces:

Query 1 — Failure clustering. Run the clustering (§11.4.1) on the failed traces, and report the clusters and the signatures.

Query 2 — Regression attribution. Introduce a deliberate harness change (a component’s parameter), and demonstrate the attribution: the split by the harness_version, the comparison, and the drill-down to the component.

Query 3 — Cost per success. Compute the cost per success from the trace’s fields, and decompose it into the three terms.

11b.5 The acceptance criteria

11b.6 What good looks like

A good lab report has four properties.

The schema is derived, not assembled. The fields are justified by the queries, not by “it seemed useful”. The derivation table is the schema’s argument.

The queries are demonstrated, not asserted. The three queries are run on the emitted traces, and the results are shown. A schema that “supports” the queries without demonstrating them is a schema whose sufficiency is untested.

The completeness is tested. The record count’s match is the completeness’s check, and it is the schema’s quality measure.

The retention policy is stated. The storage’s cost is the schema’s trade-off, and the policy is the trade-off’s decision.

MODULE 11 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E11.1. A trace schema omits the tool_arguments field. Which query does it break, and what is the consequence?

S11.1. It breaks Query 1 (failure clustering): the clustering needs the tool-call sequence, and without the arguments, two calls to the same tool with different arguments are indistinguishable. The consequence: the clustering cannot distinguish “the agent read the right file” from “the agent read the wrong file”, and the failure signature is coarser. The fix: add the field (or a hash of it, if the arguments are large).

E11.2. A cost regression is observed: the cost per task rises by 20%. The decomposition shows the token term is unchanged, the tool term rises by 40%, and the verifier term is unchanged. What is the diagnosis, and what would you investigate?

S11.2. The diagnosis is the tool term’s regression: the tools are being called more often (or the tools are slower, or the retries are higher). The investigation: (1) the per-tool call counts (which tool’s calls rose); (2) the per-tool latency (which tool is slower); (3) the retry rate (if the tools are failing more). The likely cause: a context change that made the agent re-call a tool (the compaction policy, a prompt change), or a tool’s service degradation. The lesson: the tool term is the dominant cost in agent workloads, and the diagnosis’s first step is the term’s attribution.

E11.3. A verifier’s ROC has points (0.85, 0.05) and (0.95, 0.15). The generation’s cost is 0.05, the verification’s cost is 0.02, the failure’s cost is 100, and the base rate is 0.6. Which operating point minimises the cost per correct task?

S11.3. P1: $(0.05+0.02+100\times0.05\times0.4)/(0.85\times0.6)=(0.07+2.0)/0.51=2.07/0.51=4.06$. P2: $(0.05+0.02+100\times0.15\times0.4)/(0.95\times0.6)=(0.07+6.0)/0.57=6.07/0.57=10.65$. P1 is the optimum (4.06 vs 10.65). The higher TPR’s benefit is outweighed by the FPR’s penalty: the failure’s cost (100) times the base rate (0.4) times the FPR’s increase (0.10) is 4.0, which exceeds the TPR’s benefit (the denominator’s increase). The lesson: the FPR’s penalty is the dominant factor when the failure’s cost is high.

E11.4. The same verifier’s P1 is used, and the human’s review cost is 20. Is autonomy rational at P1?

S11.4. The threshold is $(1-0.85)\times0.6\times100=0.15\times60=9$. The review’s cost is 20. Since $9<20$, the autonomy is rational at P1: the expected missed-failure cost (9) is below the review’s cost (20). The verification budget’s optimum (P1) and the autonomy’s threshold (9 < 20) are both satisfied, so P1 is the deployment’s choice. If the threshold were, say, 25, the autonomy would not be rational, and the deployment would need a better verifier or the review’s acceptance.

E11.5. A postmortem finds that a failure was caused by the model’s output, but the trace’s reasoning field shows the model’s reasoning was correct. What is the diagnosis, and what would you investigate?

S11.5. The diagnosis is a contradiction: the model’s reasoning was correct but its output was wrong, which suggests a decoding or parsing failure, not a reasoning failure. The investigation: (1) the output field (was the output malformed?); (2) the tool_status (was the call denied or failed?); (3) the harness’s parsing (did the harness mis-parse a valid output?); (4) the model’s sampling (was the output a rare sample?). The cause may be the harness’s component (the parsing), not the model’s capability, and the classification (§11.25.2) would be “the harness’s component”, not “the model’s output”. The lesson: the trace’s fields are the contradiction’s evidence, and the classification must follow the evidence, not the assumption.

E11.6. A trace has 10,000 runs, and the failure-mode distribution is 40% “context loss”, 30% “tool failure”, 20% “model error”, 10% “other”. What is the priority, and what would you do?

S11.6. The priority is context loss (40%): it is the largest cluster, and it is a harness-addressable failure (the context manager, Module 3). The action: (1) drill down into the “context loss” cluster to identify the mechanism (the trimming policy, the compaction’s schema); (2) propose a minimal edit (Module 10, §10.10); (3) evaluate it with the Module 5 discipline; (4) re-measure the distribution. The “tool failure” (30%) is the second priority (the tool’s team), and the “model error” (20%) is the model’s capability (not the harness’s fix). The distribution’s ranking is the priority’s basis, and the harness’s fix is the largest addressable cluster.

E11.7. A cost model’s FPR is estimated at 0.05 with a 95% CI of [0.03, 0.08]. The decision (the operating point) is P1 vs P2. The cost per correct task at FPR=0.05 is 5.48 (P1), and at FPR=0.08 is 7.0 (P1). P2’s cost is 9.65. Is the decision robust to the FPR’s uncertainty?

S11.7. The decision is robust: at FPR=0.08, P1’s cost is 7.0, which is still below P2’s 9.65. The decision (P1) holds across the FPR’s CI, so the uncertainty does not flip the decision. The robustness’s check is the decision’s range across the inputs’ CIs, and a decision that holds across the range is robust. The lesson: the sensitivity analysis is the decision’s robustness, and the FPR’s CI is the input’s uncertainty.

E11.8. A postmortem’s action item is “improve the testing”. What is wrong with it, and how would you rewrite it?

S11.8. The action item is not specific: it has no component, no parameter, no success criterion, and no owner. It is an intention, not an action, and it cannot be verified. The rewrite: “Add a property-based test for the parser’s totality (the property: every input produces a result or a typed error), with the owner (the parser’s team), the deadline (two weeks), and the success criterion (the property test’s failure rate falls from 5% to <1% on the held-out corpus).” The rewrite is the discipline (specific, owned, measured, verified), and it is the difference between a postmortem that fixes and one that documents.

EXIT TEST FOR MODULE 11

You are ready for Module 12 when you can, without notes:

1. State the three queries and derive the minimal trace schema from them.
2. State the OTel GenAI conventions, what they standardise, and what they do not.
3. Explain the trace hierarchy and the subagent’s trace.
4. Explain the trace’s completeness requirement and its test.
5. Demonstrate the three queries: clustering, attribution, cost per success.
6. State the single-agent failure taxonomy and its three uses.
7. State the retention tiers and the privacy requirements.
8. Explain the drill-down and its prerequisites.
9. Describe the error-analysis workflow and the automated classification’s audit.
10. Explain the failure-mode distribution’s diagnostic value and its shift’s meaning.
11. Apply the three-term invariant to diagnose a cost regression.
12. State the four latency sources and the p95’s problem.
13. State the routing criterion and the cascade’s condition.
14. Define the Pareto frontier and use it for a deployment’s choice.
15. Derive the cost per verified success, optimise over the ROC, and check the autonomy threshold.
16. State the sensitivity’s order (the FPR is the most sensitive) and the robustness’s check.
17. State the non-determinism’s problem, the four causes, and the trace’s evidence.
18. Structure a blameless postmortem and apply the action items’ discipline.
19. Explain the statistical reproduction and the fix’s validation.
20. State the three learning levels and the recurrence’s measurement.

End of Module 11. Say “next” to receive Module 12 in full.