Weeks 37–38. The final module. Prerequisite: Modules 1–21, especially Module 11 (observability, cost), Module 5 (evaluation), Module 9 (DORA, Goodhart), Module 17c (verification economics), and Part 0, §§0.7, 0.8, 0.9.

22.0 Module map and objectives

Modules 1–21 built a capable, contained, governed system. This module addresses the question that decides whether it survives contact with production: how do you operate a stochastic system, change it safely, and measure its value without gaming the measurement? The module is the curriculum’s operational close. Its central claim is that an agent system in production is a stochastic service, and a stochastic service needs the disciplines of reliability engineering (the SLO, the error budget, the canary, the rollback), the disciplines of economics (the unit cost, the forecast, the Pareto frontier), and the disciplines of organisational design (the team topology, the review practice, the productivity measurement) — adapted to the fact that the service’s behaviour is a distribution, not a deterministic function.

The module has five parts and one lab.

By the end of the module you will be able to do ten things:

1. Specify an SLO for a stochastic agent service, with the metric, the threshold, the window, and the error budget.
2. Derive the SLO from the user’s experience, rather than from the technical metric.
3. Design a canary for a harness change, with the sample size, the duration, the metrics, and the decision rule.
4. Interpret the canary’s result with the sequential test and the confidence interval.
5. Run the rollback, and state the rollback’s criteria and the rollback’s own risk.
6. Design the model-version pinning and the regression suite, and the model upgrade’s procedure.
7. Compute the unit economics (the cost per success, the forecast), and apply the optimization levers.
8. Design the team topology and the review practice for agent-authored changes.
9. Measure the productivity without the Goodhart effects, and state the measurement’s design.
10. Analyse the operational feedback loop’s stability, and design the damping.

Everything is self-contained. Where a prior module is used, the result is restated.

PART A — SLOs FOR STOCHASTIC SYSTEMS

22.1 What an SLO is

22.1.1 The definition

Definition 22.1.1 (Service Level Objective). An SLO is a target for a service’s reliability, expressed as a metric, a threshold, and a window:

The service is “in compliance” if the metric meets the threshold over the window.

The SLO is the reliability’s contract, and it is the basis for the error budget (§22.4), the alerting, and the release’s decisions. The SLO is distinct from the SLI (the indicator, the measurement) and the SLA (the agreement, the contractual consequence).

22.1.2 The SLO for a stochastic service

An agent service is stochastic: the same input can produce different outputs, and the “success” is a probability, not a fact. The SLO’s metric must therefore be the distribution’s statistic, not a point estimate:

• The success rate (the fraction of the runs that succeed), with the confidence interval (Module 5, §5.4).
• The pass^k (the reliability, Module 5, §5.5), if the deployment requires the consistency.
• The latency’s percentile (the p50, the p95, the p99), not the mean.
• The cost per success (the Module 11, §11.13’s), not the cost per call.

The SLO’s window must account for the sampling’s noise: a 99% success rate over 100 runs has a confidence interval of about ±2.8% (the Wilson interval, Module 17b, §17b.3.2), so the SLO’s window must be large enough to distinguish the threshold from the noise. A window of 100 runs cannot certify a 99% SLO; a window of 10,000 can.

22.1.3 The SLO’s four metrics

Metric 1 — The availability. The fraction of the time the service is up and accepting requests. The simplest, and the least informative for the agent (the service can be “up” and failing).

Metric 2 — The success rate. The fraction of the runs that meet the task’s verifier’s verdict. The primary’s SLO for the agent, and the one that corresponds to the user’s experience.

Metric 3 — The latency. The time-to-first-token (TTFT), the time-to-completion, or the p95/p99 of the end-to-end. The user’s patience’s SLO.

Metric 4 — The cost per success. The Module 11, §11.13’s three-term invariant divided by the success’s rate. The economics’s SLO, and the one that the FinOps (§22.21) tracks.

The four metrics are the SLO’s set, and the deployment’s SLO is the composite: the service is “in compliance” iff all four are within their thresholds.

22.2 Deriving the SLO from the user’s experience

22.2.1 The principle

The SLO’s threshold is the user’s tolerance, not the engineer’s preference. The principle:

Design principle (the user’s SLO). The SLO’s threshold is derived from the user’s experience: what fraction of the user’s tasks must succeed, at what latency, for the user to keep using the service.

The derivation’s steps:

1. Identify the user’s task (the unit the user cares about, e.g., “the ticket is resolved”, “the patch is accepted”).
2. Measure the user’s tolerance (the survey, the churn’s analysis, the support’s tickets).
3. Translate to the metric (the success rate, the latency’s percentile).
4. Set the threshold at the tolerance’s boundary, with the margin for the noise.

22.2.2 The worked derivation

A coding agent’s users:

• The survey says the users tolerate up to 3 failed attempts per accepted patch.
• The accepted patch’s rate is the success rate’s complement, and the 3 attempts correspond to a per-attempt success of about 0.67 (the geometric’s, 1 - (1-p)^3 = 0.7 → p ≈ 0.33… let me redo: if the user tolerates 3 attempts and expects one to succeed, the per-attempt success p satisfies 1 - (1-p)^3 ≥ 0.7 for a 70% chance of success within 3 attempts, giving p ≥ 0.29. Hmm, that’s low. Let me use the simpler framing: the users say “we need the patch to be correct 90% of the time on the first attempt, and we’ll accept 3 attempts for the rest” → the SLO’s success rate is 0.90 for the first attempt, and the 3-attempt’s cumulative is 0.999 if independent.

Better worked example: the users say “we need 95% of the tasks to succeed within 10 minutes, and 99% within 30 minutes.” The SLO’s metrics: the success rate at 10 min ≥ 0.95, the success rate at 30 min ≥ 0.99. The engineer’s job is to meet those, not to set them from the model’s benchmark.

22.2.3 The anti-pattern

The anti-pattern is the technical SLO: “the model’s benchmark’s success rate ≥ 70%”. The technical SLO is the model’s metric, not the user’s; it can be met while the user is unhappy (the 70% is on the benchmark, not the user’s tasks), and it can be missed while the user is happy (the benchmark is harder than the user’s tasks). The user’s SLO is the one that matters.

22.3 The SLO’s window and the error budget

22.3.1 The window

The SLO’s window is the period over which the metric is evaluated. The window’s choice:

• The rolling window (the last 30 days, the last 1,000 runs): the standard, the smooth.
• The calendar window (the month, the quarter): the reporting’s, the less responsive.
• The event’s window (the release’s, the incident’s): the investigation’s.

The rolling window is the default, and the 30-day’s or the 1,000-run’s is the practical’s, depending on the traffic’s volume.

22.3.2 The error budget

Definition 22.3.1 (Error budget). The error budget is the amount of unreliability the SLO allows over the window:

For a 99% success SLO over a 1,000-run window, the budget is 10 failed runs.

The budget’s uses:

• The release’s gate: a change is allowed if the budget is not exhausted (the “the budget is healthy, ship” rule), and a change is frozen if the budget is exhausted (the “the budget is burned, stabilize” rule).
• The prioritization’s: the budget’s consumption rate tells you where the reliability work is needed (the incident’s analysis, Module 11, §11.27).
• The risk’s acceptance’s: the budget is the explicit risk’s allowance, and the team’s decision to spend it (the new feature’s risk) is the conscious trade-off.

22.3.3 The budget’s calculation

Worked example. A 99.5% success SLO over a 30-day window, with 50,000 runs per month. The budget is 0.005 × 50,000 = 250 failed runs per month. The current month’s failures: 180. The budget’s remaining: 70. The team’s decision: the next release’s risk is the 70-failure’s budget, and the release is allowed if the canary’s failure’s rate is within the budget’s remaining.

22.4 The SLO’s monitoring

22.4.1 The measurement

The SLO’s measurement is the Module 11’s trace: the success’s rate (the verifier’s), the latency’s percentiles (the trace’s timestamps), the cost’s per success (the three-term’s invariant). The measurement’s frequency is the continuous (the real-time’s) or the batch (the daily’s), and the SLO’s alerting is the burn-rate’s (the budget’s consumption’s rate).

22.4.2 The burn-rate’s alerting

The burn-rate’s alerting (the SRE’s practice): the alert fires when the budget’s consumption’s rate is the unsustainable. The multi-window’s (the 1-hour’s, the 6-hour’s, the 3-day’s) and the multi-burn-rate’s (the 14.4בs for the 1-hour’s, the 6בs for the 6-hour’s, the 1בs for the 3-day’s) is the standard, and it reduces the alert’s noise (the short’s window’s catches the fast’s burn, the long’s window’s catches the slow’s).

22.4.3 The SLO’s review

The SLO’s review’s cadence (the monthly’s, the quarterly’s):

• The budget’s status’s: the consumed’s, the remaining’s, the trend’s.
• The threshold’s appropriateness’s: is the SLO’s still the user’s tolerance’s? (The user’s needs’s change’s, and the SLO’s must’s be’s re-derived’s.)
• The incidents’s: the budget’s consumption’s causes’s, and the actions’s.

22.5 The SLO for the multi-agent’s and the GUI’s

22.5.1 The multi-agent’s

The multi-agent’s SLO’s: the task’s success’s (the whole’s system’s), not the individual’s agents’s. The per-agent’s SLO’s are the internal’s (the worker’s success’s, the orchestrator’s), and the user’s SLO’s is the composite. The composite’s is the product’s (Module 18, §18.23.2), and the SLO’s is the task’s.

22.5.2 The GUI’s

The GUI’s SLO’s: the task’s success’s (the final’s state’s) and the latency’s (the user’s waits’s). The GUI’s latency’s is the the screen’s rendering’s (the seconds’s), not the model’s tokens’s, and the SLO’s must’s reflect’s the user’s perception’s.

22.6 The SLO’s failure’s modes

22.6.1 The four failures

Failure 1 — The technical’s SLO’s. The SLO’s is the model’s benchmark’s, not the user’s task’s. The service is “compliant” and the user is unhappy.

Failure 2 — The window’s too short’s. The SLO’s is evaluated’s on the 100 runs’s, and the noise’s makes’s the compliance’s the coin’s flip’s.

Failure 3 — The threshold’s too tight’s. The SLO’s is the 99.99%‘s, the budget’s is the 5 failures’s per month’s, and the team’s is the frozen’s (the no’s releases’s). The SLO’s is the aspirational’s, not the achievable’s, and the team’s ignores’s it’s.

Failure 4 — The threshold’s too loose’s. The SLO’s is the 80%‘s, the budget’s is the 20%‘s, and the team’s ships’s the unreliable’s. The SLO’s is the the floor’s, not the the target’s, and the user’s gets’s the poor’s.

22.6.2 The discipline

The discipline’s: the SLO’s is the user’s, the window’s is the noise’s-appropriate’s, the threshold’s is the achievable’s and’s the meaningful’s, and the budget’s is the explicit’s.

22.7 Concept check

Why is the “the benchmark’s success rate ≥ 70%” not an SLO?

Answer. Because it is the model’s metric, not the service’s reliability’s contract’s. The benchmark’s success’s is on the benchmark’s task’s, not the user’s; it’s can’s be’s met’s while the user’s tasks’s fail’s (the distribution’s shift’s, Module 5, §5.3) and’s it’s can’s be’s missed’s while the user’s is’s satisfied’s (the benchmark’s is’s the harder’s). The SLO’s is’s the user’s task’s success’s, the latency’s, the cost’s, over’s the window’s, with’s the budget’s. The benchmark’s is’s the input’s to’s the SLO’s (the model’s selection’s, the evaluation’s), not’s the SLO’s.

PART B — CANARYING HARNESS CHANGES

22.8 What a canary is

22.8.1 The definition

Definition 22.8.1 (Canary). A canary is the practice of deploying a change to a small fraction of the traffic (the canary’s group) while the majority continues on the baseline, and comparing the two groups’ metrics before the full rollout.

The canary’s rationale: the change’s effect is measured in production, on the real’s distribution, with the real’s consequences’s limited to the small’s group. It is the Module 5’s A/B test, applied to the deployment’s changes.

22.8.2 The canary’s variants

• The traffic’s split’s: the random’s fraction’s (the 1%‘s, the 5%‘s) of the requests’s to’s the canary’s. The simplest, the most’s general’s.
• The user’s split’s: the specific’s users’s (the internal’s, the beta’s) to’s the canary’s. The controlled’s, the less’s representative’s.
• The task’s split’s: the specific’s task’s types’s to’s the canary’s. The targeted’s, the risky’s for’s the representativeness’s.

The choice’s depends’s on the change’s and’s the risk’s: the random’s traffic’s for the general’s change’s, the user’s split’s for the high-touch’s, the task’s split’s for the domain-specific’s.

22.9 The canary’s statistics

22.9.1 The hypothesis

The canary’s test: the null’s (the change’s has’s no’s effect’s) vs the alternative’s (the change’s improves’s or’s degrades’s the metric’s). The test’s is the Module 5’s:

with’s the paired’s (the same’s tasks’s) or’s the unpaired’s (the different’s requests’s), and’s the confidence’s intervals’s.

22.9.2 The sample’s size

The sample’s size’s for the canary’s: the Module 5, §5.7’s power’s calculation, with’s the effect’s the minimum’s detectable’s the team’s cares’s about’s.

Worked example. The baseline’s success’s 0.92, the minimum’s effect’s 0.02 (the 2-point’s), the $\alpha=0.05$, the power’s 0.80. The $n$‘s per group’s $\approx(1.96+0.84)^2\times2\times0.92\times0.08/0.02^2 = 7.849\times0.1472/0.0004 = 2,889$. The canary’s needs’s the $\approx2,900$‘s runs’s per group’s, which’s at’s the 1,000’s runs’s per day’s is’s the $\approx3$‘s days’s.

22.9.3 The sequential’s test

The canary’s is’s the sequential’s (the data’s arrives’s over’s the time’s), and’s the always-valid’s or’s the sequential’s test’s (the SPRT’s, Module 17b, §17b.7’s) is’s the right’s. The naive’s (“the check’s at’s the end’s”) is’s the the peeking’s problem’s: the repeated’s checks’s inflate’s the false’s positive’s, and’s the sequential’s test’s corrects’s it’s.

The sequential’s canary’s: the the group’s sequential’s probability’s ratio’s test’s (the GSPRT’s) or’s the the Bayesian’s (the posterior’s on’s the effect’s), with’s the stopping’s rule’s.

22.10 The canary’s design

22.10.1 The design’s elements

The traffic’s fraction’s: the 1%–10%‘s, the small’s enough’s to’s limit’s the risk’s, the large’s enough’s to’s reach’s the sample’s size’s in’s the reasonable’s time’s.

The duration’s: the sample’s size’s / the traffic’s rate’s. The 3–7’s days’s is’s the typical’s, and’s the the full’s cycle’s (the weekly’s pattern’s) is’s the requirement’s.

The metrics’s: the SLO’s four’s (the success’s, the latency’s, the cost’s, the availability’s), plus’s the the guardrail’s (the security’s, the safety’s, the fairness’s).

The decision’s rule’s: the the pre-registered’s (the Module 5, §5.10’s): the promote’s if’s the effect’s is’s the positive’s and’s the significant’s, the rollback’s if’s the effect’s is’s the negative’s and’s the significant’s, the extend’s if’s the inconclusive’s.

22.10.2 The guardrails

The guardrails’s the metrics’s that’s must’s not degrade’s, regardless’s of the primary’s effect’s:

• The security’s (the ASR’s, Module 19’s).
• The safety’s (the harmful’s output’s rate’s).
• The fairness’s (the disparate’s impact’s, the Module 20’s).
• The cost’s ceiling’s (the per-task’s budget’s).

The guardrail’s failure’s is’s the the immediate’s rollback’s, regardless’s of the primary’s metric’s. The guardrail’s is’s the Module 7, §7.14’s fail’s-closed’s applied’s to’s the canary’s.

22.11 The canary’s failure’s modes

22.11.1 The four failures

Failure 1 — The underpowered’s. The canary’s is’s the 100’s runs’s, the effect’s is’s the 2-point’s, and’s the test’s is’s the inconclusive’s. The team’s ships’s on’s the noise’s.

Failure 2 — The peeking’s. The team’s checks’s the canary’s every’s hour’s, and’s the first’s significant’s result’s (the false’s positive’s) is’s the ship’s. The sequential’s test’s is’s the fix’s.

Failure 3 — The guardrail’s ignored’s. The primary’s metric’s improves’s, and’s the security’s guardrail’s degrades’s, and’s the team’s ships’s. The guardrail’s must’s be’s the the blocking’s, not’s the the reported’s.

Failure 4 — The distribution’s shift’s. The canary’s traffic’s is’s the specific’s (the low-traffic’s hours’s, the specific’s users’s), and’s the effect’s on’s the canary’s is’s not’s the effect’s on’s the full’s. The fix’s: the random’s split’s, the full’s cycle’s, the re-measurement’s after’s the full’s rollout’s.

22.12 The rollback

22.12.1 The mechanism

The rollback is the reversal’s of the change: the previous’s version’s restored’s, the traffic’s returned’s to’s the baseline’s, the state’s reconciled’s (the Module 7, §7.20’s).

22.12.2 The rollback’s criteria

The rollback’s criteria:

• The guardrail’s failure’s: the immediate’s.
• The primary’s significant’s degradation’s: the canary’s test’s.
• The incident’s: the production’s signal’s (the user’s report’s, the monitor’s).

22.12.3 The rollback’s risk

The rollback’s risk’s: the the rollback’s own’s change’s. The rollback’s is’s a change’s (the state’s, the version’s), and’s it’s can’s fail’s (the previous’s version’s is’s incompatible’s with’s the current’s state’s, the state’s is’s corrupted’s). The rollback’s must’s be’s tested’s (the dry’s run’s, the canary’s of’s the rollback’s), and’s the the forward’s fix’s is’s the alternative’s when’s the rollback’s is’s the risky’s.

22.12.4 The rollback’s drill

The the rollback’s drill’s: the periodic’s (the quarterly’s) exercise’s of’s the rollback’s, on’s the production’s (the small’s blast’s radius’s), to’s verify’s the rollback’s works’s. The drill’s is’s the Module 20, §20.23’s audit’s applied’s to’s the operational’s capability’s.

22.13 The canary’s for the harness’s changes

22.13.1 The harness’s changes’s

The harness’s changes’s (the Module 10’s edits’s) are’s the canary’s subject’s:

• The prompt’s change’s.
• The tool’s schema’s change’s.
• The context’s manager’s change’s.
• The policy’s engine’s change’s.
• The verifier’s change’s (the new’s test’s, the new’s check’s).

The canary’s is’s the the production’s validation’s of’s the change’s, and’s it’s is’s the Module 10, §10.6’s loop’s the production’s arm’s.

22.13.2 The canary’s vs the offline’s evaluation

The canary’s and’s the offline’s evaluation’s (the Module 5’s) are’s the complementary’s:

• The offline’s is’s the the cheap’s, the fast’s, the broad’s (the many’s tasks’s), and’s the the in-distribution’s (the eval’s set’s).
• The canary’s is’s the the real’s, the slow’s, the narrow’s (the production’s traffic’s), and’s the the distribution’s-true’s.

The offline’s is’s the the filter’s (the changes’s that’s fail’s offline’s don’t’s reach’s the canary’s), and’s the canary’s is’s the the confirmation’s (the changes’s that’s pass’s offline’s are’s validated’s in’s the production’s).

22.13.3 The canary’s statistics’s for the harness’s

The harness’s change’s effect’s is’s the the same’s statistic’s (the success’s, the latency’s, the cost’s), and’s the the sample’s size’s is’s the same’s. The canary’s the harness’s is’s the the Module 5’s paired’s design’s, with’s the the same’s tasks’s if’s the canary’s can’s be’s the task’s-based’s, or’s the the random’s traffic’s if’s not’s.

22.14 Concept check

Why is the rollback not the “the undo’s button” and what makes it the risky’s?

Answer. Because the rollback is itself a change (the version’s, the state’s), and it can fail: the previous version may be incompatible with the state the new version created (the schema’s change, the data’s migration), the state may be corrupted, and the rollback’s own change may introduce the new failure. The rollback is the the best’s available’s response to the canary’s failure, and it is not the the safe’s one; the safe’s one is the the forward’s fix’s or the the careful’s rollback’s (the tested’s, the rehearsed’s). The rollback’s risk is the reason the the state’s compatibility’s (the Module 7, §7.6’s) and the the rollback’s drill’s are the deployment’s requirements, not the optional’s.

PART C — MODEL-VERSION PINNING AND REGRESSION SUITES

22.15 The model-version pinning

22.15.1 The principle

Design principle (the pinning). The production’s model’s version’s is’s pinned’s (the exact’s checkpoint’s, the exact’s serving’s configuration’s), and the upgrade’s is’s the the deliberate’s, the tested’s change’s.

The principle’s rationale: the model’s provider’s can’s change’s the model’s (the new’s checkpoint’s, the deprecation’s, the silent’s update’s), and’s the change’s is’s the the production’s behaviour’s change’s without’s the deployment’s consent’s. The pinning’s makes’s the change’s the deployment’s decision’s.

22.15.2 The pinning’s mechanism

• The checkpoint’s: the exact’s version’s (the hash’s, the tag’s), not’s the “the latest’s”.
• The serving’s stack’s: the vLLM’s/SGLang’s/TensorRT’s version’s (Module 13’s), pinned’s.
• The chat’s template’s: the exact’s template’s (Module 13, §13.12’s), pinned’s with’s the checkpoint’s.
• The sampling’s parameters’s: the temperature’s, the top’s-p’s, the seed’s, pinned’s.
• The tool’s schemas’s: the version’s, pinned’s.

The pinning’s is’s the the reproducibility’s (Module 7, §7.7’s) applied’s to’s the model’s.

22.15.3 The deprecation’s risk

The provider’s can’s the deprecate’s the pinned’s checkpoint’s (the API’s model’s, the hosted’s endpoint’s). The deprecation’s the the forced’s upgrade’s, and’s the deployment’s must’s have’s the the migration’s plan’s (the alternative’s model’s, the re-evaluation’s, the canary’s). The open-weight’s model’s (Module 12’s) has’s the the weights’s (the no’s deprecation’s), and’s it’s is’s the the deployment’s control’s; the API’s model’s has’s the the provider’s control’s.

22.16 The regression suite

22.16.1 The definition

Definition 22.16.1 (Regression suite). The regression suite is the set of the tasks, the scenarios, and the checks that the deployment must pass before the upgrade and after the change. It is the the contract’s between the deployment’s current’s behaviour’s and’s the future’s change’s.

The suite’s components:

• The behavioural’s tasks’s: the tasks’s that’s must’s succeed’s (the Module 5’s eval’s set’s, the curated’s).
• The security’s checks’s: the Module 19’s red-team’s suite’s (the ASR’s must’s not’s increase’s).
• The cost’s checks’s: the cost’s per success’s must’s not’s exceed’s the ceiling’s.
• The latency’s checks’s: the p95’s must’s not’s exceed’s the SLO’s.
• The format’s checks’s: the Module 13’s four’s tests’s (the template’s, the reasoning’s, the stop’s, the tool’s call’s).
• The guardrail’s checks’s: the safety’s, the fairness’s, the compliance’s.

22.16.2 The suite’s design

The suite’s design’s is’s the the layered’s:

• The smoke’s (the 10–20’s tasks’s): the fast’s, the every’s change’s.
• The regression’s (the 100–500’s tasks’s): the moderate’s, the daily’s, the pre-release’s.
• The full’s (the 1,000+‘s tasks’s): the slow’s, the the release’s, the the upgrade’s.

The suite’s the the gate’s (the Module 12, §12.16’s), and’s the the failure’s is’s the the block’s (the no’s release’s), not’s the the warning’s.

22.16.3 The suite’s maintenance

The suite’s is’s the the code’s (the tests’s, the scenarios’s), and’s it’s the the rot’s (the stale’s tasks’s, the changed’s distribution’s, the obsolete’s checks’s). The maintenance’s: the the periodic’s review’s (the quarterly’s), the the new’s incidents’s added’s, the the deprecated’s removed’s, the the coverage’s measured’s (the Module 9, §9.7’s).

22.17 The model upgrade’s procedure

22.17.1 The procedure

Procedure 22.17.1 (The model upgrade).

1. The candidate’s evaluation’s (the offline’s): the new’s checkpoint’s on’s the full’s regression’s suite’s, with’s the Module 5’s design’s. The result’s is’s the the go’s/no’s-go’s for’s the canary’s.
2. The canary’s (the §22.8’s): the small’s traffic’s, the same’s metrics’s, the same’s guardrails’s.
3. The staged’s rollout’s (the 5%‘s → the 25%‘s → the 100%‘s): the progressive’s, with’s the monitoring’s at’s each’s stage’s.
4. The post-upgrade’s audit’s: the sample’s, the exploit’s, the cost’s.
5. The rollback’s (the §22.12’s) if’s any’s stage’s fails’s.

22.17.2 The upgrade’s risks

The upgrade’s risks:

• The behaviour’s change’s (the new’s model’s is’s better’s on’s the benchmark’s, worse’s on’s the deployment’s).
• The format’s change’s (the new’s checkpoint’s has’s the different’s template’s, the different’s tool’s format’s, the Module 13’s).
• The cost’s change’s (the new’s model’s is’s larger’s, slower’s, more’s expensive’s).
• The security’s change’s (the new’s model’s is’s more’s susceptible’s to’s the injection’s, the different’s refusal’s).
• The regression’s (the new’s model’s breaks’s the tasks’s the old’s succeeded’s).

The regression’s suite’s (the §22.16’s) is’s the the coverage’s of the risks, and’s the canary’s is’s the the production’s confirmation’s.

22.18 The compatibility

22.18.1 The four compatibilities

• The format’s (the Module 13, §13.12–13.15’s): the chat’s template’s, the tool’s call’s, the stop’s tokens.
• The context’s (the Module 3’s): the effective’s length’s, the compaction’s tolerance’s.
• The verifier’s (the Module 7, §7.8’s): the new’s model’s output’s is’s the verifier’s input’s, and’s the verifier’s must’s handle’s it’s.
• The harness’s (the Module 1’s): the loop’s, the policy’s engine’s, the observability’s — the new’s model’s is’s the same’s harness’s.

22.18.2 The compatibility’s test

The compatibility’s test’s is’s the Module 13’s four’s format’s tests’s + the Module 3’s effective’s length’s measurement’s + the Module 5’s evaluation’s. The tests’s are’s the the CI’s (the Module 9, §9.9’s), and’s the the pre-upgrade’s (the candidate’s), the the post-upgrade’s (the production’s).

22.19 The rollback’s for the model’s

22.19.1 The mechanism

The model’s rollback’s: the previous’s checkpoint’s restored’s, the traffic’s returned’s, the state’s reconciled’s (the traces’s from’s the new’s model’s are’s the audited’s, the state’s is’s checked’s).

22.19.2 The model’s rollback’s risk

The model’s rollback’s risk’s: the new’s model’s created’s the state’s (the edits’s, the traces’s, the caches’s) that’s the old’s model’s does’s not’s expect’s. The state’s compatibility’s (the Module 7, §7.6’s) is’s the check’s, and’s the the rollback’s drill’s (the §22.12.4’s) is’s the rehearsal’s.

22.20 Concept check

Why is the regression’s suite’s gate’s better’s than’s the canary’s alone’s?

Answer. Because the canary’s is’s the the production’s confirmation’s of’s the change’s, and’s the production’s traffic’s is’s the the narrow’s (the canary’s fraction’s) and’s the the slow’s (the days’s). The regression’s suite’s is’s the the offline’s (the many’s tasks’s, the fast’s), and’s it’s catches’s the changes’s that’s the canary’s would’s miss’s (the rare’s tasks’s, the slow’s degradation’s). The two’s are’s the the complementary’s: the suite’s is’s the the filter’s (the changes’s that’s fail’s don’t’s reach’s the canary’s), and’s the canary’s is’s the the confirmation’s (the changes’s that’s pass’s are’s validated’s in’s the production’s). The gate’s is’s the the suite’s (the pre-release’s) and’s the canary’s (the release’s), not’s the either’s alone’s.

PART D — UNIT ECONOMICS AND FINOPS

22.21 The cost per success

22.21.1 The formula

Restating Part 0, §0.9 and Module 11, §11.13:

with’s the cache’s adjustment’s (Module 3, §3.10’s) and’s the failure’s cost’s (the $C_f$‘s, the Module 17c’s) if’s the deployment’s carries’s the failure’s cost’s.

22.21.2 The three-term’s decomposition

The three-term’s is’s the Module 11, §11.13’s, and’s it’s is’s the the diagnostic’s: the tokens’s, the tools’s, the verifiers’s. The agentic’s workload’s is’s the the tools’s-dominant’s (the Module 11, §11.14.2’s), and’s the the verifier’s (the Module 17b’s) is’s the second’s.

22.21.3 The success’s rate’s role

The cost’s per success’s is’s the the cost’s per run’s / the success’s rate’s. The success’s rate’s increase’s (the Module 5’s, the Module 10’s) is’s the the cost’s reduction’s as’s well as’s the quality’s, and’s the two’s are’s the same’s investment’s.

22.22 The forecast

22.22.1 The forecast’s model

The forecast’s: the volume’s (the tasks’s per month’s) × the cost’s per success’s, with’s the growth’s and’s the seasonality’s.

where’s the $V$‘s is’s the volume’s, the $g$‘s is’s the growth’s rate’s, and’s the $t$‘s is’s the months’s.

22.22.2 The forecast’s uncertainty

The forecast’s uncertainty’s: the volume’s (the adoption’s, the churn’s), the cost’s per success’s (the price’s change’s, the model’s change’s), and’s the success’s rate’s (the distribution’s change’s). The forecast’s is’s the the range’s (the 10th’s, the 50th’s, the 90th’s percentile’s), not’s the point’s.

22.22.3 The budget

The budget’s: the forecast’s the the commitment’s, and’s the the contingency’s (the 10–20%‘s) is’s the the variance’s buffer’s. The budget’s review’s is’s the monthly’s, and’s the the variance’s (the actual’s vs the forecast’s) is’s the the signal’s.

22.23 The Pareto frontier

22.23.1 The frontier’s definition

Restating Module 11, §11.18’s: the Pareto’s frontier’s is’s the set’s of’s the (the model’s, the harness’s) pairs’s that’s are’s not’s dominated’s (no’s other’s pair’s has’s the higher’s accuracy’s and’s the lower’s cost’s).

22.23.2 The deployment’s choice

The deployment’s choice’s is’s the the frontier’s point’s that’s meets’s the the accuracy’s requirement’s (the SLO’s) at’s the the lowest’s cost’s. The choice’s is’s the the constraint’s optimization’s, and’s it’s is’s the Module 11, §11.18.2’s.

22.23.3 The frontier’s movement

The frontier’s moves’s when’s the model’s (the new’s checkpoint’s, the price’s change’s) or’s the harness’s (the Module 10’s optimization’s, the Module 17’s verifier’s) changes’s. The the tracking’s of’s the frontier’s is’s the FinOps’s ongoing’s.

22.24 The optimization levers

22.24.1 The levers

• The model’s routing’s (Module 11, §11.17’s): the cheap’s model’s for’s the easy’s tasks’s, the strong’s for’s the hard’s.
• The context’s efficiency’s (Module 3’s): the cache’s layout’s, the compaction’s.
• The tool’s efficiency’s (Module 11, §11.13.3’s): the retry’s reduction’s, the tool’s selection’s.
• The verifier’s efficiency’s (Module 17c’s): the operating’s point’s, the sequential’s verification’s.
• The success’s rate’s (the Module 10’s loop’s, the Module 16’s training’s): the highest’s leverage’s, because’s it’s reduces’s the cost’s per success’s and’s the failures’s.

22.24.2 The order

The order’s: the the success’s rate’s (the highest’s leverage’s), the the model’s routing’s (the largest’s cost’s), the the context’s efficiency’s, the the tool’s efficiency’s, the the verifier’s efficiency’s.

22.25 The FinOps practices

22.25.1 The practices

• The cost’s allocation’s: the per-team’s, the per-feature’s, the per-task’s, with’s the tags’s (the Module 11’s trace’s).
• The anomaly’s detection’s: the cost’s per success’s is’s the metric’s, and’s the sudden’s change’s is’s the alert’s.
• The unit’s economics’s review’s: the monthly’s, with’s the frontier’s, the levers’s, the forecast’s.
• The showback’s/chargeback’s: the teams’s see’s their’s costs’s, and’s the incentive’s is’s the reduction’s.

22.25.2 The governance

The governance’s: the budget’s, the forecast’s, the variance’s review’s, the escalation’s (the over-budget’s). The Module 20, §20.26’s risk’s register’s includes’s the the cost’s risk’s (the price’s change’s, the volume’s change’s).

22.26 The unit economics’s worked’s example

22.26.1 The deployment

The coding’s agent’s:

• The 1,000’s tasks’s per month’s, the growth’s 10%/month’s.
• The cost’s per success’s 0.12 (the Module 11, §11.14’s).
• The success’s rate’s 0.70.

22.26.2 The cost’s per success’s

The cost’s per success’s: the the 0.12 / 0.70 = 0.17 per accepted’s task’s. The success’s rate’s increase’s to 0.80’s reduces’s the cost’s per success’s to 0.15’s (the 12%‘s reduction’s), and’s the Module 10’s loop’s is’s the lever’s.

22.26.3 The forecast’s

The month 1’s: the 1,000 × 0.17 = 170. The month 12’s: the 1,000 × 1.1^11 × 0.17 = 1,000 × 2.85 × 0.17 = 485. The the growth’s is’s the cost’s driver’s, and’s the the success’s rate’s and’s the the frontier’s are’s the offsets’s.

22.26.4 The decision’s

The decision’s: the cost’s per success’s must’s decrease’s at’s the 10%‘s/quarter’s to’s offset’s the volume’s growth’s, and’s the Module 10’s loop’s (the success’s rate’s) and’s the Module 17c’s (the verifier’s) are’s the investments’s.

22.27 Concept check

Why is the success’s rate’s the highest’s leverage’s on’s the unit’s economics’s?

Answer. Because the success’s rate’s is’s the the denominator’s of’s the cost’s per success’s, and’s the the numerator’s (the cost’s per run’s) is’s the the largely’s fixed’s by’s the model’s, the context’s, and’s the tools’s. A success’s rate’s increase’s from 0.70’s to 0.80’s is’s the the 12.5%‘s reduction’s in’s the cost’s per success’s (0.12/0.70 = 0.171’s → 0.12/0.80 = 0.150’s), and’s the the same’s success’s rate’s increase’s is’s the the quality’s improvement’s (the user’s experience’s). The other’s levers’s (the routing’s, the context’s) reduce’s the numerator’s, and’s the success’s rate’s is’s the the only’s lever’s that’s improves’s the quality’s and’s the cost’s at’s the same’s time’s.

PART E — TEAM TOPOLOGIES, REVIEW PRACTICE, AND PRODUCTIVITY MEASUREMENT

22.28 The team topologies

22.28.1 The four teams

The team’s topologies’s (the Team’s Topologies’s framework’s, the Skelton’s and’s the Pais’s, tier B):

• The stream-aligned’s team’s: the team’s that’s owns’s the product’s value’s stream’s (the coding’s agent’s, the support’s agent’s).
• The platform’s team’s: the team’s that’s provides’s the the harness’s (the loop’s, the tools’s, the observability’s, the verifier’s) as’s the internal’s product’s.
• The enabling’s team’s: the team’s that’s helps’s the stream’s teams’s adopt’s the platform’s (the training’s, the consulting’s).
• The complicated’s-subsystem’s team’s: the team’s that’s owns’s the the verifier’s (the module’s 17b’s), the the security’s (the Module’s 19’s), the the alignment’s (the Module’s 20’s) — the deep’s expertise’s.

22.28.2 The agent’s specific’s

The agent’s specific’s topologies’s:

• The harness’s platform’s: the Module’s 1–17’s harness’s as’s the platform’s, and’s the the stream’s teams’s own’s the the tasks’s, the verifiers’s, the policies’s.
• The verifier’s team’s: the the complicated’s-subsystem’s for’s the Module’s 17b’s (the calibration’s, the construction’s), and’s the the security’s team’s for’s the Module’s 19’s.
• The evaluation’s team’s: the Module’s 5’s and’s the Module’s 10’s, the the platform’s or’s the the enabling’s.

22.28.3 The interaction’s modes

The team’s interaction’s modes’s (the framework’s):

• The collaboration’s: the two’s teams’s work’s together’s, the high’s bandwidth’s.
• The X’s-as’s-a-service’s: the one’s team’s provides’s the capability’s, the other’s consumes’s it’s.
• The facilitating’s: the one’s team’s helps’s the other’s, the temporary’s.

The choice’s is’s the the platform’s (the harness’s) as’s the the X’s-as’s-a-service’s, and’s the the verifier’s as’s the the collaboration’s (the stream’s team’s and’s the verifier’s team’s).

22.29 The review practice for agent-authored changes

22.29.1 The review’s difference

The agent-authored’s change’s review’s differs’s from’s the human’s:

• The volume’s: the agent’s produces’s the many’s changes’s, and’s the human’s review’s is’s the bottleneck’s (the Module’s 20, §20.17’s’s).
• The author’s trust’s: the agent’s is’s not’s the trusted’s colleague’s; it’s is’s the the tool’s whose’s behaviour’s is’s the the statistical’s (the error’s rate’s).
• The rationale’s: the agent’s rationale’s (the reasoning’s trace’s) is’s the the unfaithful’s (the Module’s 20, §20.4.2’s), and’s the human’s cannot’s rely’s on’s it’s as’s the the true’s explanation’s.

22.29.2 The review’s design

The review’s design’s:

• The risk’s-based’s: the high’s-risk’s changes’s get’s the human’s review’s, the low’s-risk’s get’s the agent’s + the sample’s.
• The verification’s-first’s: the review’s is’s the the verifier’s confirmation’s, not’s the the reading’s of’s the code’s. The human’s checks’s the verifier’s (the tests’s, the types’s, the security’s) and’s the the sample’s of’s the code’s.
• The signature’s: the human’s who’s approves’s is’s the the accountable’s (the Module’s 20, §20.18’s), and’s the approval’s is’s the the audit’s log’s record’s.
• The feedback’s: the human’s overrides’s are’s the the training’s signal’s (the Module’s 15’s, the Module’s 10’s).

22.30 The productivity’s measurement

22.30.1 The Goodhart’s risk

The productivity’s measurement’s (the Module’s 9, §9.15’s) is’s the the Goodhart’s risk’s: the metric’s (the changes’s per day’s, the lines’s of’s code’s, the tasks’s closed’s) is’s the the target’s, and’s the agent’s (or the team’s) optimizes’s the metric’s, not’s the value’s.

22.30.2 The measurement’s design

The measurement’s design’s:

• The DORA’s (the deployment’s frequency’s, the lead’s time’s, the change’s failure’s rate’s, the restore’s time’s) the system’s metrics’s, not’s the individual’s.
• The outcome’s (the features’s delivered’s, the incidents’s avoided’s, the user’s satisfaction’s) as’s the the goal’s, not’s the process’s.
• The audit’s (the sample’s, the module’s 20’s) to’s check’s the metric’s is’s the real’s.
• The no’s individual’s use’s (the team’s level’s only’s).

22.30.3 The measurement’s cadence

The measurement’s cadence’s: the the weekly’s operational’s (the throughput’s, the incidents’s), the the monthly’s delivery’s (the DORA’s), the the quarterly’s outcome’s (the value’s, the user’s).

22.31 The operational feedback loop’s stability

22.31.1 The loop

The operational’s loop’s: the deploy’s → the canary’s → the promote’s or’s the rollback’s → the new’s deploy’s. The loop’s is’s the Module’s 10, §10.6’s evolution’s loop’s at’s the production’s level’s, and’s the Part’s 0, §0.7’s stability’s analysis’s applies’s.

22.31.2 The stability’s condition

The loop’s is’s the the delayed’s (the canary’s takes’s the days’s) and’s the the noisy’s (the production’s traffic’s is’s the variable’s). The stability’s condition’s (the Part’s 0, §0.7’s):

where’s the $\eta$‘s is’s the change’s size’s, the $L$‘s is’s the sensitivity’s, and’s the $d$‘s is’s the delay’s.

22.31.3 The oscillation’s

The oscillation’s: the team’s adopts’s the change’s, the canary’s degrades’s, the team’s rollback’s, the team’s adopts’s the the reverse’s, the canary’s degrades’s, the team’s rollback’s. The oscillation’s is’s the the change’s too’s large’s (the $\eta$‘s), the the delay’s too’s long’s (the $d$‘s), or’s the the noise’s too’s high’s (the $n$‘s’s). The fix’s: the smaller’s changes’s, the longer’s canary’s (the more’s $n$‘s), the shorter’s delay’s (the faster’s metrics’s), and’s the the damping’s (the pre-registered’s prediction’s, the held-out’s validation’s).

22.31.4 The worked’s margin

The team’s changes’s the harness’s every’s week’s ($\eta=1$‘s), the canary’s 3’s days’s ($d=3$‘s), the sensitivity’s $L=0.5$‘s. The stability’s bound’s: $2/(0.5\times7)=0.571$. The team’s $\eta=1$‘s exceeds’s the bound’s, and’s the loop’s oscillates’s. The fix’s: the $\eta=0.5$‘s (the half’s changes’s), the $d=1$‘s (the daily’s metrics’s), the $L=0.5$‘s → the bound’s $2/(0.5\times3)=1.33$‘s, and’s the $\eta=1$‘s is’s stable’s.

22.32 The organisational adoption

22.32.1 The adoption’s phases

The adoption’s phases’s:

• The pilot’s: the small’s team’s, the one’s task’s family’s, the measured’s.
• The expansion’s: the more’s teams’s, the more’s tasks’s, the platform’s.
• The scaling’s: the organisation’s, the governance’s, the FinOps’s.
• The maturity’s: the continuous’s improvement’s, the loop’s, the frontier’s.

22.32.2 The adoption’s risks

The adoption’s risks’s:

• The pilot’s success’s is’s the misleading’s (the small’s team’s, the cherry’s-picked’s tasks’s).
• The scaling’s breaks’s the harness’s (the context’s, the cost’s, the latency’s).
• The team’s resistance’s (the “the agent’s will’s replace’s us’s”).
• The governance’s lag’s (the Module’s 20’s is’s the afterthought’s).

22.32.3 The adoption’s measurement

The adoption’s measurement’s: the the usage’s (the tasks’s per week’s), the the outcome’s (the value’s), the the trust’s (the override’s rate’s, the satisfaction’s), the the cost’s.

22.33 The curriculum’s close

22.33.1 The synthesis

The curriculum’s is’s the the discipline’s: the harness’s (the Module’s 1–10’s), the models’s (the Module’s 12–16’s), the verification’s (the Module’s 17’s), the multi-agent’s and’s the security’s (the Module’s 18–19’s), the governance’s (the Module’s 20’s), the domains’s (the Module’s 21’s), and’s the operations’s (the Module’s 22’s). The the through’s-line’s is’s the the verification’s: every module’s has’s the verifier’s at’s its’s core’s (the Module’s 5’s, the Module’s 17b’s), and’s the the autonomy’s is’s the the verifier’s consequence’s (the Module’s 17c’s).

22.33.2 The mastery’s

The curriculum’s mastery’s criteria’s (the §3.3’s):

1. The every’s claimed’s improvement’s has’s the uncertainty’s estimates’s and’s the harness’s controls’s.
2. The every’s autonomy’s increase’s is’s paired’s with’s the named’s verifier’s, its’s ROC’s, and’s the autonomy’s threshold’s.
3. The every’s external’s input’s is’s the untrusted’s, and’s the every’s architecture’s is’s checked’s against’s the lethal’s trifecta’s.
4. The every’s citation’s has’s the evidence’s tier’s and’s the reproduction’s status’s.
5. The every’s factorial’s comparison’s reports’s the model’s × the harness’s interaction’s term’s.

The mastery’s is’s the the habit’s, not’s the the credential’s. The curriculum’s is’s the the starting’s, not’s the the end’s: the field’s moves’s (the currency’s caveat’s), and’s the learner’s must’s keep’s the the weekly’s triage’s (the §4’s), the the reproduction’s discipline’s, and’s the the verification’s first’s posture’s.

22.34 Concept check

Why is the operational’s feedback’s loop’s stability’s the “the organisations’s” problem’s, not’s just’s the “the technical’s”?

Answer. Because the loop’s parameters’s ($\eta$, $d$, $L$) are’s the the organisational’s decisions’s: the $\eta$‘s is’s the the change’s size’s (the team’s cadence’s, the release’s granularity’s), the $d$‘s is’s the the delay’s (the canary’s duration’s, the review’s turnaround’s), and’s the $L$‘s is’s the the sensitivity’s (the task’s distribution’s, the model’s). The team’s that’s ships’s the big’s changes’s weekly’s (the $\eta=1$‘s) with’s the 3-day’s canary’s (the $d=3$‘s) oscillates’s, and’s the fix’s is’s the the organisational’s (the smaller’s changes’s, the faster’s feedback’s, the pre-registered’s predictions’s), not’s the model’s. The stability’s is’s the the engineering’s and’s the the management’s joint’s responsibility’s.

LAB 22 — THE PRODUCTION OPERATIONS PLAN

22.35 Task specification

Produce a production operations plan for a deployment (your own, or the one you specified in Lab 21). The plan must include the SLO, the canary, the regression suite, the unit economics, the review practice, and the rollback.

22.36 The plan’s sections

22.36.1 The SLO

1. The four metrics (the success, the latency, the cost, the availability), with the thresholds, the windows, and the error budgets.
2. The derivation from the user’s experience (the survey, the tolerance, the translation).
3. The monitoring (the burn-rate’s alerting, the multi-window’s, the review’s cadence).
4. The noise’s (the sample’s size’s, the confidence’s intervals’s).

22.36.2 The canary

5. The design (the traffic’s fraction, the duration, the metrics, the guardrails, the decision’s rule).
6. The statistics (the power’s calculation, the sequential’s test, the stopping’s rule).
7. The rollback’s criteria and the rollback’s drill.

22.36.3 The regression suite

8. The suite’s layers (the smoke, the regression, the full), with the sizes and the cadence.
9. The gate’s (the block’s on failure, the suite’s as the release’s gate).
10. The maintenance’s (the review’s, the new’s incidents’s, the coverage’s).

22.36.4 The unit economics

11. The cost per success (the three-term’s, the success’s rate’s).
12. The forecast (the volume, the growth, the range).
13. The budget and the contingency.
14. The Pareto frontier (the current’s, the target’s).
15. The optimization levers (the order, the expected’s effect).

22.36.5 The review practice

16. The risk’s-based’s review (the high’s-risk’s human’s, the low’s-risk’s sample’s).
17. The verification’s-first’s review (the verifier’s confirmation’s, the sample’s of the code’s).
18. The accountability’s (the signature’s, the audit’s log’s).
19. The feedback’s (the overrides’s to the training’s and the harness’s).

22.36.6 The operational feedback loop

20. The stability’s analysis ($\eta$, $d$, $L$, the margin).
21. The oscillation’s risk and the damping’s (the smaller’s changes’s, the pre-registered’s predictions’s).
22. The revision’s condition (the triggers’s).

22.37 The metrics

22.38 The hypotheses

• H1. The SLO derived from the user’s experience differs from the technical (benchmark) SLO, and the gap is the deployment’s risk.
• H2. The canary’s sample size at the current’s traffic is underpowered for the minimum detectable effect, and the canary’s duration must be extended.
• H3. The regression suite’s coverage has a gap (the failure mode the suite does not check), and the gap is the residual risk.
• H4. The cost per success’s forecast’s 90th percentile exceeds the budget, and the optimization levers are required.
• H5. The operational loop’s stability margin is $<1$ (the current’s cadence and canary duration), and the oscillation’s risk is real.
• H6. The review practice’s sample is the bottleneck (the high-risk changes exceed the human capacity), and the risk-based allocation is required.

22.39 Acceptance criteria

22.40 What good looks like

A good plan has five properties.

The SLO is the user’s. The thresholds come from the user’s tolerance, not from the model’s benchmark. The derivation is documented, and the gap between the user’s SLO and the technical’s is the finding.

The canary is powered. The sample size is computed from the minimum detectable effect, and the duration follows. A canary that “runs for a week” without the power calculation is a canary that cannot distinguish the signal from the noise.

The unit economics is the range, not the point. The forecast has the 10th/50th/90th percentiles, and the budget has the contingency. A forecast that is a single number is a guess with a decimal point.

The stability is computed. The margin is computed from the change’s size, the canary’s duration, and the sensitivity, and the oscillation’s risk is stated. A plan that ships weekly without the stability analysis is a plan that has not read Part 0, §0.7.

The review is verification-first. The human reviews the verifier’s confirmation and the sample of the code, not the whole code. A review practice that reads every line of the agent’s output is a review practice that has not understood the volume.

MODULE 22 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E22.1. A service’s success SLO is 99% over a 30-day window, with 20,000 runs per month. The current failures are 150. Compute the budget, the remaining, and the release’s decision.

S22.1. The budget = 0.01 × 20,000 = 200 failures. The remaining = 200 − 150 = 50. The release’s decision: the release is allowed if the canary’s expected failures are within the 50. If the canary’s projected failure’s rate is 0.005 (100 failures per 20,000), the release would consume the 100, exceeding the 50; the release is frozen or the change is reduced. The lesson: the budget is the explicit’s allowance, and the release’s decision’s is the arithmetic.

E22.2. A canary’s baseline’s success is 0.90, the minimum effect is 0.03, α=0.05, power=0.80. Compute the sample size per group.

S22.2. n ≥ (1.96+0.84)² × 2 × 0.90 × 0.10 / 0.03² = 7.849 × 0.18 / 0.0009 = 1.413/0.0009 = 1,570 per group. At 1,000 runs/day, the canary needs ~1.6 days per group. The lesson: the 3-point effect needs ~1,600 per group; a 100-run canary cannot detect it.

E22.3. The same canary with 100 runs per group: what is the minimum detectable effect at 80% power?

S22.3. Rearranging: Δ ≥ sqrt(7.849 × 0.18 / 100) = sqrt(0.01413) = 0.119, i.e. ~12 points. The 100-run canary can only detect a 12-point effect, which is huge. The lesson: the underpowered canary is the one that reports “no significant difference” and ships the change that has a 3-point regression. The power calculation is the canary’s first step.

E22.4. A regression suite has 200 tasks; the smoke is 15, the regression is 100, the full is 200. The suite’s failure rate on a known-bad change is 0.85. What does this measure, and what is the gap?

S22.4. The 0.85 is the suite’s TPR on the known-bad change (the fraction of the bad changes the suite catches). The gap is 0.15 — the 15% of the bad changes the suite passes. The gap is the residual risk, and the fix is the suite’s extension (the new tests for the missed modes). The lesson: the suite’s TPR is measurable (the known-bad changes), and the 0.85 is not the “good enough” without the gap’s analysis.

E22.5. A deployment’s cost per success is 0.15, the volume is 5,000/month, and the growth is 8%/month. Forecast the month-12 cost.

S22.5. The volume at month 12 = 5,000 × 1.08^11 = 5,000 × 2.33 = 11,650. The cost = 11,650 × 0.15 = 1,747.5. The forecast’s range: if the cost’s per success’s 10th/90th is 0.12/0.20, the range is 1,398–2,330. The lesson: the growth’s compounding is the cost’s driver, and the range (not the point) is the budget’s input.

E22.6. The same deployment’s success’s rate rises from 0.70 to 0.80. Compute the cost per success and the savings.

S22.6. The cost per run is 0.15 × 0.70 = 0.105 (the fixed cost per run). The cost per success at 0.70 = 0.105/0.70 = 0.15; at 0.80 = 0.105/0.80 = 0.131. The savings = 0.019 per success, a 12.5% reduction. The lesson: the success’s rate’s is the highest leverage (the denominator), and the 10-point’s increase’s is the 12.5%‘s cost’s reduction plus the quality’s improvement.

E22.7. A team ships a change every week (η=1), the canary is 3 days (d=3), the sensitivity is L=0.6. Compute the stability margin, and state the fix.

S22.7. The bound = 2/(0.6 × (1+6)) = 2/4.2 = 0.476. The margin = 0.476/1 = 0.476 < 1. Unstable. The fix: the smaller change (η=0.4, the margin 1.19), or the shorter canary (d=1, the bound 2/(0.6×3)=1.11, the margin 1.11), or both. The lesson: the weekly-big-change + the 3-day-canary oscillates, and the fix is the smaller changes and the faster feedback.

E22.8. A review practice reviews every agent-authored change by reading the code, and the volume is 200 changes/day. What is the problem, and what is the fix?

S22.8. The problem is the volume’s bottleneck: 200 changes/day at even 10 minutes each is 33 hours/day, which is infeasible. The practice is the one that either rubber-stamps (the false accepts) or blocks (the false denies). The fix: the risk-based’s review (the high-risk’s human’s, the low-risk’s verifier + sample) and the verification-first’s review (the verifier’s confirmation, the sample of the code). The lesson: the review’s design is the volume’s function, and the read-everything’s practice is the one that has not understood the agent’s volume.

EXIT TEST FOR MODULE 22

You are ready to complete the curriculum when you can, without notes:

1. Define an SLO and specify the four metrics for an agent service.
2. Derive the SLO from the user’s experience, and state the technical SLO’s anti-pattern.
3. Compute the error budget and the release’s decision.
4. State the burn-rate’s alerting and the SLO’s review’s cadence.
5. Design a canary with the power calculation, the sequential test, the guardrails, and the decision rule.
6. State the canary’s four failures and the rollback’s risk and drill.
7. State the model’s pinning and the deprecation’s risk.
8. Design the regression suite’s layers, the gate, and the maintenance.
9. State the model upgrade’s procedure and the four compatibilities.
10. Compute the cost per success and the forecast’s range.
11. State the Pareto frontier and the optimization levers’ order.
12. State the four team topologies and the agent’s specific allocation.
13. Design the review practice for agent-authored changes (the risk-based, the verification-first, the signature, the feedback).
14. State the productivity measurement’s Goodhart risk and the design.
15. Compute the operational loop’s stability margin and the damping.
16. Produce the production operations plan with the ten sections.

End of Module 22. The curriculum is complete. The learner’s next step is the capstone (the curriculum’s §3), the ongoing literature practice (the §4), and the open questions (the §6).