Week 18. Part III begins. Prerequisite: Modules 1–11, especially Module 2 (tool interfaces), Module 3 (context), Module 5 (evaluation), and Part 0, §§0.5, 0.9.

12.0 Module map and objectives

Modules 1–11 built a harness that runs on any model. Part III asks: which model, and how do you know? This module is the selection methodology. It does not tell you which model to pick — the landscape changes monthly, and any answer given here would be stale before it was read. It tells you how to pick, with the evidence discipline the curriculum has built, and it gives you the vocabulary to read a model card without being misled.

The module has four parts and one lab.

By the end of the module you will be able to do eight things:

1. Read a model card’s architecture claims and translate them into harness-relevant quantities: the active parameters, the KV-cache size, the context cost.
2. Compute a hardware bill of materials from the weights, the KV cache, and the context length, and say whether a given deployment fits.
3. Classify a model’s licence into the three classes and produce a commercial-use verdict for a stated deployment.
4. Apply the per-checkpoint rule: the licence is a property of the weights file, not the brand.
5. Execute the dossier protocol: the five fields, the reproduction, the parse-failure measurement, the format check.
6. Measure the tool-call parse-failure rate and propagate it into the verifier’s operating point.
7. State why a vendor benchmark number is not a selection input until it is reproduced.
8. Produce a go/no-go recommendation with the evidence and the revision condition.

Everything is self-contained. Where Part 0 is used, the result is restated.

PART A — ARCHITECTURE VOCABULARY

12.1 Why architecture matters to the harness engineer

12.1.1 The harness’s questions

The harness engineer does not need to implement a transformer. They need to answer five questions that the architecture determines.

1. Does the model fit? The weights and the KV cache must fit in the available memory.
2. How fast is it? The latency depends on the active parameters, the attention’s cost, and the KV cache’s size.
3. How much context can it use? The nominal context is a configuration; the effective context is a measurement (Module 3, §3.2), and the attention’s design determines the cost of using it.
4. Can it call tools? The tool-call format is a property of the training and the chat template (Module 2, §2.1), and the parse-failure rate is a measurement.
5. What does it cost? The token price and the cache’s price determine the economics (Part 0, §0.9).

The architecture’s vocabulary is the language for answering these questions. The rest of Part A defines the terms.

12.1.2 The two numbers that matter

A dense model is described by one number: its parameter count $P$. A sparse mixture-of-experts (MoE) model is described by two: the total parameter count $P_{\text{total}}$ and the active parameter count $P_{\text{active}}$ per token.

Definition 12.1.1 (Total and active parameters). For a sparse MoE model, $P_{\text{total}}$ is the number of parameters in the weights, and $P_{\text{active}}$ is the number of parameters that participate in the computation for a single token. The ratio $P_{\text{total}}/P_{\text{active}}$ is the sparsity.

The two numbers answer different questions.

• $P_{\text{total}}$ determines the memory. The weights must be stored, so the memory is proportional to $P_{\text{total}}$.
• $P_{\text{active}}$ determines the compute (and hence the latency and the throughput). The floating-point operations per token are proportional to $P_{\text{active}}$.

Worked example. GLM-5.2 is reported at 744 billion total parameters and about 40 billion active per token, a sparsity of about 18.6:1. The weights are 744B parameters; the per-token compute is 40B. A dense 744B model would have the same memory and about 18.6× the compute. A dense 40B model would have the same compute and about 18.6× less memory. The MoE is the memory-for-compute trade-off: it buys a larger model’s capacity at a smaller model’s compute cost, in exchange for the larger model’s memory cost.

The implication for the harness. Total parameter count is a marketing number in the sense that it does not predict latency or throughput; the active parameter count does. A harness that plans capacity from the total parameter count will over-provision the compute and under-provision the memory.

12.2 Sparse mixture-of-experts

12.2.1 The mechanism

A sparse MoE replaces some of the transformer’s feed-forward blocks with a mixture of experts: a set of $N$ expert networks, and a router that selects the top-$k$ experts for each token.

Definition 12.2.1 (Sparse MoE layer). For an input token $x$, the router computes a score $s_i(x)$ for each expert $i$, selects the top-$k$ experts, and outputs

where $E_i$ is expert $i$‘s network and $g_i(x)$ is the gate’s weight (the softmax over the selected experts’ scores).

The total parameters are $P_{\text{total}} = P_{\text{shared}} + N\cdot P_{\text{expert}}$, where $P_{\text{shared}}$ is the non-expert part (attention, embeddings, the dense blocks) and $P_{\text{expert}}$ is one expert’s parameters. The active parameters are $P_{\text{active}} = P_{\text{shared}} + k\cdot P_{\text{expert}}$.

Worked example. GLM-5.2’s configuration: 78 transformer layers, hidden size 6,144, the first three feed-forward blocks dense, the remaining 75 layers with 256 routed experts, 8 selected per token, plus one shared expert. The active count is the shared part plus 8 experts’ parameters plus the shared expert’s; the total is the shared part plus 256 experts’. The 744B/40B figures are the two numbers.

12.2.2 The routing’s consequences

Three consequences matter for the harness.

Consequence 1 — Expert parallelism. The experts are distributed across devices, and a token’s computation is a network all-to-all (the token is sent to the devices holding its selected experts, and the result is returned). The communication cost is the MoE’s operational overhead, and it is the reason MoE serving requires high-bandwidth interconnects (NVLink, InfiniBand). The harness’s capacity planning must account for the communication, not just the compute.

Consequence 2 — Routing imbalance. The router may concentrate on a few experts, leaving the others under-used. The imbalance reduces the effective capacity and can cause expert collapse (a few experts do all the work). The serving stack’s load balancing is the mitigation, and the harness’s throughput measurement should check for it.

Consequence 3 — The quantisation sensitivity. The experts’ weights are the bulk of the model, and quantising them (FP8, FP4) reduces the memory. The quantisation’s effect on the long-horizon tool-use accuracy is often worse than on the short benchmarks (Module 13, §13.5), and the harness must measure it rather than assume it.

12.2.3 The memory-for-compute trade-off

Proposition 12.2.2 (The MoE’s trade-off). A sparse MoE with total $P_{\text{total}}$ and active $P_{\text{active}}$ has the memory of a $P_{\text{total}}$-dense model and the compute of a $P_{\text{active}}$-dense model. The trade-off is favourable when the memory is abundant and the compute is scarce (a large-memory, low-compute deployment), and unfavourable when the memory is scarce (an edge deployment).

Proof. The weights’ memory is $P_{\text{total}}\times b$ bytes; the FLOPs per token are proportional to $P_{\text{active}}$. A dense model with the same memory has $P_{\text{total}}$ active parameters and $P_{\text{total}}/P_{\text{active}}$ times the compute; a dense model with the same compute has $P_{\text{active}}$ parameters and $P_{\text{active}}/P_{\text{total}}$ times the memory. $\square$

The proposition is the reason MoE models are attractive for data-centre inference (the memory is available, the compute is the bottleneck) and unattractive for edge inference (the memory is the constraint). The harness’s hardware BOM (§12.10) is the arithmetic.

12.3 Attention variants and the KV cache

12.3.1 The KV cache

The KV cache stores the key and value activations for the tokens already processed, so that the model does not recompute them for each new token. Its size is the dominant memory cost at long context.

Definition 12.3.1 (KV cache size). For a model with $n_{\text{layers}}$ layers, $n_{\text{kv\_heads}}$ key/value heads, head dimension $d_{\text{head}}$, context length $L$, and $b$ bytes per element:

The factor 2 is for the keys and the values. The $n_{\text{kv\_heads}}$ is the number of key/value heads, which is smaller than the number of query heads in grouped-query attention (GQA), and the ratio is the GQA’s memory saving.

Worked example. A model with 78 layers, 8 KV heads, head dimension 128, context 128k, FP16 (2 bytes): $M_{\text{KV}} = 2 \times 78 \times 8 \times 128 \times 131{,}072 \times 2 \approx 41.8$ GB. At 1M context, the same model’s cache is $\approx 327$ GB — larger than many multi-GPU nodes’ total memory. The KV cache is the long-context’s binding constraint, not the weights.

12.3.2 Multi-head latent attention (MLA)

MLA compresses the key and value activations into a latent vector before caching, rather than caching the full per-head keys and values. The cache stores the latent (dimension $d_c$, much smaller than $n_{\text{kv\_heads}}\times d_{\text{head}}$), and the per-head keys and values are reconstructed on the fly.

Definition 12.3.2 (MLA cache size). $M_{\text{KV}}^{\text{MLA}} = n_{\text{layers}} \times d_c \times L \times b$, where $d_c$ is the latent dimension.

The saving is the ratio $(2 n_{\text{kv\_heads}} d_{\text{head}})/d_c$. For a typical configuration, the saving is a factor of 5–10, which is what makes long context feasible.

12.3.3 DeepSeek sparse attention (DSA) and IndexShare

DSA splits attention into two stages: a lightweight indexer scores the earlier tokens for each query position and selects the top-$k$ (e.g., 2,048), and the main attention operates only on the selected subset. The core attention’s cost scales as $O(Lk)$ rather than $O(L^2)$.

The indexer, however, still compares each query with all preceding positions, so its cost remains quadratic in $L$. At very long contexts, the repeated indexer work across layers becomes noticeable.

IndexShare reuses one layer’s selected token positions in three nearby layers (the pattern is full, shared, shared, shared). The measurement behind it is that adjacent DSA layers’ token selections overlap by 70–100%. The shared layers still compute their own queries, attention weights, and feed-forward updates; they reuse only the selected positions. The reported effect is a 2.9-fold reduction in per-token FLOPs at a one-million-token context — a compute estimate, not an end-to-end speedup.

The harness’s lesson. The long-context architecture’s benefit is real, and it is not a linear speedup. A harness that plans latency from the nominal context and the FLOP reduction will over-estimate the speedup, because the prefill, the indexer, and the communication remain. The measurement (Module 11, §11.15) is the only reliable input.

12.3.4 Hybrid linear attention

A hybrid model interleaves full (softmax) attention layers with linear-attention or state-space layers (e.g., Gated DeltaNet, Kimi Delta Attention). The ratio is typically about 3:1 (three linear layers per full-attention layer).

The benefit is the KV cache’s reduction: the linear layers’ state is a fixed-size vector, not a growing cache, so the cache grows only with the full-attention layers. The cost is the reasoning’s quality: the linear attention is an approximation, and it does not yet displace dense attention in every reasoning regime. The harness’s lesson: the hybrid is an attention-budgeting choice, and its effect on the long-horizon tool-use tasks is a measurement, not a given.

12.4 The long-context claim

12.4.1 Nominal versus effective, restated

Module 3, §3.2 defines the nominal context (the configuration) and the effective context (the length at which the performance holds within a tolerance). The distinction is the harness’s first lesson about long-context models: a 1M-token configuration is a ceiling, not a budget, and the effective length on an integration task is typically a fraction of the nominal.

The architecture determines the cost of the nominal context (the KV cache, the indexer’s quadratic term, the prefill’s time), and the training determines the effective context (the length distribution the model was trained on, the long-context data’s quality). The two are different, and the model card usually reports only the nominal.

12.4.2 The prefill/decode asymmetry

The long-context cost is asymmetric between the two phases.

Prefill processes the whole input in parallel; its cost is the attention’s $O(L^2)$ (or $O(Lk)$ for DSA) and is compute-bound.

Decode generates one token at a time; its cost is the attention over the cached context, and it is memory-bandwidth-bound (the cache must be read for each token). The decode’s latency is proportional to the cache’s size, not the FLOPs.

The asymmetry matters for the harness: a long-context agent’s time-to-first-token is the prefill’s cost (compute-bound), and its inter-token latency is the decode’s cost (bandwidth-bound). A model that is fast at prefill can be slow at decode, and vice versa. The measurement (Module 11, §11.15) must separate the two.

12.4.3 The context’s cost model

Proposition 12.4.1 (The context’s cost). For a context of length $L$, the prefill’s compute is $O(L^2)$ for dense attention, $O(Lk)$ for DSA’s main attention plus $O(L^2)$ for the indexer, and $O(L)$ for a linear-attention layer. The decode’s per-token latency is proportional to the cache’s size, $M_{\text{KV}}(L)$. The total cost is the sum of the prefill’s (once per request) and the decode’s (per generated token).

The proposition is the architecture’s cost summary. Its practical form: the context’s cost is not linear in $L$, and the harness’s cost model (Part 0, §0.9) must include the superlinear term. A harness that prices the context at a constant per-token rate will under-estimate the long-context cost.

12.5 The architecture-to-harness mapping

12.5.1 The five questions, answered

12.5.2 The vocabulary’s limits

The architecture vocabulary is necessary and not sufficient. It tells you what the model can do in principle; it does not tell you what it does on your task. The gap is the selection protocol’s subject (Part C): the dossier, the reproduction, the parse-failure measurement. The architecture narrows the shortlist; the protocol decides.

12.5.3 Concept check

A model card says “744B parameters, 40B active, 1M context.” What are the three quantities the harness engineer must compute before believing the deployment fits?

Answer. (1) The weights’ memory: $744\text{B} \times b$ bytes, where $b$ is the bytes per parameter at the chosen quantisation. (2) The KV cache at the target context: $M_{\text{KV}}(L)$ from Definition 12.3.1, with the attention’s compression (MLA, hybrid) applied. (3) The total VRAM, which is the sum of the weights, the cache, the activations, and the framework’s overhead. The 40B active tells you the compute, not the memory; the 1M context tells you the ceiling, not the cache’s size.

PART B — LICENCE ANALYSIS

12.6 The three classes

12.6.1 The classification

The licence is the first filter, because it eliminates candidates before any technical evaluation, and a licence mistake is the expensive kind (you ship something you have no right to ship).

Definition 12.6.1 (The three licence classes).

Class 1 — OSI-open (Apache 2.0, MIT). Permissive: commercial use, modification, and redistribution are allowed. Apache 2.0 adds a patent grant and a notice/attribution requirement; MIT is a bare attribution. This is the class for a product you distribute.

Class 2 — Custom / community. Broad, royalty-free rights with conditions: an acceptable-use policy, mandatory attribution, naming rules, sometimes a scale cap, and a duty to pass the restrictions downstream. This is not OSI-open, even though it is loosely called “open.”

Class 3 — Non-commercial / restricted. Free to download and evaluate, but not for commercial deployment without a separate licence. Includes research-only cards, some RAIL variants, revenue-gated licences, and non-production licences.

12.6.2 The class members

The classes’ members, as of mid-2026 (verify each against the model card):

Class 1 (OSI-open). Most Qwen sizes, Mistral’s open models, Phi (MIT), gpt-oss (Apache 2.0), DeepSeek-R1 (MIT), DeepSeek V4 (MIT), Gemma 4 (Apache 2.0 for the 26B A4B variant), OLMo, SmolLM, GLM-5 (MIT), Kimi K2.5 (MIT).

Class 2 (custom/community). Llama (Meta Community License), Gemma 1–3 (Gemma Terms of Use), Kimi K3 (Moonshot’s own document, with a commercial-inference clause above a large annual-revenue threshold).

Class 3 (non-commercial/restricted). Original Codestral (MNPL non-production), some Cohere Command terms, various research-only community fine-tunes.

12.6.3 The class is not the verdict

The class is a starting point, not a verdict. Three reasons.

Reason 1 — The class is a family, not a checkpoint. The same brand can have different licences for different sizes or versions. Llama’s terms have been revised across versions; Gemma 1–3 and Gemma 4 differ; Mistral’s checkpoints mix Apache 2.0 and Modified MIT.

Reason 2 — The conditions are the verdict. A Class 2 licence is usable if every condition is met (the acceptable-use policy, the attribution, the naming, the scale cap, the downstream pass-through). A deployment that cannot meet a condition is a deployment that cannot use the model, even though the class is “permissive enough.”

Reason 3 — The conditions follow the derivatives. A fine-tune of a Llama model inherits the Llama licence; a Gemma-derived model inherits the Gemma terms’ prohibited-use policy. The licence is not just the base model’s; it is the derivative’s too.

12.7 The commercial-use decision

12.7.1 The per-checkpoint rule

Design principle (the per-checkpoint rule). The licence is a property of the weights file, not the brand. The verdict must be based on the LICENSE file that ships with the specific checkpoint, not on the marketing page or the family’s reputation.

The principle’s justification: the terms are attached to the weights, and the weights are what you deploy. A brand-level claim (“Qwen is Apache 2.0”) is a summary that may be false for the specific checkpoint (some Qwen checkpoints have been released under more restrictive terms).

12.7.2 The conditions that matter

The conditions that decide a commercial deployment:

Revenue thresholds. Some licences are free below a revenue bar and require a commercial licence above it. Mistral’s Modified MIT checkpoints have a $20M global monthly revenue cap. Kimi K3’s commercial-inference clause applies above a large annual-revenue threshold. The threshold is a hard stop for a deployment above it.

Scale caps. Llama’s Community License allows commercial use below 700M monthly active users, above which a separate Meta licence is required. The cap is irrelevant for most deployments and decisive for a consumer platform.

Acceptable-use policies. Llama’s and Gemma’s licences incorporate a prohibited-use policy that binds the deployer and the downstream users. The policy’s restrictions are the deployment’s constraints, and a use case that violates them is a use case the licence does not permit, regardless of the class.

Attribution and naming. Llama requires “Built with Llama” on derivatives and the “Llama” name in derived model names. The attribution is a product requirement, not a formality.

Downstream pass-through. Gemma’s terms require the downstream users to be bound to the prohibited-use policy; a fine-tune cannot be used for a purpose that would have been prohibited on the base. The pass-through is the licence’s viral condition, and it constrains the product’s distribution.

Geographic restrictions. Llama 4’s Community License reportedly blocks EU-based companies from accepting the terms outright. The restriction is a hard stop for an EU entity, and it is not visible from the class.

The absence of indemnity. None of the ten families’ licences provides an indemnity for third-party claims. The deployment bears the legal risk, and the risk is a governance input (Module 20), not a licence permission.

12.7.3 The verdict’s form

The commercial-use verdict is a structured statement, not a yes/no. It has four fields:

1. The checkpoint. The exact model and size, with the LICENSE file’s hash or URL.
2. The class. OSI-open / custom / non-commercial.
3. The conditions. Each condition, and whether the deployment meets it (the revenue, the scale, the use case, the attribution, the pass-through, the geography).
4. The verdict. Permitted / permitted with conditions / not permitted, with the conditions named.

The verdict is the dossier’s first field (§12.9), and it is the go/no-go’s first gate: a model that fails the verdict is not evaluated further.

12.7.4 Concept check

A model family is marketed as “open source.” Its licence is a custom community licence with a 700M-user cap and an acceptable-use policy. Is it OSI-open, and can you ship it?

Answer. It is Class 2, not OSI-open: the custom licence has conditions (the cap, the policy, the attribution) that OSI-open licences do not. You can ship it if the deployment meets every condition: it is below the cap, its use case is not prohibited, it provides the attribution, and it passes the conditions downstream. The class is not the verdict; the conditions are.

12.8 The licence review procedure

12.8.1 The procedure

Procedure 12.8.1 (Licence review).

1. Fetch the checkpoint’s LICENSE file from the model hub, not the marketing page.
2. Classify it: OSI-open, custom, or non-commercial.
3. Extract the conditions: the revenue thresholds, the scale caps, the acceptable-use policy, the attribution, the pass-through, the geography.
4. Check each condition against the deployment: the revenue, the user count, the use case, the product’s attribution, the downstream distribution, the jurisdiction.
5. Check the derivatives: if the deployment fine-tunes the model, the derivative inherits the conditions; if it distils into a student (Module 10, §10.11), the student’s provenance is the teacher’s licence.
6. Record the verdict in the dossier’s structured form.
7. Re-check when the model’s version changes: the terms move per version, per size, per release month.

12.8.2 The failure modes

Four failure modes, each with a diagnostic.

Failure 1 — The brand-level assumption. The team assumes the family’s reputation is the checkpoint’s licence. Diagnostic: the LICENSE file’s text differs from the marketing summary. Fix: the per-checkpoint rule.

Failure 2 — The missed threshold. The team deploys a revenue-capped model and crosses the cap. Diagnostic: the revenue’s tracking is not linked to the licence’s threshold. Fix: the threshold is a monitoring alert, not a one-time check.

Failure 3 — The derivative’s inheritance. The team fine-tunes a Class 2 model and ships the derivative under a Class 1 assumption. Diagnostic: the derivative’s licence is not the base’s. Fix: the derivative’s licence review is part of the training’s checklist (Module 15, §15.6).

Failure 4 — The pass-through’s neglect. The team distributes a Gemma-derived model without binding the downstream users to the prohibited-use policy. Diagnostic: the distribution’s terms do not include the pass-through. Fix: the product’s terms are reviewed against the licence’s pass-through requirement.

PART C — THE SELECTION PROTOCOL

12.9 The dossier template

12.9.1 The five fields

The dossier replaces the curriculum’s earlier model table (which the critique flagged as the weakest section, because it mixed tier-C sources and unverified claims). The dossier is a learner-filled artefact, and every field is a primary-source measurement.

Definition 12.9.1 (The model-selection dossier). A dossier for a candidate model $m$ and a deployment $D$ has five fields:

1. Licence verdict. The checkpoint, the class, the conditions, the verdict (§12.7.3).
2. Hardware bill of materials. The weights’ memory, the KV cache at the target context, the total VRAM, the interconnect requirement, and the fit verdict (§12.10).
3. A reproduced benchmark number. One vendor-reported number, reproduced in the learner’s own harness, with the Module 5 design (§12.11).
4. The tool-call parse-failure rate. Measured under the learner’s harness, with the precision impact computed (§12.12).
5. The format-compatibility notes. The chat template, the reasoning tokens, the stop tokens, the tool-call serialisation (§12.13).

12.9.2 The deletion rule

Design principle (the deletion rule). Any field that cannot be filled from primary sources within one week is a field that deletes the candidate from the shortlist. A candidate whose licence cannot be verified, whose hardware does not fit, whose benchmark cannot be reproduced, or whose tool calls cannot be parsed is not a candidate.

The rule is the discipline’s operational form: it prevents the dossier from becoming a marketing summary, and it forces the shortlist to the candidates that can actually be evaluated. The one-week bound is the effort’s budget; a candidate that needs more is a candidate whose evaluation cost exceeds its expected value.

12.10 The hardware bill of materials

12.10.1 The arithmetic

The BOM is the sum of four terms.

Definition 12.10.1 (The VRAM budget).

where

and $M_{\text{activations}}$ and $M_{\text{framework}}$ are the runtime’s overheads (typically 10–20% of the weights and cache).

The bytes per parameter $b_{\text{weight}}$ depends on the quantisation: FP16 = 2, BF16 = 2, FP8 = 1, FP4 = 0.5. The quantisation’s memory saving is real, and its accuracy cost is a measurement (Module 13, §13.5).

12.10.2 The worked BOM

The model. GLM-5.2, 744B total, 40B active, 78 layers, 1M nominal context, MLA + DSA.

The quantisation. FP8 (1 byte per parameter).

The weights. $744\text{B}\times1 = 744$ GB.

The KV cache. With MLA, the cache is $n_{\text{layers}}\times d_c\times L\times b$. Take $d_c\approx 512$ (a representative MLA latent dimension), $L=128\text{k}$, FP8: $78\times512\times131{,}072\times1\approx 5.2$ GB. At 1M context: $\approx 41$ GB. (The exact $d_c$ must be read from the model’s config; the example is illustrative.)

The activations and framework. ~10% of the weights and cache: $\approx 75$ GB at 128k.

The total at 128k. $744 + 5.2 + 75\approx 824$ GB. The fit: an 8×H100 node (640 GB) does not fit at FP8; a 16×H100 node (1.28 TB) fits. At 1M context, the cache adds 41 GB and the activations grow; the fit is tighter.

The lesson. The total parameter count (744B) is the memory’s dominant term, and the KV cache is small because MLA compresses it. A model without MLA at the same context would have a cache in the hundreds of GB, and the fit would fail. The architecture’s compression is what makes the deployment possible, and the BOM’s arithmetic is what proves it.

12.10.3 The interconnect

The MoE’s expert parallelism requires the all-to-all communication, and the interconnect’s bandwidth determines the throughput. The BOM must state the interconnect (NVLink within a node, InfiniBand or Ethernet across nodes) and the bandwidth, because a model that fits but cannot communicate is a model that runs at a fraction of its capacity. The capacity planning (Module 13, §13.6) is the measurement.

12.11 Reproduction discipline

12.11.1 The rule

Design principle (reproduction). A vendor-reported benchmark number is not a selection input until it is reproduced in the learner’s own harness, on the learner’s task distribution, with the Module 5 design.

The rule is the curriculum’s evidence-tier discipline (Module 5, §5.14) applied to the model selection. The vendor’s number is a tier C claim: it is a reason to shortlist the candidate, not a reason to select it.

12.11.2 The reproduction’s design

The reproduction is a paired comparison (Module 5, §5.4, Proposition 0.1.8) between the candidate and a baseline (the incumbent model, or a reference), on the learner’s task set:

1. Fix the harness (Module 1, §1.4.5): the same loop, tools, context manager, and verifier for both models.
2. Choose the metric (the success rate, the pass^k, the cost per success).
3. Power the study (Module 5, §5.7): the sample size for the effect the vendor claims.
4. Run the paired comparison, with the confidence interval.
5. Report the reproduction’s result — including the failure to reproduce, which is a finding (Module 10, §10.18’s overfitting check, applied to the vendor’s claim).

12.11.3 The reproduction’s outcome

Three outcomes, each with a decision.

Reproduced. The learner’s number is within the confidence interval of the vendor’s. The candidate’s claim is corroborated, and the selection proceeds.

Not reproduced, and the learner’s number is lower. The vendor’s claim is overfit to the vendor’s harness or task distribution. The candidate’s learner-specific number is the selection input, and the vendor’s is discarded.

Not reproduced, and the learner’s number is higher. The candidate is better on the learner’s task than the vendor claimed. The learner’s number is the input; the discrepancy is a finding about the task distribution’s difference.

The three outcomes are the reproduction’s value: the learner’s own number is the only one that is about the learner’s deployment.

12.12 The tool-call parse-failure rate

12.12.1 The silent failure

The tool-call parse-failure rate is the fraction of the model’s tool-call attempts that the harness cannot parse into a valid call. It is the most common silent failure when an open-weight model runs in a harness built for another model family (Module 2, §2.1.2; Module 13, §13.4).

The failure’s forms:

• The tool call is emitted in the wrong format (not JSON, or a different JSON schema).
• The tool call is truncated (the output’s stop token is wrong).
• The reasoning tokens are mixed with the tool call (the model’s “thinking” and the call are in one stream).
• The tool call’s arguments are in the wrong type (a string where an object is expected).

The failure is silent because the model often produces a plausible-looking output that the harness mis-parses, and the model does not know it failed.

12.12.2 The measurement

The measurement is a per-model, per-harness experiment:

1. Fix the task set (a set of tasks that require tool calls).
2. Run the model with the harness’s parser.
3. Count the parse failures: the attempts where the parser could not produce a valid call.
4. Classify the failures: the format, the truncation, the reasoning’s mixing, the type.
5. Report the rate, with the confidence interval, and the classification.

The measurement’s discipline is the Module 5 paired design: the same tasks, the same harness, the same parser, with the candidate and the baseline. The difference is the model’s parse-failure rate.

12.12.3 The precision impact

The parse-failure rate is not a cosmetic issue; it is a verifier-reliability issue.

Proposition 12.12.1 (Parse failure reduces the effective TPR). Let $p_{\text{parse}}$ be the parse-failure rate, and let the verifier’s TPR and FPR be measured on the parsed outputs. The harness’s effective operating point on the model’s raw outputs is

if the parse failures are treated as rejections. The precision (Part 0, §0.5) becomes

so the precision is unchanged, but the acceptance rate falls by $(1-p_{\text{parse}})$, which increases the cost per correct task by $1/(1-p_{\text{parse}})$.

Proof. A parse failure is a rejection (the harness cannot accept an unparseable call). The accepted set is the parsed-and-accepted set, which is $(1-p_{\text{parse}})$ of the attempts. The TPR and FPR both scale by $(1-p_{\text{parse}})$, so the precision is unchanged; the acceptance rate scales, so the cost per correct task (Part 0, §0.5, Theorem 0.5.7) is divided by $(1-p_{\text{parse}})$. $\square$

Worked example. A model with a 15% parse-failure rate and a verifier with TPR 0.9, FPR 0.05, base rate 0.6. The precision is unchanged (0.964), but the cost per correct task is $1/(1-0.15)=1.18\times$ the cost of a model with a 0% parse-failure rate, because 15% of the attempts are wasted. A 15-point parse-failure rate is an 18% cost premium, and it is invisible in the model’s benchmark numbers.

12.12.4 The mitigation

The mitigation is a per-model choice (Module 2, §2.1.3’s decision rule):

• Grammar-constrained decoding eliminates the parse failures for the syntax and the schema, at the grammar tax’s cost.
• The chat template and the tool-call format must match the model’s training (the native template).
• The parser must handle the model’s reasoning tokens and stop tokens.

The mitigation’s effect is a measurement: the parse-failure rate after the mitigation, and the cost’s change. The choice is the decision rule’s: constrain iff $p_f c_r > c_g + q$, where $p_f$ is the parse-failure rate, $c_r$ the retry’s cost, $c_g$ the constraining’s cost, and $q$ the grammar tax.

12.13 Format compatibility

12.13.1 The four format decisions

The format compatibility is the model’s fit with the harness’s interface. It has four decisions.

Decision 1 — The chat template. The model’s expected format (the role markers, the system prompt’s placement, the special tokens). A harness that uses the wrong template produces a model that behaves worse than its capability, because the input is out of distribution. The template is in the model’s tokenizer_config.json (the chat_template field), and the harness must use it, not a generic one.

Decision 2 — The reasoning tokens. Some models separate their reasoning (“thinking”) tokens from their output; others mix them. The harness must know which, because the mixed reasoning can crowd the output’s budget (Module 4, §4.8.3) and can confuse the parser. The model card states the format; the harness must test it.

Decision 3 — The stop tokens. The model’s generation stops at its stop tokens, and a wrong stop token produces a truncated output. The harness must use the model’s stop tokens, not the vendor’s default. The failure is silent (the output looks truncated, and the model does not know).

Decision 4 — The tool-call serialisation. The model’s tool-call format (the function-calling syntax, the argument’s encoding). The format is a property of the training, and a model with a native tool template (Qwen, Mistral, Llama, Hermes, Granite, GLM, Phi-4-mini) is more reliable than one that needs grammar-constrained decoding bolted on (Gemma 2/3, base Phi-4, DeepSeek-R1). The native template is the prior; the measurement (§12.12) is the verdict.

12.13.2 The compatibility check

The check is a unit test, not a judgement:

1. The template test. Send a known conversation through the harness’s formatter, and compare the result to the model’s expected format (the tokenizer’s apply_chat_template).
2. The reasoning test. Send a prompt that elicits reasoning, and check whether the reasoning is separated from the output.
3. The stop-token test. Generate a known sequence, and check that it stops at the expected token.
4. The tool-call test. Send a tool-call prompt, and check that the parser produces a valid call.

The four tests are the harness’s CI (Module 9, §9.9): a model that fails a test is a model whose format is incompatible, and the failure is caught before the deployment, not in the production.

12.13.3 Concept check

Why is the tool-call parse-failure rate a selection input, not a post-selection fix?

Answer. Because it is a property of the model-harness pair, and it determines the cost per correct task (Proposition 12.12.1). A model with a 15% parse-failure rate is 18% more expensive per correct task than a model with 0%, and the difference is not visible in the benchmark numbers. The mitigation (grammar-constrained decoding, the native template) is a design choice with its own cost (the grammar tax), and the choice’s evaluation is a selection input. A model that cannot be parsed in the harness is a model that cannot be selected, regardless of its benchmark score.

PART D — THE CANDIDATE LANDSCAPE AS HYPOTHESES

12.14 The families

12.14.1 The status of any list

Any list of “the best models” is stale on the day it is written, and the curriculum’s posture is that the list is a starting hypothesis, not a recommendation. The families below are the ones that recur across the 2026 sources; each entry is a hypothesis with a verification requirement, and the dossier (§12.9) is the verification.

12.14.2 The families

The Qwen family (Alibaba). The Qwen3.x and Qwen3.5/3.6 series, with sizes from the small dense models to the MoE models (e.g., 397B-A17B, 122B-A10B, 35B-A3B), under Apache 2.0 for most checkpoints. The family is reported to be the strongest open-weight tool-caller, occupying about 10 of 13 slots on the Berkeley Function-Calling Leaderboard (BFCL) v4, with the score collapsing below about 9B parameters (the 4B is marginal, the 2B unreliable). Verification: the BFCL’s version (v3 vs v4 are not cross-comparable), the checkpoint’s licence, and the parse-failure rate in the learner’s harness.

The GLM family (Z.ai). The GLM-5.x series, with GLM-5.2 at 744B total / 40B active, MLA + DSA, and a 1M context. The licence is reported as MIT for GLM-5. Verification: the exact checkpoint’s licence, the hardware BOM, and the long-context effective length.

The Kimi family (Moonshot). The K2.x and K3 series. Kimi K2.5 is reported under MIT; K2.6 under a Modified MIT; K3 under Moonshot’s own licence with a commercial-inference clause above a large annual-revenue threshold. Verification: the per-checkpoint licence (the K3 clause is a Class 2 condition, not a Class 1 permission), the model’s tool-call format, and the parse-failure rate.

The DeepSeek family. The V3.x and V4 series. DeepSeek-R1 is MIT; DeepSeek V4 (Pro and Flash) is reported under MIT. The V4 architecture is reported as a large MoE (1.6T total / 49B active in the curriculum’s earlier table, which the critique flagged as tier-C; verify against the model card). Verification: the architecture claims from the model card (not hobbyist inference repos), the licence, and the tool-call reliability (DeepSeek-R1 is reported to lack a native tool template and to need grammar-constrained decoding).

The Mistral family. The open models (Apache 2.0 for most) and the Modified MIT checkpoints with a $20M monthly revenue cap. Verification: the per-checkpoint licence (the Apache/Modified MIT mix is the trap), and the tool-call format.

The Gemma family (Google). Gemma 4 under Apache 2.0 (the 26B A4B variant), Gemma 1–3 under the Gemma Terms of Use with the prohibited-use policy and the pass-through. Verification: the per-version licence (the 1–3 and the 4 differ), and the tool-call reliability (Gemma 2/3 is reported to need grammar-constrained decoding).

The Phi and gpt-oss families. Phi-4-mini (MIT, native tool tokens, reported as the safest single small pick), gpt-oss-20b/120b (Apache 2.0) as fine-tuning substrates and baselines. Verification: the small models’ tool-call reliability (the 4B trio is the practical floor), and the licence.

The specialists. IQuest-Q1 (320B sparse MoE, 15B active, 512K context, custom licence, built for command-line coding agents); the domain-specific releases (the Lean 4 proof engineering models, the verifier-rich domains). Verification: the custom licence’s terms, the hardware BOM, and the task-specific reproduction.

12.14.3 The table’s proper form

The families are summarised in the table below, with the evidence status and the verification requirement. The table is not a ranking; it is a shortlist’s starting point.

12.15 What the table does not tell you

12.15.1 The four gaps

The table is a starting point, and it has four gaps that the dossier fills.

Gap 1 — The licence’s conditions. The table’s “reported licence” is a summary; the conditions (the revenue thresholds, the scale caps, the acceptable-use policies) are the verdict. The dossier’s first field is the conditions, not the class.

Gap 2 — The hardware’s fit. The table’s “role” is a capability summary; the BOM is the fit. A model that is “best” and does not fit is not a candidate.

Gap 3 — The reproduced number. The table’s role is the vendor’s claim; the reproduction is the learner’s number. The vendor’s number is a shortlist’s reason, not a selection’s reason.

Gap 4 — The parse-failure rate. The table does not report the tool-call parse-failure rate, and the rate is the cost’s and the reliability’s input. A model with a high rate is more expensive, regardless of its benchmark.

12.15.2 The landscape’s dynamics

The landscape moves monthly. The curriculum’s currency caveat applies: the model families change, the licences change per checkpoint, and the benchmarks’ versions change. The method is stable; the entries are not. The dossier is the method’s application, and it is the only stable selection input.

12.16 The go/no-go decision

12.16.1 The decision’s structure

The decision is a sequence of gates, each of which can stop the candidate.

Procedure 12.16.1 (The go/no-go).

1. Licence gate. The verdict is “permitted” or “permitted with conditions” that the deployment meets. If not, no-go.
2. Hardware gate. The BOM fits the deployment’s hardware. If not, no-go (or a hardware change’s cost).
3. Reproduction gate. The vendor’s number is reproduced, or the learner’s number is acceptable. If not, no-go (or a re-scope of the task).
4. Parse-failure gate. The parse-failure rate is below the threshold at which the cost premium is acceptable. If not, no-go (or the mitigation’s evaluation).
5. Format gate. The four compatibility tests pass. If not, no-go (or the formatter’s fix).
6. The recommendation. The candidate is a go, with the evidence and the revision condition (what would change the decision).

12.16.2 The revision condition

The decision is a hypothesis, and it has a revision condition (Module 1, §1.8.4’s field 4):

• The licence’s change. A new checkpoint’s licence may differ; the verdict is re-checked.
• The hardware’s change. A new node or a new quantisation may change the fit.
• The reproduction’s change. A new task distribution or a new model version may change the number.
• The parse-failure’s change. A new harness or a new template may change the rate.

The revision condition is the decision’s maintenance, and the dossier is the record that makes it revisable.

12.16.3 The worked decision

The candidate. A model with a reported 70% SWE-bench score, an MIT licence, and a 744B/40B MoE.

Gate 1 — Licence. MIT, no conditions. Pass.

Gate 2 — Hardware. The BOM at FP8 is ~824 GB at 128k; the deployment has an 8×H100 node (640 GB). Fail. The candidate needs a 16×H100 node, which the deployment does not have. No-go (or the hardware’s cost is the decision’s new input).

The lesson. The licence passed, the benchmark was attractive, and the hardware failed. The gate’s order is the cheapest-first: the licence is a document review, the hardware is an arithmetic, and the reproduction is an experiment. The cheap gates stop the candidate before the expensive evaluation, and the decision’s cost is the gates’ order.

12.16.4 Concept check

Why is the hardware gate before the reproduction gate?

Answer. Because the hardware’s arithmetic is cheap (a document review and a calculation) and the reproduction is expensive (an experiment). A candidate that does not fit is a candidate whose reproduction is wasted effort. The gates’ order is the cost’s order, and it is the same principle as Module 7’s pre-execution checks: the cheap checks first, the expensive ones only on the survivors.

LAB 12 — THE MODEL-SELECTION DOSSIER

12.17.1 Task specification

Select a use case from your practice (a coding agent, a customer-service agent, a research agent). Shortlist three open-weight models from the families of §12.14, and produce a dossier for each, with the five fields of Definition 12.9.1. Apply the deletion rule (§12.9.2): a candidate that cannot fill a field within one week is deleted, and the deletion is recorded with the reason.

12.17.2 The five fields

Field 1 — Licence verdict. The checkpoint, the LICENSE file’s URL or hash, the class, the conditions, and the four-field verdict (§12.7.3).

Field 2 — Hardware BOM. The weights’ memory, the KV cache at the target context (with the attention’s compression), the total VRAM, the interconnect, and the fit verdict (§12.10).

Field 3 — The reproduced benchmark. One vendor-reported number, reproduced in your harness with the Module 5 design (paired, powered, with the confidence interval). If the reproduction is infeasible in the lab’s budget, state the infeasibility and the minimum detectable effect (§12.11.2).

Field 4 — The parse-failure rate. Measured under your harness, with the classification of the failures and the precision/cost impact (Proposition 12.12.1).

Field 5 — The format-compatibility notes. The four tests (§12.13.2), with the results.

12.17.3 The metrics

12.17.4 The hypotheses

State before running:

• H1. At least one candidate fails the licence gate (a condition the deployment does not meet), because the landscape’s licences are per-checkpoint and the conditions are easy to miss.
• H2. At least one candidate fails the hardware gate, because the largest models do not fit a single node.
• H3. The parse-failure rate is non-zero for at least one candidate, and the candidates with native tool templates have lower rates than the ones without.
• H4. The reproduced benchmark number is lower than the vendor’s for at least one candidate, because the vendor’s harness or task distribution differs.
• H5. The go/no-go decision is driven by the cheap gates (the licence and the hardware), not the expensive ones (the reproduction and the parse-failure).

12.17.5 Acceptance criteria

12.17.6 What good looks like

A good lab report has five properties.

The licence verdict is per-checkpoint. The report quotes the LICENSE file, not the marketing page, and the conditions are checked against the deployment.

The BOM is arithmetic, not a claim. The weights, the cache, and the total are computed, with the quantisation and the context stated. A BOM that says “it fits on 8 GPUs” without the arithmetic is a guess.

The reproduction is honest about its power. The report states the sample size and the minimum detectable effect. A reproduction on 20 tasks that “confirms” a 5-point claim is not a reproduction.

The parse-failure rate is measured, not assumed. The report runs the measurement and classifies the failures. A report that says “the model supports tool calls” without the rate has missed the selection’s most consequential input.

The decision is gated and revisable. The gates are applied in the cost’s order, the first failure stops the candidate, and the recommendation has a revision condition.

MODULE 12 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E12.1. A model has 400B total parameters and 13B active, and it is quantised to FP8. Compute the weights’ memory. If the deployment has a single 80GB GPU, does it fit?

S12.1. $400\text{B}\times1\text{ byte}=400$ GB. The single 80GB GPU does not fit the weights alone. The model needs at least five 80GB GPUs (400/80), plus the KV cache and the activations, so a 8×80GB node is the minimum. The 13B active parameters tell you the compute is manageable; the 400B total tells you the memory is not.

E12.2. A model has 78 layers, 8 KV heads, head dimension 128, and FP16. Compute the KV cache at 32k context. At 128k?

S12.2. $M_{\text{KV}} = 2\times78\times8\times128\times32{,}768\times2 = 2\times78\times8\times128\times65{,}536$. Compute: $78\times8=624$; $624\times128=79{,}872$; $79{,}872\times65{,}536\approx5.234\times10^9$; $\times2\times2=2.094\times10^{10}$ bytes $\approx20.9$ GB. At 128k, four times the context: $\approx83.7$ GB. The cache alone exceeds a single 80GB GPU at 128k, which is why the long-context deployment needs multiple GPUs even when the weights fit.

E12.3. A model’s licence is MIT, but the checkpoint’s LICENSE file says “MIT with a revenue cap of $10M/month." Is the model Class 1 or Class 2, and what is the verdict for a deployment with$2M/month revenue?

S12.3. The licence is Class 2 (custom, with a condition), not Class 1, because the revenue cap is a condition that MIT does not have. The verdict for a $2M/month deployment is permitted with conditions: the deployment is below the cap, so the condition is met, but the revenue must be monitored (the threshold is a monitoring alert, §12.8.2), and the licence must be re-checked if the revenue approaches the cap.

E12.4. A model’s parse-failure rate is 12%, and the verifier’s cost per correct task is 4.0 without the failures. What is the cost with the failures?

S12.4. By Proposition 12.12.1, the cost is divided by $(1-0.12)=0.88$, so $4.0/0.88=4.55$. The premium is 0.55 per correct task, a 13.6% increase. The lesson: the parse-failure rate is a cost multiplier, and a 12% rate is a 13.6% premium that the benchmark numbers do not show.

E12.5. A model’s licence is Llama’s Community License, and the deployment is a consumer app with 800M monthly active users. Is the deployment permitted?

S12.5. No. The Llama Community License’s scale cap is 700M monthly active users; above it, a separate licence from Meta is required, granted at Meta’s sole discretion. The deployment is above the cap, so it is not permitted under the standard licence, and the separate licence’s negotiation is the only path. The lesson: the scale cap is a hard stop for a consumer platform, and it is invisible from the licence’s class.

E12.6. A team reproduces a vendor’s SWE-bench claim on 30 tasks and finds a 4-point improvement with a 95% CI of [-2, 10]. The vendor claimed 6 points. What is the conclusion?

S12.6. The reproduction is inconclusive: the CI includes zero (the improvement is not significant) and includes the vendor’s 6 points (the claim is not contradicted). The study is underpowered for a 6-point effect (30 tasks is far below the ~4,100 per arm that Module 5, §5.7 requires for a 3-point effect on a 60% baseline). The conclusion is not “the vendor is wrong” and not “the vendor is right”; it is “the reproduction cannot distinguish the two, and a larger sample is needed.” The lesson: the reproduction’s power is the conclusion’s validity.

E12.7. A model has a native tool template, and another model requires grammar-constrained decoding. Which is the better selection, and what would you measure?

S12.7. The native-template model is the prior for the better selection, because it is more reliable (the tool calls are in distribution) and cheaper (no grammar tax). But the prior is not the verdict: you would measure both models’ parse-failure rates and cost per correct task in your harness (§12.12), and the measurement decides. A native-template model with a 5% parse-failure rate and a grammar-constrained model with a 1% rate but a 10% grammar tax is a trade-off that the measurement resolves, not the prior.

E12.8. A candidate passes the licence and hardware gates, and the reproduction is within the CI of the vendor’s claim. The parse-failure rate is 20%. What is the recommendation?

S12.8. The parse-failure rate is the decision’s remaining risk: 20% is a $1/(1-0.2)=1.25\times$ cost premium, and it is the reliability’s input (the effective TPR falls by 20%). The recommendation is conditional go: select the candidate if the mitigation (grammar-constrained decoding, the native template’s fix) reduces the rate to an acceptable level, and the mitigation’s cost (the grammar tax) is below the premium it removes. If the mitigation cannot reduce the rate, the candidate is a no-go, because the cost premium and the reliability’s loss are the selection’s dominant factors.

EXIT TEST FOR MODULE 12

You are ready for Module 13 when you can, without notes:

1. Define total and active parameters, and explain why total is a marketing number for latency.
2. Describe the sparse MoE’s mechanism, the expert parallelism, and the routing imbalance.
3. State the MoE’s memory-for-compute trade-off and when it is favourable.
4. Compute the KV cache’s size from the layers, the KV heads, the head dimension, the context, and the bytes.
5. Explain MLA’s compression, DSA’s two-stage cost, and hybrid linear attention’s attention budget.
6. Distinguish nominal from effective context, and explain the prefill/decode asymmetry.
7. State the three licence classes and the per-checkpoint rule.
8. Name the conditions that decide a commercial deployment (the revenue caps, the scale caps, the acceptable-use policies, the attribution, the pass-through, the geography).
9. Produce a four-field licence verdict.
10. State the dossier’s five fields and the deletion rule.
11. Compute a hardware BOM, including the weights, the KV cache, and the total VRAM.
12. State the reproduction rule and design a reproduction with the Module 5 discipline.
13. Measure a parse-failure rate and compute its precision/cost impact.
14. State the four format decisions and the four format tests.
15. Explain why the go/no-go gates are ordered by cost, and apply the order to a candidate.

End of Module 12. Say “next” to receive Module 13 in full.