Weeks 19–20. Prerequisite: Modules 1–12, especially Module 12 (architecture), Module 3 (caching and context), Module 2 (tool calls), and Part 0, §§0.8, 0.9.

13.0 Module map and objectives

Module 12 selected the model. This module makes it run: the serving stack that executes it, the quantisation that fits it, the chat template and tool-call parser that connect it to the harness, and the capacity plan that sizes it. The module’s theme is that serving is where the model’s architecture meets the harness’s interface, and the failures are mostly at the interface: a wrong template, a wrong stop token, a wrong parser, a quantisation that degrades the long-horizon accuracy while the short benchmark looks fine.

The module has four parts and one lab.

By the end of the module you will be able to do eight things:

1. Explain continuous batching, paged attention, and prefix caching, and say which one affects which metric (throughput, latency, cost).
2. Compute the speculative decoding’s speedup from the acceptance rate and the draft length, and say when it is worth it.
3. Decide whether disaggregated prefill/decode or expert parallelism is worth the operational cost for a given deployment.
4. Choose a quantisation format, and measure its effect on the long-horizon tool-use accuracy rather than on a short benchmark.
5. Diagnose the four format failures (template, reasoning tokens, stop tokens, tool-call serialisation) and unit-test them.
6. Characterise the agent workload and compute the throughput and p95 latency from the batch size, the service time, and the utilisation.
7. Size a serving deployment from the target throughput and the per-GPU throughput.
8. Replay a captured agent workload against a serving stack and measure the throughput, p95, and parse-failure rate.

Everything is self-contained. Where Part 0 or a prior module is used, the result is restated.

PART A — SERVING STACKS

13.1 What a serving stack does

13.1.1 The four functions

A serving stack is the software between the harness’s request and the model’s computation. It has four functions.

Function 1 — Batching. Group the requests into a batch so that the GPU’s parallelism is used. The batch’s composition and scheduling determine the throughput.

Function 2 — Memory management. Allocate the weights and the KV cache, and manage the cache’s growth and eviction. The memory’s limit determines the batch’s size and the context’s length.

Function 3 — Scheduling. Decide which requests run in which step, and when a request’s tokens are generated. The schedule determines the latency and the fairness.

Function 4 — Communication. For a multi-GPU deployment, move the activations and the cache between the devices. The communication’s cost determines the scaling’s efficiency.

The four functions are the stack’s design space, and the stacks (vLLM, SGLang, TensorRT-LLM, llama.cpp) differ in how they implement them. The harness engineer does not implement the stack; they choose the stack for the workload and measure the four functions’ effects.

13.1.2 The two metrics

The stack’s performance is characterised by two metrics, and the agent workload stresses both.

Throughput. The tokens per second, aggregated over the requests. It determines the cost (the GPU’s utilisation) and the capacity (the GPUs needed).

Latency. The time to first token (TTFT) and the inter-token latency (ITL). The TTFT is the prefill’s cost; the ITL is the decode’s cost (Module 12, §12.4.2). The p95 is the deployment’s relevant metric (Module 11, §11.15.3).

The two metrics trade off: a larger batch improves the throughput and worsens the latency (the requests wait longer). The stack’s scheduler is the trade-off’s implementation, and the harness’s workload determines which side matters.

13.1.3 The agent workload’s stress

The agent workload stresses the stack in three ways that the chatbot workload does not.

Stress 1 — Long prefixes. The agent’s context is long (the system prompt, the tool schemas, the history), so the prefill is expensive and the cache is large. The prefix caching (Module 3, §3.10) is the mitigation, and the stack’s prefix-cache implementation is the differentiator.

Stress 2 — Bursty arrivals. The agent issues tool calls in bursts (Module 11, §11.15.3’s $c_a^2$), so the request rate is not Poisson. The Kingman formula’s burstiness term is the latency’s penalty.

Stress 3 — Variable generation. The agent’s output length varies (a short tool call, a long reasoning trace), so the service time’s variance (the $c_s^2$) is high, and the tail latency is worse.

The three stresses are the reason the agent workload’s capacity plan (Part D) is different from the chatbot’s, and the reason the stack’s features (the prefix cache, the continuous batching) are more valuable for the agent.

13.2 Continuous batching

13.2.1 The problem

A naive batching waits for all the requests in a batch to finish before starting the next batch. If one request generates a long sequence, the short requests wait for it, and the GPU’s utilisation collapses (the short requests’ slots are idle once they finish).

13.2.2 The mechanism

Continuous batching (iteration-level scheduling; Orca, Yu et al., OSDI 2022) schedules at the token-generation step, not the request level.

Definition 13.2.1 (Continuous batching). At each decode step, the scheduler:

1. Removes the sequences that have finished (the EOS token, the max length).
2. Adds the new requests (the prefill runs, and the sequences join the batch).
3. Runs the decode step for the batch’s sequences.
4. Repeats.

The batch’s composition changes at every step, so the GPU’s slots are always filled, and a long sequence does not block the short ones.

13.2.3 The throughput’s gain

Proposition 13.2.2 (Continuous batching’s throughput). For a workload with a mix of short and long sequences, the continuous batching’s throughput is approximately

where $B_{\text{eff}}$ is the effective batch size (the average number of sequences in the batch) and $\bar{\ell}_{\text{step}}$ is the average per-step latency. The naive batching’s throughput is

where $\max_i \ell_i$ is the longest sequence’s length and $\bar{\ell}_i$ is the average. The gain is the ratio, which is approximately the ratio of the longest sequence’s length to the average.

Argument. The naive batching’s batch runs for the longest sequence’s duration, so the short sequences’ slots are idle for the difference. The continuous batching fills the slots, so the effective batch size is closer to the configured one. The ratio is the gain, and it is large when the sequence lengths are heterogeneous — which is the agent workload’s case (a short tool call, a long reasoning trace). $\square$

Worked example. A batch of 8 requests: 7 generate 50 tokens each, 1 generates 2,000 tokens. The naive batching’s batch runs for 2,000 steps, and the 7 short requests’ slots are idle for 1,950 steps. The effective work is $7\times50+2{,}000=2{,}350$ tokens over 2,000 steps, so the effective batch size is $2{,}350/2{,}000=1.175$. The continuous batching, once the 7 short requests finish at step 50, admits new requests, and the effective batch size stays near 8. The throughput’s gain is about $8/1.175\approx6.8\times$. The number is illustrative; the lesson is the heterogeneity’s penalty and the continuous batching’s remedy.

13.2.4 The latency’s effect

Continuous batching improves the throughput and worsens the ITL for the requests that are in the batch: the batch’s size is larger, so each step’s computation is larger, so each token’s latency is higher. The TTFT is also affected: a new request waits for the batch’s slot, and the wait is the queueing delay.

The trade-off is the stack’s batch-size policy: a larger batch is a higher throughput and a worse latency. The policy should be tuned to the workload’s latency requirement (the p95’s target), and the tuning is the capacity plan’s input (Part D).

13.2.5 Concept check

Why does continuous batching help the agent workload more than the chatbot workload?

Answer. Because the agent workload’s sequence lengths are more heterogeneous. The agent’s outputs range from a short tool call (a few tokens) to a long reasoning trace (hundreds to thousands), and the naive batching’s longest-sequence penalty is proportional to the heterogeneity. The chatbot’s outputs are more homogeneous (a few hundred tokens each), so the penalty is smaller, and the continuous batching’s gain is smaller. The heterogeneity is the agent’s characteristic, and the continuous batching is its remedy.

13.3 Paged attention

13.3.1 The problem

The KV cache grows with the sequence’s length, and the naive allocation reserves a contiguous block of memory for each sequence’s maximum length. The reservation causes two problems.

Problem 1 — Internal fragmentation. A sequence that reserves 4,096 tokens but generates 100 wastes the 3,996 tokens’ memory.

Problem 2 — External fragmentation. The free memory is fragmented into blocks that are too small for a new sequence’s reservation, so the memory is unused even when the total free memory is sufficient.

13.3.2 The mechanism

Paged attention (vLLM, Kwon et al., SOSP 2023) applies the operating system’s virtual-memory idea to the KV cache.

Definition 13.3.1 (Paged attention). The KV cache is divided into fixed-size blocks (e.g., 16 tokens per block). Each sequence has a block table mapping its logical positions to physical blocks. The blocks are allocated on demand, and the sequence’s cache is the set of blocks its table references. A block is freed when its sequence finishes, and the free blocks are reused.

The mechanism eliminates the internal fragmentation (the last block is partially used, at most 15 tokens wasted) and the external fragmentation (the blocks are fixed-size, so any free block fits any sequence’s need). The memory’s utilisation rises from 20–40% (the naive allocation) to 90%+ (the paged allocation), which is the batch size’s and the context’s gain.

13.3.3 The sharing

The block table enables sharing: two sequences with the same prefix can share the prefix’s blocks (the copy-on-write for the divergent part). The sharing is the basis of the prefix caching (§13.4): a new request whose prefix matches an existing sequence’s prefix references the existing blocks, and the prefill is skipped.

13.3.4 The memory’s arithmetic

Proposition 13.3.2 (The paged cache’s capacity). With a block size $B_{\text{block}}$ and a total memory $M_{\text{KV}}$, the maximum number of cached tokens is $M_{\text{KV}}/(b \cdot B_{\text{block}} \cdot \text{bytes per token})$, where the bytes per token is the cache’s per-token size (Definition 12.3.1 divided by $L$). The batch’s maximum size is the cached tokens divided by the average context’s length.

Worked example. A model with a per-token cache of 0.5 MB (a representative value for a 70B-class model at FP16 with GQA), an 80 GB GPU with 40 GB for the cache: the maximum cached tokens is $40{,}000/0.5=80{,}000$ tokens. At an average context of 8,000 tokens, the batch’s maximum is 10 sequences. At 32,000, it is 2.5. The cache is the batch’s binding constraint, and the context’s length is the trade-off.

13.3.5 Concept check

Why does paged attention matter more for the agent workload than for the chatbot workload?

Answer. Because the agent’s contexts are longer and more variable. The naive allocation’s fragmentation wastes a fraction of the memory, and the fraction is larger when the contexts are longer (the reservation’s waste is proportional to the maximum length, and the agent’s maximum is large). The paged allocation’s utilisation gain is the batch’s and the context’s capacity, and the capacity is what the agent’s long contexts need. The chatbot’s shorter contexts waste less, so the gain is smaller.

13.4 Prefix and KV-cache reuse

13.4.1 The mechanism

The prefix caching (Module 3, §3.10) is the stack’s implementation of the reuse: the stack hashes the prefix’s blocks, and a new request whose prefix’s hash matches an existing cache entry reuses the blocks and skips the prefill.

Definition 13.4.1 (Prefix cache hit). A request’s prefix cache hit occurs when the request’s first $n$ tokens match a cached prefix’s first $n$ tokens. The hit’s length $n$ is determined by the longest matching prefix, and the prefill’s cost is reduced by the hit’s length.

The stack’s implementation differs: vLLM’s automatic prefix caching, SGLang’s RadixAttention (a radix tree of the prefixes), TensorRT-LLM’s KV cache reuse. The mechanism is the same; the data structure and the eviction policy differ.

13.4.2 The hit rate’s arithmetic

Restating Module 3, §3.10’s cost formula, with the stack’s variables:

where $L_p$ is the prefix’s length, $h$ is the hit rate, and $C_{\text{cached}}$ and $C_{\text{prefill}}$ are the per-token prices.

The hit rate’s determinants (Module 3, §3.10.2):

• The prefix’s stability. The stable prefix (the system prompt, the tool schemas) is cached; the volatile content (the conversation) misses from the change’s point.
• The cache’s capacity. The stack’s cache has a finite size; the eviction (LRU or a radix tree’s policy) determines the hit rate.
• The block’s granularity. The cache is block-granular (16 tokens), so a prefix that differs by a few tokens misses from the block’s boundary.

13.4.3 The agent’s layout

The agent’s context layout (Module 3, §3.11) is the hit rate’s lever: the stable content first (the system prompt, the tool schemas), the volatile content last (the instruction, the tool results). The layout is the harness’s decision, and the stack’s caching is the reward.

Worked example. A harness with a 4,000-token system prompt and tool schemas, reused across 1,000 requests per day. The prefix cache’s hit rate is 0.9 (the layout is stable). The prefill’s price is 3 per million tokens; the cached price is 0.3. The per-request cost is $0.9\times4{,}000\times0.3/10^6 + 0.1\times4{,}000\times3/10^6 = 0.00108+0.0012=0.00228$. Without the cache: $4{,}000\times3/10^6=0.012$. The saving is 5.3× on the prefix, which is the same arithmetic as Module 3’s, now on the stack’s side.

13.5 Speculative decoding

13.5.1 The mechanism

Speculative decoding uses a small, fast draft model (or a draft head) to propose $k$ tokens, and the target model verifies the proposal in one forward pass.

Definition 13.5.1 (Speculative decoding). For each step:

1. The draft model proposes $k$ tokens $t_1,\dots,t_k$.
2. The target model computes the probabilities for the $k+1$ positions in one forward pass.
3. The tokens are accepted from left to right, with the acceptance probability $\min(1, p_{\text{target}}(t_i)/p_{\text{draft}}(t_i))$; on the first rejection, the target’s distribution is sampled for the corrected token.
4. The accepted tokens are appended, and the process repeats.

The acceptance rate $\alpha$ is the probability that a proposed token is accepted. The expected number of tokens per target forward pass is

(for $\alpha<1$). At $\alpha=0.8$, $k=4$: $(1-0.8^5)/0.2 = 0.672/0.2 = 3.36$ tokens per target pass. The target’s forward pass is the expensive one, so the decode’s speedup is up to 3.36×, modulo the draft’s cost.

13.5.2 The speedup’s condition

The speedup is bounded by the target’s forward pass’s cost divided by the combined cost:

where $c_{\text{target}}$ is the target’s per-step cost and $c_{\text{draft}}$ is the draft’s. The speedup is favourable when the draft is much cheaper than the target (a small model) and the acceptance rate is high (the draft is aligned with the target).

The acceptance rate’s determinants: the draft’s quality (a better draft is more aligned), the task’s predictability (a predictable text is more accepted), and the temperature (a higher temperature is less predictable). The agent’s tool calls are highly predictable (the schema constrains the format), so the acceptance rate is high for the tool-call portion and lower for the reasoning portion.

13.5.3 The trade-offs

The benefit: the decode’s speedup, which reduces the ITL and the cost (the target’s forward passes are fewer).

The cost: the draft’s compute (a small model’s forward passes), the memory (the draft’s weights), and the complexity (the draft’s training or the draft head’s training).

The decision: use speculative decoding when the decode is the bottleneck (the ITL is the latency’s dominant term) and the draft is available (a small model of the same family, or a trained draft head). The benefit is largest for the workloads with predictable output (the tool calls), and smallest for the open-ended reasoning.

13.6 Disaggregated prefill and decode

13.6.1 The problem

The prefill and the decode have different resource profiles (Module 12, §12.4.2): the prefill is compute-bound, and the decode is memory-bandwidth-bound. In a colocated deployment (the same GPUs serve both), the two phases interfere: a long prefill blocks the decode steps, and the decode’s memory-bandwidth demand starves the prefill.

13.6.2 The mechanism

Disaggregation (DistServe, Splitwise) separates the two phases onto different pools: the prefill pool computes the prompt’s cache, and the decode pool generates the tokens. The KV cache is transferred from the prefill pool to the decode pool.

Definition 13.6.1 (Disaggregated serving). The prefill pool and the decode pool are separate deployments, each sized for its phase’s resource profile. The request flows: the prefill pool computes the cache, the cache is transferred to the decode pool, and the decode pool generates the tokens. The transfer’s cost is the cache’s size divided by the interconnect’s bandwidth.

13.6.3 The trade-off

The benefit: the phases are sized independently (the prefill pool for the compute, the decode pool for the bandwidth), and the interference is eliminated. The TTFT and the ITL are both improved, and the throughput is higher.

The cost: the cache’s transfer (a 100k-token cache at 0.5 MB per token is 50 GB, which is a substantial transfer), the two pools’ operational complexity, and the transfer’s latency (which adds to the TTFT).

The decision: disaggregate when the workload has long prompts (a large prefill cost) and a latency requirement that the colocated deployment cannot meet. The transfer’s cost is the constraint: a cache that is too large to transfer is a cache that must stay colocated.

13.6.4 The worked transfer

A 32k-token context with a 0.5 MB per-token cache is 16 GB. At an 800 Gbps interconnect (100 GB/s), the transfer is $16/100=0.16$ s. The prefill’s compute at 32k tokens is roughly 0.2–0.4 s on a modern GPU; the transfer’s 0.16 s is comparable, so the disaggregation’s benefit is the decode’s improvement minus the transfer’s cost. The arithmetic is the decision’s input, and the measurement (Part D) is the confirmation.

13.7 Expert parallelism for MoE

13.7.1 The mechanism

For a sparse MoE (Module 12, §12.2), the experts are distributed across devices, and a token’s computation requires the all-to-all communication: the token is sent to the devices holding its selected experts, and the results are gathered.

Definition 13.7.1 (Expert parallelism). The experts are partitioned across $E$ devices; each token’s $k$ selected experts may be on different devices. The dispatch sends the token to the experts’ devices; the combine returns the results. The communication is $O(k\cdot d_{\text{model}})$ per token, and the all-to-all’s bandwidth is the bottleneck.

13.7.2 The load balancing

The router’s imbalance (Module 12, §12.2.2) causes the devices to be unevenly loaded: the devices with the popular experts process more tokens, and the devices with the unpopular ones idle. The mitigation:

• The auxiliary loss (the load-balancing loss during training), which encourages the router to spread the tokens.
• The capacity factor (the maximum tokens per expert), which drops the overflow tokens or routes them to a second expert.
• The inference-time balancing, which monitors the load and adjusts the routing.

The imbalance’s effect is the throughput’s loss: a device at 100% utilisation is the bottleneck, and the others’ capacity is wasted. The harness’s throughput measurement (Part D) should check the per-device utilisation, not just the aggregate.

13.7.3 The decision

Expert parallelism is necessary when the model does not fit a single device (Module 12, §12.10’s BOM) and worth it when the interconnect is fast (NVLink within a node, InfiniBand across nodes). A model that fits a single device does not need it; a model that spans nodes needs the interconnect’s bandwidth to be the scaling’s constraint.

13.8 The engine comparison

13.8.1 The four engines

13.8.2 The choice

The choice is the workload’s and the hardware’s:

• The data-centre, the general workload: vLLM or SGLang. The prefix caching and the continuous batching are the agent’s requirements, and both engines have them.
• The structured-output-heavy workload: SGLang’s RadixAttention and the structured generation are the differentiators.
• The NVIDIA-only, the latency-critical: TensorRT-LLM’s kernels and the in-flight batching.
• The edge, the CPU, the local: llama.cpp’s GGUF and the CPU support.

The choice is the measurement’s, not the benchmark’s: the engines’ reported throughput numbers are on the vendors’ hardware and workloads, and the harness’s workload may differ. The lab’s replay (Part D) is the measurement.

13.8.3 Concept check

Why is the engine choice a measurement rather than a benchmark comparison?

Answer. Because the engines’ performance depends on the workload’s characteristics (the sequence length’s distribution, the prefix’s stability, the arrival’s burstiness) and the hardware (the GPU’s memory, the interconnect). A vendor’s benchmark is on the vendor’s workload, which is not the harness’s workload. The engine that is fastest on the vendor’s benchmark may be slower on the agent workload (a different prefix’s hit rate, a different sequence’s distribution). The replay (Part D) is the harness’s own measurement, and it is the selection’s input.

PART B — QUANTISATION

13.9 The formats

13.9.1 The weight-only and the weight-and-activation

Quantisation has two families.

Weight-only quantisation (AWQ, GPTQ, GGUF K-quants) stores the weights at a lower precision and dequantises them for the computation. The memory is reduced; the compute is unchanged (the dequantisation is the overhead). The accuracy’s loss is small for the 4-bit formats, larger for the 2-bit.

Weight-and-activation quantisation (FP8, FP4) stores both the weights and the activations at the lower precision, and the computation uses the lower-precision arithmetic. The memory and the compute are both reduced; the accuracy’s loss is the activations’ precision’s cost.

13.9.2 The formats

The GGUF formats are llama.cpp’s, and the K-quants (the _K_M variants) are the mixed-precision ones: the important layers (the attention’s, the first and last) are kept at a higher precision, and the rest are quantised more aggressively. The mixed precision is the reason the K-quants’ accuracy is better than the uniform 4-bit’s.

13.9.3 The choice’s decision

The choice is the memory’s constraint and the accuracy’s requirement:

• The data-centre, the memory is sufficient: FP8. The 2× saving is enough, and the accuracy’s risk is low.
• The data-centre, the memory is tight: FP4, with the accuracy’s measurement. The 4× saving is the reason, and the accuracy’s risk is the measurement’s subject.
• The edge, the memory is very tight: Q4_K_M or Q5_K_M, with the accuracy’s measurement. The GGUF is the format, and the K-quant’s mixed precision is the accuracy’s mitigation.
• The accuracy-critical, the memory is sufficient: FP16/BF16. The baseline, and the reference for the other formats’ measurements.

13.10 The accuracy’s cost

13.10.1 The short benchmark’s blind spot

The quantisation’s accuracy is usually measured on a short benchmark (a multiple-choice question set, a short-answer set). The measurement reports a small loss (1–3 points), and the conclusion is that the quantisation is safe.

The conclusion is wrong for the agent workload, for three reasons.

Reason 1 — The errors compound. The agent’s trajectory is a sequence of steps, and each step’s output conditions the next. A small per-step error rate compounds over the trajectory: a 1% per-step error rate over 100 steps is a $1-0.99^{100}=63$ chance of at least one error. The short benchmark measures one step; the agent measures 100.

Reason 2 — The tool-call format’s fragility. The quantisation can degrade the model’s ability to emit the exact tool-call format (the JSON’s syntax, the schema’s arguments). A short benchmark does not test the format; the agent’s parse-failure rate does (Module 12, §12.12).

Reason 3 — The long-context’s degradation. The quantisation’s error is larger at long contexts (the attention’s precision matters more when the context is large), and the agent’s contexts are long.

13.10.2 The evidence

The 2026 evidence (tier P, the community’s measurements; verify against the primary sources) reports that the quantisation’s degradation on the long-horizon agentic tasks is larger than on the short benchmarks: a 4-bit quantisation that loses 1–2 points on MMLU can lose 5–15 points on a long-horizon agent benchmark, and the loss is larger for the smaller models and the longer trajectories. The degradation is the compounding’s effect, and it is the measurement’s subject.

13.10.3 The measurement protocol

The measurement is the Module 5 design, applied to the quantisation:

1. Fix the harness (the same loop, the tools, the parser).
2. The conditions: FP16 (the baseline), FP8, FP4, Q4_K_M (as applicable).
3. The task set: the long-horizon tasks (the agent’s real workload), not the short benchmark. The task set must have the trajectories longer than 20 steps (the compounding’s regime).
4. The metrics: the success rate (the primary), the parse-failure rate, the pass^k (the reliability, Module 5, §5.5), and the cost per success.
5. The paired comparison: the same tasks, the same harness, the paired bootstrap (Module 5, §5.4).
6. The report: the degradation, the confidence interval, and the mechanism (the compounding, the format’s fragility, the context’s length).

The protocol’s discipline is the same as Module 12’s reproduction: the measurement is on the harness’s task set, not the vendor’s benchmark.

13.11 The quantisation’s decision

13.11.1 The decision’s structure

The decision is a trade-off between the memory’s saving and the accuracy’s cost:

(Part 0, §0.5, Theorem 0.5.7), where the quantisation affects the TPR (the accuracy’s loss) and the $C_g$ (the memory’s saving, the throughput’s gain).

The quantisation’s effect on the TPR is the measurement’s (§13.10.3); the effect on the $C_g$ is the throughput’s (a lower precision is more tokens per second, and a smaller memory is a larger batch). The trade-off is the arithmetic.

13.11.2 The worked decision

The baseline (FP16). TPR $=0.70$, $C_g=0.05$, $C_v=0.02$, $C_f=100$, $\pi=0.6$.

The FP8. The TPR is 0.69 (a 1-point loss); the $C_g$ is 0.03 (a 40% throughput’s gain). The cost per correct task: FP16 $=(0.05+0.02+100\times0.3\times0.4)/(0.7\times0.6)=(0.07+12)/0.42=28.7$. FP8 $=(0.03+0.02+12)/(0.69\times0.6)=(0.05+12)/0.414=29.1$. The FP8 is slightly worse (29.1 vs 28.7), because the TPR’s loss outweighs the cost’s saving at this failure’s cost.

The FP4. The TPR is 0.65 (a 5-point loss); the $C_g$ is 0.02. The cost: $(0.04+12)/(0.65\times0.6)=12.04/0.39=30.9$. The FP4 is worse (30.9 vs 28.7).

The lesson: the quantisation’s decision depends on the failure’s cost. At a high $C_f$ (100), the accuracy’s loss dominates, and the FP16 is the choice. At a low $C_f$ (say, 5), the cost’s saving dominates: FP16 $=(0.07+5\times0.3\times0.4)/(0.42)=(0.07+0.6)/0.42=1.60$; FP8 $=(0.05+0.6)/0.414=1.57$; FP4 $=(0.04+0.6)/0.39=1.64$. The FP8 is the best at the low failure’s cost. The quantisation is deployment-specific, and the decision’s input is the failure’s cost and the measured TPR.

13.11.3 Concept check

Why is a quantisation that loses 1 point on MMLU not necessarily safe for an agent?

Answer. Because the agent’s workload is not MMLU. The agent’s trajectory is long, and a small per-step error compounds: a 1% per-step error rate over 100 steps is a 63% chance of at least one error. The short benchmark measures one step; the agent measures 100. The quantisation also degrades the tool-call format’s exactness (the parse-failure rate) and the long-context’s attention, and both are the agent’s characteristics. The measurement must be on the long-horizon tasks, not the short benchmark.

PART C — CHAT TEMPLATES AND TOOL-CALL PARSERS

13.12 The chat template

13.12.1 The mechanism

The chat template is the model’s expected format for a conversation: the role markers, the special tokens, the system prompt’s placement, and the message’s separators.

Definition 13.12.1 (Chat template). A chat template is a function $T$ that maps a list of messages (the roles and the contents) to the model’s input string:

The template is defined in the model’s tokenizer_config.json (the chat_template field), as a Jinja2 template, and applied by the apply_chat_template method.

13.12.2 The failure modes

Three failure modes, each with a diagnostic.

Failure 1 — The wrong template. The harness uses a generic template (the OpenAI format, the Llama 2 format) instead of the model’s. The model’s input is out of distribution, and its behaviour degrades (a lower success rate, a higher parse-failure rate). Diagnostic: compare the harness’s formatted string to the tokenizer’s apply_chat_template output; they should match.

Failure 2 — The missing special tokens. The template omits the model’s special tokens (the BOS, the EOS, the role markers). The model’s generation starts in the wrong state, and it may not stop at the right token. Diagnostic: the token IDs of the formatted string; the special tokens should be present.

Failure 3 — The system prompt’s placement. The model expects the system prompt at a specific position (the start, or the first message). The harness places it elsewhere, and the model ignores it or misinterprets it. Diagnostic: the model’s behaviour with and without the system prompt; the difference should be the expected one.

13.12.3 The test

The template’s test is a unit test, not a judgement:

Test 13.12.2 (The template test). For a set of representative conversations (the system prompt only, the system + user, the multi-turn, the tool results), format the conversation with the harness’s formatter and with the tokenizer’s apply_chat_template, and assert that the strings are identical. The test fails if the harness’s formatter diverges.

The test is the CI’s (Module 9, §9.9), and it is the first of the four format tests (Module 12, §12.13.2).

13.13 The reasoning tokens

13.13.1 The two formats

The reasoning tokens are the model’s “thinking” — the chain of thought it generates before the answer. The models differ in how they expose the reasoning:

• The separated format: the reasoning is in a distinct block (the thinking... response tags, or a separate field), and the answer is the content after the block.
• The mixed format: the reasoning and the answer are in the same stream, with no structural separation.

13.13.2 The parser’s problem

The parser must distinguish the reasoning from the answer and the tool call:

• The separated format: the parser extracts the content after the “ (or the answer field), and it does not include the reasoning in the tool call’s arguments.
• The mixed format: the parser must heuristically separate the reasoning (the tool call is the structured part, and the rest is the reasoning). The heuristic is error-prone, and the failures are the silent kind (a tool call that includes part of the reasoning, or a reasoning that includes the tool call).

13.13.3 The budget’s interaction

The reasoning tokens consume the output’s budget (Module 4, §4.8.3). A model with a long reasoning and a tight budget may produce a truncated tool call. The harness’s budget must account for the reasoning’s expected length, and the parser must handle the truncation (a parse failure, not a silent wrong call).

13.13.4 The test

Test 13.13.2 (The reasoning test). Send a prompt that elicits the reasoning (a math problem, a multi-step task), and check: (1) the reasoning is separated (if the model’s format is separated); (2) the answer is correct and complete; (3) the tool call (if any) is parseable and does not include the reasoning.

The test is the second of the four format tests.

13.14 The stop tokens

13.14.1 The mechanism

The stop tokens are the tokens at which the model’s generation stops: the EOS token, the chat template’s end-of-turn token, the tool-call’s closing token.

13.14.2 The failure

A wrong stop token produces a truncated output: the model continues past the intended stop, or stops before it. The truncation is silent (the output looks plausible), and the failure is the parse or the tool call’s incompleteness.

Worked example. A model’s chat template ends the assistant’s turn with <|im_end|>. The harness uses <|endoftext|> (the model’s EOS, not the turn’s end). The model generates the answer and continues with a new turn’s content (the model’s next-turn prediction), and the harness’s output includes the extra content. The tool call’s arguments may be followed by the model’s next thought, and the parser fails.

13.14.3 The test

Test 13.14.2 (The stop-token test). Generate a known sequence with the harness’s stop tokens, and assert that the generation stops at the expected token. Then generate with the model’s native stop tokens, and assert that the two outputs are identical (up to the stop). A divergence is a stop-token mismatch.

The test is the third of the four format tests.

13.15 The tool-call serialisation

13.15.1 The two approaches

The native template. The model is trained with a tool-call format (the Qwen’s <tool_call>, the Llama’s <|python_tag|>, the Hermes’s <tool_call>), and the harness’s parser matches the training. The native template is the reliable approach, because the model’s output is in distribution.

The grammar-constrained decoding. The model is not trained with the format (or the format is unreliable), and the harness constrains the decoder to the schema (Module 2, §2.1.3). The grammar is the reliable approach, at the grammar tax’s cost (Module 2, §2.1.3’s decision rule).

13.15.2 The failure modes

Failure 1 — The format’s mismatch. The model emits the format $A$, and the harness parses the format $B$. The parse fails, or worse, the parser extracts a wrong call.

Failure 2 — The argument’s type. The model emits the arguments as a string, and the harness expects an object (or vice versa). The schema’s validation fails.

Failure 3 — The parallel calls. The model emits multiple calls in one response, and the harness’s parser handles only one. The extra calls are lost, or the parse fails.

Failure 4 — The nested arguments. The model emits a nested object, and the harness’s parser flattens it or rejects it.

13.15.3 The test

Test 13.15.2 (The tool-call test). For a set of tool-call prompts (a single call, a parallel call, a nested argument, an invalid call), assert: (1) the parser produces the expected calls; (2) the arguments’ types are correct; (3) the invalid call is rejected with a clear error; (4) the parse-failure rate is measured (Module 12, §12.12).

The test is the fourth of the four format tests.

13.15.4 Concept check

Why is the tool-call serialisation the most common silent failure when running an open model in a foreign harness?

Answer. Because the failure is not a crash; it is a plausible-looking output that the harness mis-parses. The model emits its native format (which the harness does not expect), the parser extracts a wrong call (or fails to extract one), and the model does not know it failed (the harness’s error is returned as an observation, and the model may or may not recover). The failure is silent because the output looks like a tool call, and the harness’s log shows a parse error, not a model error. The unit test (the four format tests) is the defence, and the parse-failure rate’s measurement is the check.

13.16 The unit-test suite

13.16.1 The four tests

The four format tests (§13.12.3, §13.13.4, §13.14.3, §13.15.3) are the unit-test suite:

13.16.2 The CI’s integration

The four tests are the CI’s (Module 9, §9.9): they run on every model’s version change (a new checkpoint, a new template) and every harness’s change (the formatter, the parser). A test failure blocks the deployment, and the failure’s diagnostic is the test’s name (the template, the reasoning, the stop, the tool call).

13.16.3 The model’s onboarding

The four tests are the onboarding checklist for a new model: a model that fails a test is a model that is not ready for the harness, and the failure is the fix’s target (the template’s correction, the parser’s extension, the stop-token’s configuration).

PART D — CAPACITY PLANNING

13.17 The agent workload’s characteristics

13.17.1 The four characteristics

The agent workload differs from the chatbot workload in four ways, and each affects the capacity plan.

Characteristic 1 — The long prefixes. The agent’s context is long (the system prompt, the tool schemas, the history), so the prefill’s cost and the cache’s size are large. The prefix caching (Module 3, §3.10) reduces the prefill’s cost, and the cache’s size is the batch’s constraint.

Characteristic 2 — The bursty arrivals. The agent’s tool calls arrive in bursts (a tool’s result triggers several calls; a slow tool stalls the agent), so the arrival’s burstiness $c_a^2$ is high (Module 11, §11.15.3).

Characteristic 3 — The variable service time. The agent’s outputs vary (a short tool call, a long reasoning trace), so the service’s variance $c_s^2$ is high.

Characteristic 4 — The multi-turn structure. The agent’s run is a sequence of requests (a tool call, a result, another call), and the requests are dependent (the next request depends on the previous result). The dependency means the latency is the sum of the steps, not the max, and the capacity must handle the per-step latency, not just the aggregate throughput.

13.17.2 The workload’s measurement

The workload is measured from the trace (Module 11, §11.1.3):

• The request’s length (the prompt’s tokens, the output’s tokens).
• The arrival’s rate and the burstiness (the inter-arrival’s variance).
• The service’s time and the variance.
• The prefix’s hit rate (the cache’s effectiveness).
• The dependency’s structure (the steps’ sequence).

The measurement is the replay’s input (Part D’s lab), and it is the capacity plan’s basis.

13.18 The throughput model

13.18.1 The two bounds

The serving’s throughput is bounded by two resources.

The compute bound. The GPU’s FLOPs divided by the model’s per-token FLOPs (the active parameters, Module 12, §12.1.2). The compute bound is the upper limit when the memory is sufficient.

The memory-bandwidth bound. The GPU’s memory bandwidth divided by the per-token memory access (the weights and the cache). The memory-bandwidth bound is the decode’s limit (Module 12, §12.4.2).

The throughput is the minimum of the two, and the binding one depends on the batch’s size: a small batch is memory-bandwidth-bound (the weights are read but the compute is underused); a large batch is compute-bound (the weights’ read is amortised over the batch).

Definition 13.18.1 (The throughput’s bound). For a batch of $B$ sequences, the throughput is

where the per-token FLOPs and bytes are the batch’s (the weights’ read is amortised, the cache’s read is per-sequence).

13.18.2 The batch’s effect

The batch’s size is the throughput’s lever, bounded by the cache’s memory (Module 12, §12.3.1). The optimal batch is the largest that fits the cache, subject to the latency requirement (the p95, §13.19).

Worked example. A model with 13B active parameters, 2 FLOPs per parameter per token (the forward pass’s rule of thumb): $26\times10^9$ FLOPs per token. An H100’s FP8 throughput is $\approx 2\times10^{15}$ FLOPs/s, so the compute bound is $2\times10^{15}/(26\times10^9)\approx77{,}000$ tokens/s per GPU. The memory-bandwidth bound: an H100’s 3.35 TB/s, and the per-token read is the weights (13B × 1 byte = 13 GB) plus the cache (0.5 MB), so the bound is $3.35\times10^{12}/(13\times10^9)\approx258$ tokens/s per sequence… this is per-sequence, not per-token. Let me redo: the decode’s per-step cost is the weights’ read (13 GB) plus the cache’s read (0.5 MB × B). The tokens per second is $3.35\times10^{12}/(13\times10^9 + 0.5\times10^6\times B)$. At $B=1$: $3.35\times10^{12}/(13.0005\times10^9)\approx258$ tokens/s. At $B=32$: $3.35\times10^{12}/(13\times10^9+16\times10^6)=3.35\times10^{12}/13.016\times10^9\approx257$ tokens/s — wait, the cache’s read is per-sequence, so the total is $13\text{ GB} + 0.5\text{ MB}\times32 = 13\text{ GB} + 16\text{ MB} \approx 13.016\text{ GB}$. The throughput is $3.35\times10^{12}/13.016\times10^9\approx257$ tokens/s per batch step, and each step produces 32 tokens (one per sequence), so the aggregate throughput is $257\times32\approx8{,}200$ tokens/s. The compute bound at $B=32$: the FLOPs per step is $26\times10^9\times32 = 8.32\times10^{11}$, and the GPU’s 2e15 FLOPs/s gives $2\times10^{15}/8.32\times10^{11}\approx2{,}400$ steps/s, so the throughput is $2{,}400\times32=76{,}800$ tokens/s. The binding bound is the memory-bandwidth (8,200), so the GPU is memory-bandwidth-bound at $B=32$. The lesson: the decode is memory-bandwidth-bound at the typical batch sizes, and the weights’ read is the dominant term; the batch’s size amortises the weights’ read, so the throughput rises with the batch until the compute bound binds.

I should present this more cleanly. Let me simplify the numbers and present the two bounds and the binding one.

13.18.3 The capacity’s arithmetic

The GPUs needed is the target throughput divided by the per-GPU throughput (at the chosen batch size):

Worked example. A target of 20,000 tokens/s, and a per-GPU throughput of 8,200 tokens/s (the memory-bandwidth bound at $B=32$): $N=3$ GPUs. The cache’s memory is the constraint: at the target’s context (say 16k tokens), the cache’s per-sequence size (0.5 MB × 16,000 = 8 GB) times the batch (32) is 256 GB, which exceeds a single GPU’s 80 GB — so the batch must be smaller or the GPUs’ memory must be pooled. The capacity plan’s arithmetic is the two constraints’ (the compute/bandwidth and the memory), and the binding one is the plan’s number.

13.19 The p95 latency

13.19.1 Kingman’s formula, applied

Restating Part 0, §0.8, Theorem 0.8.3:

where $W_q$ is the queueing delay, $\rho$ is the utilisation, $c_a^2$ and $c_s^2$ are the arrival’s and the service’s squared coefficients of variation, and $S$ is the service time.

The agent’s $c_a^2$ and $c_s^2$ are high (the bursty arrivals, the variable service), so the queueing delay is high, and the p95 is dominated by the queue.

13.19.2 The p95’s estimate

The p95’s approximation (for an M/G/1-like queue):

which is a rough estimate; the exact p95 depends on the distribution. The practical approach: measure the p95 in the replay (Part D’s lab), and use Kingman’s formula to explain it (the utilisation’s effect, the burstiness’s effect).

13.19.3 The levers

The p95’s levers, from Kingman:

• Reduce $\rho$: add capacity (more GPUs, a smaller batch), which is the cost’s trade-off.
• Reduce $c_a^2$: smooth the arrivals (a queue, a rate limit, a batch’s scheduler).
• Reduce $c_s^2$: limit the output’s variance (a max-tokens cap, a routing’s split).
• Reduce $\mathbb{E}[S]$: a faster model (a smaller active parameter count, a speculative decoding, a faster engine).

The levers’ order is the cost’s: the $\mathbb{E}[S]$‘s reduction is the model’s and the engine’s (a one-time change), the $\rho$‘s reduction is the capacity’s (an ongoing cost), and the $c^2$‘s reduction is the workload’s (a design’s change).

13.20 The capacity plan’s worked example

13.20.1 The workload

A coding agent’s workload:

• The requests per run: 40 (the tool calls plus the reasoning).
• The runs per hour: 100.
• The requests per hour: 4,000.
• The average prompt: 8,000 tokens (the prefix’s cache hit rate 0.8).
• The average output: 200 tokens.
• The service time: the prefill’s 0.1 s (with the cache’s hit) plus the decode’s 200 × 0.02 s = 4 s, so $\mathbb{E}[S]\approx4.1$ s.
• The target: the p95 latency ≤ 10 s.

13.20.2 The utilisation

The arrival rate $\lambda=4{,}000/3{,}600=1.11$ requests/s. The service rate $\mu=1/4.1=0.244$ requests/s per GPU. The utilisation per GPU: $\rho=\lambda/(N\mu)$. For $N=8$ GPUs: $\rho=1.11/(8\times0.244)=1.11/1.952=0.569$.

13.20.3 The queueing

The agent’s $c_a^2=3$ (the bursty arrivals) and $c_s^2=2$ (the variable service). Kingman: $\mathbb{E}[W_q]\approx(0.569/0.431)\times(5/2)\times4.1=1.32\times2.5\times4.1=13.5$ s. The total is $W_q+S=13.5+4.1=17.6$ s, which exceeds the 10 s target. The plan needs more GPUs or a lower $c_a^2$.

At $N=16$: $\rho=1.11/(16\times0.244)=0.284$; $\mathbb{E}[W_q]\approx(0.284/0.716)\times2.5\times4.1=0.397\times2.5\times4.1=4.07$ s; the total is $4.07+4.1=8.17$ s, which meets the target (the p95 is higher than the mean, so the margin should be larger; the plan would use $N=20$ for the margin).

13.20.4 The lesson

The capacity plan is the arithmetic of the utilisation, the burstiness, and the service time. The agent’s burstiness ($c_a^2=3$) is the dominant factor: a chatbot’s $c_a^2=1$ would give $\mathbb{E}[W_q]=(0.397)\times(3/2)\times4.1=2.44$ s at $N=16$, and the total would be 6.5 s. The agent’s burstiness is the p95’s penalty, and the plan’s number is the burstiness’s consequence.

13.21 The replay

13.21.1 The method

The replay is the lab’s method: capture a real agent workload’s trace (the requests’ prompts, the outputs’ lengths, the arrival’s times), and replay it against the serving stack, measuring the throughput, the p95, and the parse-failure rate.

Procedure 13.21.1 (The replay).

1. Capture the trace. The requests’ prompts, the outputs’ lengths, the arrival’s times, and the harness’s model (the sampling’s parameters).
2. Replay the trace against the serving stack, with the same arrival’s pattern (the bursty times, not the smoothed ones).
3. Measure the throughput (the tokens/s), the TTFT and the ITL (the p50 and the p95), and the parse-failure rate.
4. Compare the engines and the models on the same trace.
5. Report the results with the confidence intervals (the Module 5 discipline).

13.21.2 The replay’s value

The replay is the harness’s own measurement of the stack’s performance, and it is the selection’s input (Module 12, §12.9’s dossier’s field 5, extended). The vendor’s benchmark is on the vendor’s workload; the replay is on the harness’s. The difference is the selection’s value.

13.21.3 The replay’s limits

The replay’s limits: (1) the trace is a sample, and the workload’s distribution may shift; (2) the replay does not capture the tool’s latency (the tools are stubbed or mocked); (3) the replay’s model is the same as the trace’s, so the model’s version’s change is not captured. The limits are the replay’s scope, and the report should state them.

LAB 13 — SERVE, MEASURE, REPLAY

13.22.1 Task specification

Serve one MoE model and one dense model with two different engines (e.g., vLLM and SGLang, or vLLM and TensorRT-LLM), and characterise their performance under a replayed agent workload.

13.22.2 The models

• The MoE: one of the Module 12 families’ MoE models (e.g., a Qwen MoE, a DeepSeek MoE, a GLM MoE) that fits the available hardware.
• The dense: a dense model of a comparable active-parameter count (so the compute is comparable, and the MoE’s memory-for-compute trade-off is the difference).

13.22.3 The engines

Two engines, from the four of §13.8.1. The choice should be justified (the workload’s and the hardware’s), and the engines’ configurations should be stated (the batch size, the prefix caching, the speculative decoding).

13.22.4 The workload

Capture a real agent workload’s trace (the lab’s own harness’s runs, or a public trace). The trace must include the prompts, the outputs’ lengths, and the arrival’s times (the bursty pattern).

13.22.5 The metrics

13.22.6 The hypotheses

State before running:

• H1. The continuous batching’s throughput is higher for the agent workload than the naive batching’s, and the gain is larger for the heterogeneous outputs (the long reasoning vs the short tool calls).
• H2. The prefix caching reduces the TTFT for the repeated prefixes, and the reduction is larger for the engine with the better cache implementation.
• H3. The MoE’s throughput is higher than the dense model’s at the same active-parameter count (the memory-for-compute trade-off’s benefit), and its memory is higher.
• H4. The parse-failure rate is non-zero for at least one model-engine pair, and the rate is the model’s format’s (not the engine’s).
• H5. The p95 latency is dominated by the queueing (the utilisation’s effect), and the burstiness’s $c_a^2$ is the largest contributor.

13.22.7 Acceptance criteria

13.22.8 What good looks like

A good lab report has five properties.

The workload is real. The trace is a captured workload, not a synthetic one, and the bursty arrivals are preserved. A synthetic Poisson workload would miss the agent’s burstiness, and the p95 would be under-estimated.

The parse-failure rate is measured per model-engine pair. The rate is the model’s format’s, and the engine does not change it (the engine serves the model’s output; the harness parses it). A report that measures the rate only for one pair has missed the model’s contribution.

The prefix cache’s hit rate is reported. The hit rate is the layout’s effectiveness (Module 3, §3.11), and it is the TTFT’s and the cost’s input. A report that omits the hit rate cannot explain the TTFT’s difference between the engines.

The capacity plan is computed. The lab’s result is the per-GPU throughput, and the plan is the GPUs needed for a target. A report that stops at the throughput has not answered the deployment’s question.

The decision is the pair’s. The model and the engine are a pair, and the decision is the pair’s, not the model’s or the engine’s separately. A report that recommends a model without the engine (or vice versa) has missed the interaction.

MODULE 13 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E13.1. A batch has 10 requests: 9 generate 100 tokens each, 1 generates 1,000 tokens. Compute the naive batching’s effective batch size and the continuous batching’s throughput gain.

S13.1. The naive batch runs for 1,000 steps. The effective work is $9\times100+1{,}000=1{,}900$ tokens over 1,000 steps, so the effective batch size is 1.9. The continuous batching, once the 9 short requests finish at step 100, admits new requests, so the effective batch size stays near 10. The gain is $10/1.9\approx5.3\times$. The lesson: the heterogeneity’s penalty is the naive batching’s effective batch size, and the continuous batching’s gain is the penalty’s removal.

E13.2. A speculative decoding has a draft of 4 tokens, an acceptance rate of 0.8, and the draft’s cost is 1/10 of the target’s. Compute the expected tokens per target pass and the speedup.

S13.2. Expected tokens $=(1-0.8^5)/(1-0.8)=(1-0.32768)/0.2=0.67232/0.2=3.36$. The speedup: the combined cost per target pass is $c_{\text{target}}+4\times0.1\,c_{\text{target}}=1.4\,c_{\text{target}}$, and the tokens are 3.36, so the speedup is $3.36/1.4=2.4\times$. The lesson: the speedup is the expected tokens divided by the combined cost, and the draft’s cost is the limit.

E13.3. A model has 80 layers, 8 KV heads, a 128 head dimension, and FP16. Compute the KV cache at 16k and at 64k. If the GPU has 80 GB and the weights are 40 GB, what is the maximum context that fits?

S13.3. At 16k: $2\times80\times8\times128\times16{,}384\times2=2\times80\times8\times128\times32{,}768$. Compute: $80\times8=640$; $640\times128=81{,}920$; $81{,}920\times32{,}768\approx2.684\times10^9$; $\times2\times2=1.074\times10^{10}$ bytes $\approx10.7$ GB. At 64k: four times, $\approx42.9$ GB. The GPU has $80-40=40$ GB for the cache and the activations, so the maximum context is between 16k (10.7 GB) and 64k (42.9 GB), roughly 48k before the activations’ overhead. The lesson: the cache’s linear growth is the context’s constraint, and the arithmetic is the fit’s decision.

E13.4. A quantisation’s FP8 loses 1 point of TPR (0.70 → 0.69) and reduces the generation’s cost by 40% (0.05 → 0.03). The verification’s cost is 0.02, the failure’s cost is 10, and the base rate is 0.6. Is the FP8 the better choice?

S13.4. FP16: $(0.05+0.02+10\times0.05\times0.4)/(0.70\times0.6)=(0.07+0.2)/0.42=0.27/0.42=0.643$. FP8: $(0.03+0.02+0.2)/(0.69\times0.6)=(0.25)/0.414=0.604$. The FP8 is better (0.604 vs 0.643), because the generation’s cost’s saving outweighs the TPR’s 1-point loss at the low failure’s cost (10). At the high failure’s cost (100), the FP16 would win (as in §13.11.2). The lesson: the quantisation’s decision depends on the failure’s cost, and the arithmetic is the decision’s basis.

E13.5. A chat template test fails: the harness’s formatted string has <|im_start|>user while the tokenizer’s apply_chat_template has <|start_header_id|>user<|end_header_id|>. What is the failure, and what is the fix?

S13.5. The failure is the wrong template: the harness uses a generic format (the ChatML <|im_start|>) while the model expects a different format (the Llama 3 <|start_header_id|>). The model’s input is out of distribution, and its behaviour degrades. The fix: use the tokenizer’s chat_template (the apply_chat_template method) as the harness’s formatter, and the template test (Test 13.12.2) as the CI’s check. The lesson: the template is the model’s, not the harness’s generic one, and the test catches the divergence.

E13.6. An agent workload has $\lambda=2$ requests/s, $\mathbb{E}[S]=3$ s, $c_a^2=2.5$, $c_s^2=1.5$, and 10 GPUs with $\mu=1/3$ requests/s each. Compute the utilisation and the expected queueing delay. Does it meet a 10 s p95 target (roughly)?

S13.6. The utilisation: $\rho=2/(10\times0.333)=2/3.33=0.6$. Kingman: $\mathbb{E}[W_q]=(0.6/0.4)\times((2.5+1.5)/2)\times3=1.5\times2\times3=9$ s. The total is $9+3=12$ s, which exceeds the 10 s target (and the p95 is higher than the mean). The plan needs more GPUs (e.g., 13: $\rho=2/(13\times0.333)=0.462$; $\mathbb{E}[W_q]=(0.462/0.538)\times2\times3=0.859\times6=5.15$ s; the total is 8.15 s, meeting the target). The lesson: the burstiness ($c_a^2=2.5$) doubles the queueing relative to a Poisson workload ($c_a^2=1$), and the capacity’s plan must account for it.

E13.7. A model’s tool-call parse-failure rate is 8% with the native template and 3% with the grammar-constrained decoding, but the grammar adds a 5% cost (the grammar tax). The verifier’s cost per correct task without the parse failures is 2.0. Which is better?

S13.7. The native template: the effective cost is $2.0/(1-0.08)=2.0/0.92=2.17$. The grammar: the parse failure is 3%, so the cost is $2.0/(1-0.03)=2.06$, and the grammar tax adds 5%: $2.06\times1.05=2.16$. The two are nearly equal (2.17 vs 2.16), so the decision is marginal. The lesson: the grammar’s benefit (the parse failure’s reduction) and its cost (the grammar tax) can cancel, and the measurement is the decision’s basis; a small change in either would flip the decision.

E13.8. A capacity plan says 16 GPUs meet the throughput target, but the p95 is 12 s against a 10 s target. What are the levers, and which would you try first?

S13.8. The levers (from Kingman): (1) reduce $\rho$ (more GPUs — 20 would reduce the queueing); (2) reduce $c_a^2$ (smooth the arrivals — a queue, a rate limit); (3) reduce $c_s^2$ (limit the output’s variance — a max-tokens cap, a routing’s split); (4) reduce $\mathbb{E}[S]$ (a faster model — speculative decoding, a smaller active parameter count). The first to try depends on the cost: the $\mathbb{E}[S]$‘s reduction is a one-time change (a faster engine, speculative decoding), the $\rho$‘s reduction is an ongoing cost (more GPUs), and the $c^2$‘s reduction is a workload’s design (a rate limit may be free). The cheapest first: the engine’s or the speculative decoding’s $\mathbb{E}[S]$‘s reduction, then the rate limit’s $c_a^2$‘s reduction, then the GPUs’ $\rho$‘s reduction. The lesson: the p95’s levers are ordered by the cost, not by the effect.

EXIT TEST FOR MODULE 13

You are ready for Module 14 when you can, without notes:

1. State the four serving-stack functions and the two metrics, and explain the agent workload’s three stresses.
2. Explain continuous batching, the throughput’s gain, and the latency’s trade-off.
3. Explain paged attention, the fragmentation’s problem, and the cache’s capacity’s arithmetic.
4. Explain the prefix caching, the hit rate’s determinants, and the layout’s lever.
5. Compute the speculative decoding’s expected tokens and speedup, and state the condition.
6. Explain the disaggregated prefill/decode, the transfer’s cost, and the decision.
7. Explain the expert parallelism and the load balancing.
8. Compare the four engines and state why the choice is a measurement.
9. Distinguish the weight-only and the weight-and-activation quantisation, and state the formats’ trade-offs.
10. Explain the short benchmark’s blind spot (the compounding, the format’s fragility, the long context) and the measurement’s protocol.
11. Apply the quantisation’s decision (the cost per correct task) and the failure’s cost’s role.
12. Define the chat template, the three failure modes, and the template’s test.
13. Explain the reasoning tokens’ formats, the parser’s problem, and the budget’s interaction.
14. Explain the stop tokens’ failure and the test.
15. Explain the tool-call serialisation’s approaches, the failure modes, and the test.
16. State the four format tests and the CI’s integration.
17. State the agent workload’s four characteristics and the throughput’s two bounds.
18. Compute the capacity plan (the utilisation, Kingman’s queueing, the GPUs) and the p95’s levers.
19. Describe the replay’s method, value, and limits.

End of Module 13. Say “next” to receive Module 14 in full.