Weeks 4–5. Prerequisite: Modules 1–2 and Part 0, §§0.3, 0.8.

3.0 Module map and objectives

Module 1 established that the context $c_t=\phi(h_t)$ is a lossy, non-Bayesian surrogate for the belief $b_t$, and that the harness’s observation function determines what the model can know. Module 2 treated the action half of the interface. This module treats the observation half: how the context is assembled, what it costs, what it loses, and how to measure the loss.

The module has six parts and one lab.

By the end of the module you will be able to do seven things the rest of the curriculum assumes:

1. Measure the effective context length of a model on your task, as distinct from its nominal window, and report the gap.
2. Specify a compaction policy formally as a map $\kappa:\mathcal{H}\to\mathcal{C}$, and bound its information loss using the data-processing inequality.
3. Compare truncation, summarisation, and structured extraction on the same trajectory, and choose among them using downstream success rather than summary quality.
4. Construct an oracle summary and use it to measure compaction inefficiency as the gap between the policy’s success and the oracle’s.
5. Design an external-memory architecture (MemGPT-style paging, scratchpad, or evolving playbook) and justify the choice against the task’s memory requirements.
6. Decide when retrieval should be a pipeline stage and when it should be an agent action, using a decision-theoretic criterion.
7. Lay out a context for cache efficiency, compute the hit rate, and estimate the cost saving.

Everything is self-contained. Where Part 0 or Module 1 is used, the result is restated in the form needed.

PART A — THE CONTEXT AS A SCARCE, ADVERSARIALLY EXPOSED RESOURCE

3.1 Positional degradation and distractor sensitivity

3.1.1 The Lost in the Middle result

Liu et al. (arXiv:2307.03172, evidence tier R: independently reproduced) established a result that has shaped every serious context-management design since: when a language model must use information from a long context, its performance depends on where in the context the information appears. The curve is U-shaped. Performance is highest when the relevant information is at the beginning of the context, next-highest when it is at the end, and lowest when it is in the middle.

The precise experimental design matters, because the result is often overstated.

Setup. The model is given a question, a set of documents, and one document that contains the answer. The position of the answer-bearing document is varied across the context (first, second, …, last). The model must answer the question. Performance is measured as accuracy.

Result. Accuracy is high when the answer document is first (the “primacy” position), high when it is last (the “recency” position), and substantially lower when it is in the middle. The shape is U-like, and the depth of the trough depends on the model, the total context length, and the difficulty of the question.

Why it happens. Two mechanisms are proposed, and both are plausible.

Attention dilution. A transformer’s attention is spread over all tokens in the context. With more tokens, the attention weight on any single token is smaller. Information at the beginning benefits from the causal mask’s structural bias and from the model’s training on instruction-following formats (where the instruction is first). Information at the end benefits from recency — the most recent tokens are closest to the generation point. Information in the middle has neither advantage.

Training distribution. Models are trained on documents where the key information tends to be at the start (abstracts, topic sentences) or the end (conclusions, summaries). The middle is under-represented in the training distribution, so the model has less practice attending to it.

3.1.2 The harness implications

The U-curve is not a curiosity. It has three direct consequences for context layout.

Consequence 1 — The system prompt and the current instruction must be at opposite ends. The system prompt is the stable instruction; it goes first. The current step’s instruction (what to do now) goes last. Both are in high-attention positions. This is why every well-designed harness places them there, and why a harness that puts the current instruction in the middle of a long history is fighting the model’s attention.

Consequence 2 — Retrieved content must be placed near the instruction, not buried in the middle. If the agent retrieves a document to answer the current question, the document should be adjacent to the question, at the end of the context. A retrieval-augmented harness that places retrieved documents in a block before the history, with the question at the very end, is putting the documents in the trough.

Consequence 3 — The middle of a long context is where information goes to be ignored. This is the operational form of the loss. If your compaction policy retains the middle and drops the ends, it is retaining the least useful part. The common truncation policy — drop the oldest (the beginning) and keep the most recent (the end) — is actually well-matched to the U-curve: it preserves the high-attention regions and drops the low-attention ones. This is not a coincidence; it is why oldest-first truncation works as well as it does.

3.1.3 Distractor sensitivity

A separate finding, often conflated with the U-curve but distinct: the presence of irrelevant content in the context degrades performance on the relevant content. This is the distractor effect. A context that contains the relevant document plus twenty irrelevant ones produces worse answers than the same context with only the relevant document.

The mechanism is the same attention dilution, but the operational implication is different. The U-curve says where to put information. Distractor sensitivity says how much irrelevant information to tolerate. A harness that loads twenty files to answer a question about one is not merely wasting tokens; it is actively degrading the model’s ability to use the one that matters.

Proposition 3.1.1 (The distractor cost is superlinear in irrelevant content). For a fixed relevant content, the degradation from irrelevant content grows faster than linearly in the amount of irrelevant content, because each irrelevant token dilutes attention on every relevant token and the dilution compounds with the number of attention heads and layers.

Status. This proposition is a hypothesis, not a theorem. The empirical evidence (Liu et al., and follow-up work) shows degradation with distractors; the superlinearity is suggested by the attention-dilution mechanism but has not been cleanly established. Treat it as a design heuristic: when in doubt, include less. The cost of omitting a marginally relevant document is usually smaller than the cost of including ten irrelevant ones.

3.1.4 Concept check

Why does oldest-first truncation work as well as it does, given that it is a crude positional policy?

Answer. Because it drops content from the beginning of the context — which is a high-attention position only for the system prompt and the original instruction, both of which are explicitly preserved — and retains the recent content, which is in the recency position. The middle, which the truncation also drops over time, is the trough where information is least likely to be used anyway. The policy is crude, but it aligns with the attention curve: it keeps what the model attends to and drops what it ignores. The failure mode is the scrolled-out-action construction of Module 1, §1.5.2, where the decisive fact is old — and that failure is precisely the case where a purely positional policy fails.

3.2 Effective versus nominal context

3.2.1 The distinction

Definition 3.2.1 (Nominal context length). The maximum number of tokens the model can accept, as advertised by the vendor or specified in the model card.

Definition 3.2.2 (Effective context length). The context length up to which the model’s performance on the task is within a stated tolerance of its performance at a short reference length. Formally, for a task distribution and a metric $M$, the effective length at tolerance $\epsilon$ is

where $M(\ell)$ is the metric at context length $\ell$ and $\ell_0$ is a short reference length.

The two are different, and the gap is large. Models advertised at 1M tokens often have effective lengths of 100k–200k on retrieval-and-reasoning tasks, and effective lengths of 20k–50k on tasks that require integrating information across the whole context. The gap is task-dependent, model-dependent, and — crucially — not disclosed by the vendor.

3.2.2 Why the gap exists

Three mechanisms, each compounding.

Training distribution. A model is trained on sequences of a certain length distribution. Beyond that distribution, it is extrapolating, and its attention patterns are less reliable. Positional encodings (RoPE and its variants) are often extended at inference time (e.g., by scaling the base frequency), and the extension is approximate: the model handles long contexts worse than it handles the lengths it was trained on.

Attention dilution. As in §3.1. The mechanism does not stop at the training length; it gets worse.

Retrieval-and-integration difficulty. A task that requires finding one fact in a long context (a “needle in a haystack”) is easier than a task that requires integrating facts from many positions. The nominal context length is typically measured on the needle task; the effective length on the integration task is much shorter. This is the most important distinction and the most often missed: a model that can retrieve a needle from 1M tokens cannot necessarily reason over 1M tokens.

3.2.3 Measuring effective length

The measurement is a task-specific calibration. The procedure:

1. Choose a task distribution representative of your workload.
2. Choose a metric (success rate, exact match, F1 — whatever your verifier produces).
3. Choose a length ladder (e.g., 2k, 8k, 32k, 128k, 512k tokens), constructed by padding the context with irrelevant content (to isolate the length effect from the content effect) or by using naturally long instances.
4. Measure the metric at each length, with enough instances per length to get a confidence interval (Part 0, §0.4.4).
5. Report $L_{\text{eff}}(\epsilon)$ for a stated tolerance, and the full curve, not just a point.

The full curve is more useful than the point, because the shape tells you where the degradation begins. A model that is flat to 100k and then falls sharply has a different operational profile from a model that degrades gradually from 10k.

Worked example. A model advertised at 1M tokens is evaluated on a code-editing task at lengths 4k, 16k, 64k, 256k, 1M. Success rates: 0.72, 0.71, 0.68, 0.51, 0.29. At $\epsilon=0.05$, $L_{\text{eff}}(0.05)=64k$. The model’s effective length is 64k, not 1M. A harness that routinely constructs 200k-token contexts is operating in the degraded regime, and the degradation is invisible unless it is measured. This is the practical argument for compaction: the nominal window is not the useful window.

3.2.4 The million-token trap

The current generation of open-weight models advertises million-token contexts (Module 12). The advertisement is technically true — the model will accept the tokens — and operationally misleading, because the effective length is much shorter, and because the cost of the extra tokens is real while the benefit is often negative (distractor sensitivity).

The operational rule: treat the nominal context as a hard ceiling, and the effective context as a soft budget. Design the harness to keep the context within the effective length, using compaction and retrieval to fit. The nominal length is a safety margin for occasional long inputs, not a target.

3.2.5 Concept check

A vendor advertises a 1M-token context. Your task requires integrating facts from across a 200k-token repository. Can you rely on the advertised number?

Answer. No. The advertised number is the nominal length. The effective length on an integration task is typically far shorter — often 10–20% of nominal — because integration requires attending to many positions simultaneously, and attention dilution and training-distribution mismatch compound. You must measure the effective length on your task, using your metric, before designing the harness around a 200k-token context. If the effective length is 64k, the harness must retrieve and compact to fit within 64k, not assume that 200k is usable.

PART B — COMPACTION AS LOSSY PROJECTION

3.3 The formal object

3.3.1 Definition and the loss bound

Definition 3.3.1 (Compaction policy). A compaction policy is a map

from histories to compacted contexts, such that $|\kappa(h)|\le|\phi(h)|$ for all $h$ (the compacted context is no longer than the full context). The context presented to the model is $c_t=\kappa(h_t)$, where $\kappa$ may be composed with the formatter: $c_t=\phi_{\text{fmt}}(\kappa(h_t))$.

The composition with the formatter matters: $\kappa$ is the content reduction, and the formatter is the presentation. The information loss is in $\kappa$, not in the formatter (the formatter is deterministic and, in the well-designed case, injective).

Proposition 3.3.2 (The compaction loss bound). For any compaction policy $\kappa$,

with equality iff $\kappa$ is sufficient for the belief on the reachable histories.

Proof. $b_t\to h_t\to c_t$ is a Markov chain (the belief is a function of the history; the compacted context is a function of the history). Apply the data-processing inequality (Part 0, Theorem 0.3.4). $\square$

Corollary 3.3.3 (The loss is a divergence). If the model’s downstream belief is $\hat b_t=\Pr(b_t\mid c_t)$ and the true belief is $b_t$, the expected log-loss penalty is $D_{\mathrm{KL}}(b_t|\hat b_t)$, which is non-negative and zero iff $c_t$ is sufficient.

This is the formal justification for measuring compaction loss through downstream task success rather than through summary quality. Summary quality is a proxy for $I(b_t;c_t)$ — how much of the belief is retained — but the task success is the decision-relevant quantity: it is a monotone function of the log-loss under a proper scoring rule (Part 0, Proposition 0.3.8). A summary can be judged excellent by a human and still drop the one fact the task needs; the task success will reveal this, and the human judgement will not.

3.3.2 The projection error

Definition 3.3.4 (Projection error). For a task distribution $\mathcal{D}$ and a compaction policy $\kappa$, the projection error is

where $S(\cdot)$ is the task success rate and $\kappa_{\text{full}}$ is the identity (no compaction). It is the downstream success lost to compaction.

The projection error is the operational measure of the loss. It is non-negative by the DPI (compaction cannot help, in the information-theoretic sense — though in practice it can help by removing distractors, which is a separate effect). It is task-specific: a compaction policy that loses 5 points on one task may lose 20 on another, because the tasks differ in how much of the history is decision-relevant.

Proposition 3.3.5 (The projection error decomposes). If the compaction has three components — truncation, summarisation, and formatting — and each is applied independently, then to first order

where each $\varepsilon$ is the error from applying that component alone. The decomposition is approximate because the components interact (a summary of truncated content loses more than the sum of the losses), but it is a useful diagnostic: it tells you which component is doing the damage.

3.3.3 The two failure modes of compaction

Compaction fails in two distinct ways, and they require different remedies.

Failure mode 1 — Dropping the decisive fact. The compaction removes a piece of information that the task needs. This is the truncation loss of Module 1, §1.5.3. The remedy is retrieval or externalisation: if the fact can be reacquired, the model can recover; if it cannot, the compaction must be redesigned to preserve it.

Failure mode 2 — Corrupting the retained fact. The compaction keeps the information but distorts it. A summary that says “the function returns a list” when the original said “returns a list of integers, or None on failure” has corrupted the fact. This is the summarisation loss. The remedy is a more faithful summarisation policy, or a structured extraction that preserves the exact form.

The two modes are often conflated in practice (“the summary lost information”), but they have different remedies, and a diagnostic that distinguishes them is more useful than one that does not. The lab measures both: dropping is detected by comparing the retained content to the original; corrupting is detected by comparing the model’s behaviour on the retained content to its behaviour on the original.

3.4 Three compaction policies

3.4.1 Truncation

Definition. $\kappa_{\text{trunc}}$ retains a window of the history: the system prompt, the original instruction, and the most recent $k$ messages. Older messages are dropped.

Properties.

• Loss: total for dropped content. The model has no access to it.
• Cost: minimal. Truncation is $O(1)$ per step.
• Failure mode: the scrolled-out-action construction (Module 1, §1.5.2). If the decisive fact is old, truncation drops it.
• Match to the attention curve: good. Truncation preserves the high-attention ends and drops the low-attention middle.
• Recoverability: none, unless the dropped content can be reacquired by a tool call. If the model can re-read a file, truncation is safe; if the fact came from a one-time observation (a tool output that cannot be reproduced), truncation is lossy.

3.4.2 Summarisation

Definition. $\kappa_{\text{sum}}$ replaces a block of history with a summary generated by an LLM: $\kappa_{\text{sum}}(h_{1:t})=\text{LLM}(\text{"summarise this history"}|h_{1:t})$, possibly with a target length.

Properties.

• Loss: lossy in both modes — it can drop facts and corrupt them. The loss is semantic, not positional: the summariser decides what matters, and its decision may not match the task’s.
• Cost: one model call per compaction, which is itself a token cost. The cost must be amortised over the steps the summary serves.
• Failure mode: the summariser’s judgement of relevance may be wrong; it may hallucinate facts not in the history; it may compress a fact into a form the downstream model cannot use.
• Match to the attention curve: neutral. The summary is placed where the history was, so it can be anywhere.
• Recoverability: partial. The summary may retain enough for the model to know it needs to reacquire the full content, if the harness provides a retrieval tool.

The summarisation quality trap. The natural way to evaluate a summary is to ask a human (or a model) “is this a good summary?” This is the wrong measure. A summary can be excellent by that standard and still lose the task. The correct measure is downstream success (Definition 3.3.4). The lab is designed to make this point empirically: you will produce summaries that a human would rate highly and that lose 15 points of task success, and summaries that look mediocre and lose nothing.

3.4.3 Structured extraction

Definition. $\kappa_{\text{ext}}$ replaces a block of history with a structured record of the facts it contained: key-value pairs, a task-relevant schema, a list of decisions and their rationales. The schema is designed for the task, not for general readability.

Properties.

• Loss: lossy for facts not covered by the schema; faithful for facts that are. The loss is designed — the schema encodes what the task needs, so the extraction preserves exactly that and discards the rest.
• Cost: one model call per extraction, plus the schema design cost (human effort, paid once).
• Failure mode: the schema is wrong — it omits a fact the task needs, or it imposes a structure the task does not have. This is a design failure, and it is detectable: the projection error on a held-out task set reveals it.
• Match to the attention curve: neutral.
• Recoverability: good for extracted facts (they are in a stable, structured form); poor for unextracted facts (they are gone).

The schema design problem. Structured extraction is only as good as its schema. The schema should be derived from the task: what does the model need to know to take the next action? For a code-editing task, the schema might be: the files read and their relevant contents, the tests run and their outcomes, the edits made and their rationale. For a research task, the schema might be: the questions asked, the sources found, the claims supported. The schema is a hypothesis about what matters, and the lab tests it.

3.4.4 The comparison

The table is a starting point, not a decision. The choice depends on the task, and the lab is how you make it for your task.

3.4.5 Concept check

Why is “the summary looks good” a misleading evaluation criterion?

Answer. Because summary quality measures the summary against a human’s or a model’s judgement of what a good summary is, which is not the same as what the task needs. A summary can be well-written, faithful, and comprehensive by that standard, and still omit the one fact that determines the next action — because the summariser did not know which fact was decisive. The task success measures the summary against the decision it supports, which is the only thing that matters. The lab is designed to show the gap empirically.

3.5 The oracle upper bound

3.5.1 Definition

Definition 3.5.1 (Oracle summary). An oracle summary for a trajectory $h$ and a task $x$ is a compaction $\kappa_{\text{oracle}}$ that retains exactly the information needed to complete the task — no more, no less. It is constructed with knowledge of the task and the correct answer, so it is not available at run time. It is an upper bound on what any compaction policy can achieve.

Proposition 3.5.2 (The oracle bounds the achievable success). For any compaction policy $\kappa$,

The gap

is the compaction inefficiency: the success that the policy leaves on the table relative to a compaction that knows what matters.

3.5.2 Constructing the oracle

The oracle is constructed by a human (or a strong model with the answer) who reads the trajectory, identifies the facts the task requires, and writes a compaction that contains exactly those facts. The construction is labour-intensive, which is why it is an upper bound rather than a deployable policy. Its value is diagnostic: it tells you how much room there is.

Three uses of the oracle.

Use 1 — Bounding the achievable gain. If the oracle achieves 0.85 and your policy achieves 0.82, there is little room; the policy is near-optimal for this task, and further effort should go elsewhere. If the oracle achieves 0.85 and your policy achieves 0.60, there is 25 points of room, and the policy is the problem.

Use 2 — Diagnosing the loss. Compare the oracle’s retained facts to the policy’s retained facts. The facts the oracle kept and the policy dropped are the dropped decisive facts (failure mode 1). The facts both kept but the policy distorted are the corrupted facts (failure mode 2). This decomposition tells you which failure mode dominates and therefore which remedy to apply.

Use 3 — Calibrating the policy’s relevance model. The oracle’s selection of facts is a labelled example of what matters. If the policy is an LLM summariser, the oracle’s facts can be used to evaluate the summariser’s relevance judgements, and the errors can inform a better prompt or a better schema.

3.5.3 The inefficiency is task-dependent

The compaction inefficiency $\gamma(\kappa)$ is not a property of the policy alone. It depends on the task. A policy that works well on a task where the decisive fact is recent (truncation suffices) may work poorly on a task where it is old (truncation drops it). The lab measures the inefficiency across a task set, and the interesting result is usually the variance: the policy is near-oracle on some tasks and far from it on others, and the pattern of where it fails is the diagnostic.

3.5.4 Concept check

The oracle summary is not deployable. Why construct it?

Answer. Because it is the only way to know how much room there is. Without it, a policy that achieves 0.60 on a task where the ceiling is 0.62 looks mediocre but is near-optimal; a policy that achieves 0.60 on a task where the ceiling is 0.90 looks identical but is leaving 30 points on the table. The oracle distinguishes the two cases, and the distinction determines whether the next investment should be in compaction or elsewhere. It is a diagnostic instrument, not a product.

PART C — EXTERNALISED MEMORY

3.6 MemGPT: virtual context management

3.6.1 The architecture

MemGPT (Packer et al., arXiv:2310.08560, evidence tier R) applies the operating-system idea of virtual memory to the context window. The architecture has two tiers.

Main context. The tokens currently in the model’s window. This is the analogue of RAM: fast, limited, expensive.

External context. A store outside the window — a database, a file system, a vector store. This is the analogue of disk: slow, large, cheap.

The agent manages the two tiers through function calls: it can page in content from external to main (a read/retrieve operation), page out content from main to external (a write/store operation), and search the external store. The model decides when to page, based on the task.

3.6.2 Why this is a harness design, not a model capability

The key insight of MemGPT is that the paging is done by the model, through tool calls, not by the harness through a fixed policy. The model reads a file, decides it needs to remember it, and writes a summary to external memory; later, it searches external memory when it needs the fact. This makes memory management an agent decision rather than a harness policy.

The consequence for the harness: it must expose the paging operations as tools (read, write, search), and it must provide the external store. The rest is the model’s job. This is a different division of labour from truncation or summarisation, where the harness decides what to keep. In MemGPT, the model decides.

3.6.3 The trade-offs

Advantages.

• The model’s judgement of relevance is used. The model knows what it is trying to do, so its paging decisions are task-aware in a way a fixed policy is not.
• Unbounded effective memory. The external store can be arbitrarily large; the main context stays bounded.
• Recoverability. Facts paged out can be paged back in, which mitigates failure mode 1 (dropped facts).

Disadvantages.

• The model must be capable of the paging decisions. A model that does not reliably write to memory, or does not search it when it should, gets no benefit. This is a capability requirement, and it is not met by all models.
• Paging costs tokens and steps. Each read, write, and search is a tool call, which adds latency and cost.
• The external store must be maintained. It can grow unboundedly, become inconsistent, or contain stale facts. This is the memory-poisoning risk of Module 19.
• The harness must still bound the main context. Paging in too much defeats the purpose; the model must exercise restraint, which is not guaranteed.

3.6.4 The design question

MemGPT is not a universal improvement over truncation. It is a different point in the design space, appropriate when:

• The task requires memory beyond the effective context length.
• The model is capable of reliable paging decisions.
• The facts are reacquirable from the external store (so that paging out is not lossy).
• The cost of paging is justified by the task’s value.

When these conditions do not hold, a simpler policy (truncation with reacquirable facts) is preferable. The decision procedure is the same as Module 2’s protocol procedure: adopt the more complex mechanism only when its benefits are actually needed.

3.7 Scratchpads and evolving playbooks

3.7.1 The scratchpad

A scratchpad is a region of the context (or an external store) where the agent writes intermediate work: plans, partial results, notes to itself. It is a working-memory device, and it serves two purposes.

Purpose 1 — Externalising intermediate state. The model’s reasoning is lossy across steps: it may forget a conclusion it reached three steps ago. Writing the conclusion to a scratchpad makes it durable, because it is in the context (or the store) and the model can re-read it.

Purpose 2 — Reducing the burden on attention. A fact in a scratchpad can be placed in a high-attention position (near the current instruction) even if its origin is old, which mitigates the U-curve problem.

The scratchpad is a harness component: the harness provides the region, and the model writes to it through tool calls or through structured output. The design question is what goes in it — a plan, a running summary, a list of decisions — and that is task-dependent.

3.7.2 Evolving playbooks

Agentic Context Engineering (arXiv:2510.04618, evidence tier P: preprint, unreplicated) proposes an evolving playbook: a structured document that accumulates strategies, heuristics, and lessons learned across runs, and is updated based on experience. The playbook is part of the context, so the agent benefits from prior experience without retraining the model.

The mechanism has three parts.

Accumulation. After a run, a reflection step extracts lessons (“when the test fails with error X, try Y”) and appends them to the playbook.

Retrieval. At the start of a run, the relevant sections of the playbook are loaded into the context, selected by task similarity.

Curation. The playbook is periodically pruned: lessons that do not help are removed, and contradictory lessons are reconciled. Without curation, the playbook grows unboundedly and its content becomes stale or contradictory — the instruction rot problem.

3.7.3 The evidence status and the design caution

The playbook idea is attractive and the preprint reports positive results, but it is tier P: a single-lab, unreplicated result. The curriculum treats it as a hypothesis, not a recommendation. The specific cautions:

• The reflection step is a model call, and its quality is bounded by the model’s ability to extract generalisable lessons. A model that overfits to the specific failure will write a lesson that does not transfer.
• The retrieval step is a relevance judgement, and a wrong retrieval loads irrelevant lessons — which, by the distractor sensitivity of §3.1.3, degrades performance.
• The curation step is the hardest and the least studied. Without it, the playbook is a liability; with it, the curation policy is itself a design problem.

The right posture is: read the preprint, understand the mechanism, and test it on your task before adopting it. The lab’s structured-extraction policy is a simple version of the same idea (a schema for what matters), and the lab’s results will tell you whether the more elaborate version is worth pursuing.

3.7.4 Concept check

Why is the curation step of an evolving playbook the hardest part?

Answer. Because the playbook is a form of memory that accumulates without bound, and memory that is not curated becomes stale and contradictory. A lesson that was correct for an old version of the codebase may be wrong for the current one; two lessons may give conflicting advice; a lesson may be overfitted to a single failure and not generalise. Detecting which lessons are helping requires measuring their effect, which requires controlled comparison — and the playbook’s size makes the comparison expensive. The curation policy is therefore a research problem, not a maintenance chore, and the preprint’s positive results depend on it in ways that are not yet well understood.

PART D — RETRIEVAL AS A DECISION

3.8 Just-in-time versus pre-loading

3.8.1 The two strategies

Pre-loading. The harness retrieves all potentially relevant content before the first model call and places it in the context. This is the classic retrieval-augmented generation (RAG) pattern: embed the query, retrieve top-$k$ documents, stuff them into the prompt, generate.

Just-in-time (JIT) loading. The harness exposes retrieval as a tool and lets the model decide when to call it. The model sees a description of what is available and retrieves on demand. This is the pattern in modern coding agents: the agent has a search tool and a read tool, and it decides which files to look at.

The two differ in who makes the retrieval decision — the harness or the model — and in when — before or during the run.

3.8.2 The trade-off

Pre-loading.

• Advantage: the model has the content from the start, so it does not spend steps retrieving. For a task where the relevant content is known in advance and the retrieval is reliable, this is efficient.
• Disadvantage: the retrieval is a guess made before the model knows what it needs. If the guess is wrong, the model has the wrong content and no way to get the right content. And if the guess is over-inclusive (top-50 instead of top-5), the distractors degrade performance (§3.1.3).

JIT loading.

• Advantage: the retrieval is conditioned on what the model has learned so far, so it is more likely to be relevant. The model can retrieve the second file only after reading the first and discovering that the second is needed.
• Disadvantage: each retrieval is a step (latency and cost), and the model must know what to retrieve — which requires it to have a good search strategy, which is a capability that varies across models.

3.8.3 The decision-theoretic criterion

The choice can be made precise. Let:

• $p_{\text{pre}}$ be the probability that a pre-loaded context contains the content the task needs.
• $p_{\text{jit}}$ be the probability that a JIT strategy retrieves the needed content, given that the model searches for it.
• $c_{\text{pre}}$ be the cost of pre-loading (tokens for the retrieved content, including distractors).
• $c_{\text{jit}}$ be the expected cost of JIT (the expected number of retrieval steps times the cost per step, including the model call).
• $V$ be the value of a success.

The expected net value of pre-loading is $p_{\text{pre}}V-c_{\text{pre}}$; of JIT is $p_{\text{jit}}V-c_{\text{jit}}$. JIT is preferred iff

i.e.

The left side is the retrieval-quality advantage of JIT (conditioning on what the model has learned); the right side is the cost disadvantage (extra steps). JIT is preferred when its quality advantage exceeds its cost disadvantage.

3.8.4 When each applies

The criterion resolves into a few regimes.

Pre-loading is preferred when:

• The relevant content is identifiable from the task description alone (e.g., “answer this question about this document” — the document is the content).
• The retrieval is reliable (high $p_{\text{pre}}$).
• The model is weak at search (low $p_{\text{jit}}$).
• The cost of retrieval steps is high (long latency, expensive tools).

JIT is preferred when:

• The relevant content depends on what the model discovers mid-run (e.g., “fix this bug” — the file to read depends on the error).
• The content space is large and pre-loading would be over-inclusive (many distractors).
• The model is capable of search (high $p_{\text{jit}}$).
• The cost of retrieval steps is low.

The coding-agent case is the canonical JIT case: the agent cannot know which files are relevant until it has seen the error, so pre-loading is a guess; and the repository is too large to pre-load, so JIT is the only feasible strategy.

3.8.5 The hybrid

In practice, the best designs are hybrids. The harness pre-loads a small, high-confidence set (the task description, the most obvious file) and exposes JIT retrieval for the rest. The pre-load is cheap and reduces the first-step uncertainty; the JIT handles the long tail. The decision criterion applies to each piece of content independently: pre-load it if it is high-confidence, expose it if it is not.

3.8.6 Concept check

Why is pre-loading the wrong default for a coding agent?

Answer. Because the relevant content depends on what the agent discovers. The agent cannot know which file contains the bug until it has read the error message and traced the call stack. Pre-loading would require guessing the relevant files before the error is known, and the guess would be wrong often enough that the pre-loaded content would be mostly distractors — which, by §3.1.3, actively degrade performance. JIT loading lets the agent retrieve the next file only after it knows why it needs it.

3.9 The retrieval decision, decomposed

When retrieval is a tool, the model makes three decisions, and the harness design shapes all three.

When to retrieve. The model must decide that it lacks information and that retrieval would help. A model that retrieves too eagerly wastes steps; one that retrieves too rarely acts on incomplete information. The harness can shape this through the tool description (telling the model when retrieval is appropriate) and through the cost (making retrieval cheap enough that the model is not discouraged).

What to retrieve. The model must formulate a query. A vague query retrieves irrelevant content; a precise query retrieves the right content but requires the model to know what it is looking for. The harness shapes this through the tool interface: a search tool that accepts a natural-language query is more ergonomic than one that requires a structured query language (Module 2, ACI thesis).

How much to retrieve. The model must decide the scope. Retrieving too little misses the content; retrieving too much floods the context with distractors. The harness shapes this through the tool’s parameters (a limit argument, a context_lines argument) and through the result format (returning snippets with line numbers rather than full files, as SWE-agent does).

The three decisions are the retrieval analogue of the three decisions in tool use generally, and the ACI thesis applies: the ergonomics of the retrieval tool determine how well the model makes them.

PART E — PROMPT CACHING AND CACHE-AWARE LAYOUT

3.10 The economics of prefix caching

3.10.1 The mechanism

Modern serving stacks cache the KV (key-value) activations for a prefix of the context. When a request shares a prefix with a previous request, the cached activations are reused, and the model does not recompute the prefill for that prefix. The cached tokens are priced lower than the uncached tokens.

The cost formula (Part 0, §0.8, Proposition 0.8.4):

where $L_p$ is the prefix length, $h$ is the cache hit rate, $C_{\text{cached}}$ and $C_{\text{prefill}}$ are the per-token prices, and $C_{\text{gen}}$ is the generation cost.

3.10.2 The hit rate

The hit rate $h$ is the fraction of the prefix that was already cached. It depends on three factors.

Prefix stability. If the prefix is identical across requests, $h$ is high. If it changes, the cache misses from the point of change onward. This is why the system prompt and the tool schemas — which are stable — are ideal cache candidates, and the conversation history — which grows — is not.

Cache capacity. The cache has a finite size, and entries are evicted (by LRU or a similar policy). A prefix that is used infrequently may be evicted between requests.

Cache granularity. The cache stores activations at token or block granularity. A prefix that differs by one token from a cached prefix misses from that token onward.

3.10.3 The cost saving

The saving is proportional to $L_p$ and to the price differential $C_{\text{prefill}}-C_{\text{cached}}$.

Worked example. A 4,000-token system prompt plus tool schemas, reused across 1,000 requests per day. Prefill price 3 per million tokens; cached price 0.3 per million. At $h=0.9$:

Without caching: $4000\times3/10^6=0.012$. Saving: $0.012-0.00228=0.00972$ per request, or $9.72$ per 1,000 requests per day, or about 292 per month. A 5.3× reduction on the prefix portion. For a high-volume agent, this is the difference between a viable and a non-viable unit economics (Module 22).

3.11 Cache-aware layout

3.11.1 The principle

Put stable content first and volatile content last. The cache stores a prefix; if the prefix is stable, it is cached; if the content changes early, the cache misses from that point. The layout should therefore be:

1. System prompt (stable).
2. Tool schemas (stable, if the tool set does not change per request).
3. Static task context (the repository contents, if they do not change per request; the reference documents, if they are the same across a batch).
4. Conversation history (grows monotonically — partially cacheable, since the earlier turns are stable).
5. Current instruction (changes every step — never cacheable, but placed last for recency).

The principle is the same as the U-curve layout: stable content at the beginning, volatile content at the end. The two principles reinforce each other, which is why the canonical layout is also the cache-optimal one.

3.11.2 The measurement

The hit rate is measurable from the serving stack’s metrics. The procedure:

1. Instrument the requests to record the prefix length, the cached length, and the hit rate.
2. Measure the hit rate over a representative workload.
3. Identify the cache-miss points — where in the prefix the misses occur — and correlate them with content changes.
4. Restructure the layout to move the volatile content later.
5. Re-measure and report the change.

The lab does not include this measurement (it requires a serving stack), but Module 13 does, and the principle should be recorded now: the cache hit rate is a design variable, not a given. A harness that lays out its context carelessly has a lower hit rate than one that lays it out deliberately, and the difference is money.

3.11.3 The interaction with compaction

Compaction changes the prefix. A compaction that rewrites the early history invalidates the cache from the point of rewrite onward. This is a hidden cost of compaction: it is not just the token cost of the compaction call, but the cache invalidation it causes. A compaction policy that rewrites the entire history is more cache-hostile than one that appends a summary at the end and leaves the early content stable. This is a design consideration that the token-cost analysis alone misses, and it is worth recording as a caution: compaction and caching interact, and the interaction should be measured, not assumed.

3.11.4 Concept check

Why is the canonical context layout (system prompt first, current instruction last) also the cache-optimal layout?

Answer. Because the system prompt is the most stable content (it does not change across requests) and the current instruction is the most volatile (it changes every step). Putting the stable content first maximises the length of the cacheable prefix; putting the volatile content last means the cache miss occurs at the end, where it invalidates the least. The U-curve layout and the cache layout are the same because both want the stable, high-attention content first and the current, high-attention content last.

PART F — PROGRAMMATIC OPTIMISATION

3.12 DSPy as a counterpoint to hand-tuning

3.12.1 The idea

DSPy (Khattab et al., arXiv:2310.03714, evidence tier R) treats an LLM pipeline as a program with learnable parameters. The parameters are the prompts, the few-shot examples, and the control flow; the learning signal is a metric on the task. The framework compiles the program by searching over the parameter space (prompt templates, example selections) to maximise the metric.

The contrast with hand-tuning is the contrast between a search and a craft. Hand-tuning is the iterative editing of prompts by a human, guided by intuition and spot checks. DSPy is an automated search, guided by a measured metric. The former is flexible and can exploit human judgement; the latter is systematic and can explore a larger space.

3.12.2 What DSPy optimises

DSPy’s parameters are:

• Prompt instructions: the natural-language instructions for each module.
• Few-shot demonstrations: the examples included in the prompt.
• The composition: how modules are wired together (though in practice the composition is often fixed by the programmer).

The optimiser (e.g., a Bayesian optimiser or a random search) evaluates candidate configurations on a training set and selects the best on a validation set.

3.12.3 The limits

DSPy is not a replacement for context engineering, and it is important to be clear about why.

Limit 1 — It optimises the prompt, not the context. DSPy’s parameters are the prompt and the examples, not the compaction policy, the retrieval strategy, or the memory architecture. A DSPy-optimised prompt on a poorly compacted context is still a prompt on a poorly compacted context. The two are complementary: DSPy can optimise the instruction, and context engineering can optimise what the instruction sees.

Limit 2 — It needs a metric, and the metric is the bottleneck. The optimisation is only as good as the metric. If the metric is a proxy (e.g., an LLM judge rather than a verifier), the optimiser will exploit the proxy (reward hacking, Module 16). The DSPy result inherits the verifier’s reliability (Part 0, §0.5).

Limit 3 — It can overfit. The optimiser searches over a large space on a finite training set, so it can overfit. Held-out validation is essential, and the same pre-registration discipline applies.

Limit 4 — It is a search, and search is expensive. Each candidate evaluation is a set of model calls. The optimisation cost is real, and it must be amortised over the deployment.

3.12.4 When to use it

DSPy is worth considering when:

• The metric is reliable (a verifier, not a judge).
• The prompt is complex enough that hand-tuning is expensive.
• The task is stable enough that the optimised prompt will not go stale.
• The optimisation cost is justified by the deployment volume.

It is not worth considering when:

• The metric is unreliable (the optimiser will exploit it).
• The prompt is simple (hand-tuning is faster).
• The task changes frequently (the optimisation goes stale).
• The deployment volume is low (the cost is not amortised).

3.12.5 Concept check

Why does DSPy not solve the context-engineering problem?

Answer. Because DSPy optimises the prompt — the instructions and examples — not the context. The context is what the prompt is about: the history, the retrieved content, the memory. A DSPy-optimised prompt on a lossy context is a well-written instruction applied to incomplete information; the model still cannot use what is not there. Context engineering and prompt optimisation are orthogonal: one decides what the model sees, the other decides how it is asked to use it.

LAB 3 — THREE COMPACTION POLICIES ON A 200-STEP TRAJECTORY

3.13.1 Task specification

Select a task from your practice that meets three criteria:

1. It produces long trajectories: at least 200 steps in the uncompacted run. If your task is shorter, concatenate multiple runs or choose a task that requires extensive exploration.
2. It has a deterministic verifier.
3. It has enough instances for statistical comparison: at least 30, ideally 50.

3.13.2 The three policies

Policy T (truncation). Retain the system prompt, the original instruction, and the most recent 20 messages. Drop everything else.

Policy S (summarisation). When the context exceeds a threshold (e.g., 8,000 tokens), summarise the oldest half of the history with an LLM, replace it with the summary, and continue. Use a fixed summarisation prompt: “Summarise the following trajectory, preserving any facts needed to complete the task.”

Policy E (structured extraction). When the context exceeds the threshold, extract a structured record using a task-specific schema. For a coding task, the schema might be: files read (path, relevant excerpts), tests run (command, outcome), edits made (path, rationale). Replace the oldest half with the record.

3.13.3 The oracle

For a sample of the task instances (at least 10), construct an oracle summary: a human-written compaction that contains exactly the facts needed to complete the task. The oracle is the upper bound.

3.13.4 Metrics

3.13.5 Pre-registered hypotheses

State before running:

• H1. Truncation has the highest projection error, because it drops the most content.
• H2. Structured extraction has the lowest projection error, because its schema is designed for the task.
• H3. Summarisation has the highest variance in projection error, because its relevance judgements vary across tasks.
• H4. The compaction inefficiency is largest for the tasks where the decisive fact is old (the scrolled-out-action case).
• H5. The token cost is lowest for truncation (no compaction call) and highest for summarisation (a compaction call per window).

3.13.6 Acceptance criteria

3.13.7 What good looks like

A good lab report has five properties.

It measures downstream success, not summary quality. The report should not contain a section titled “Summary quality.” It should contain a section titled “Task success under each policy,” with confidence intervals.

It uses the oracle. The projection error alone tells you how much you lost; the compaction inefficiency tells you how much room there was. A policy that loses 5 points on a task with a 0.90 oracle ceiling is near-optimal; a policy that loses 5 points on a task with a 0.65 ceiling is leaving 30 points on the table.

It decomposes the loss. The dropped-vs-corrupted decomposition tells you which remedy to apply. A policy that mostly drops facts needs better retrieval; a policy that mostly corrupts facts needs a more faithful compaction.

It reports the interaction with the task. The interesting result is usually that the policies rank differently on different task types. A report that says “policy E had the highest average success” has missed the point; the point is where each policy wins.

It states the revision condition. Under what observation would you change the policy? A larger effective context, a different task distribution, a different model — each would change the answer, and the memo should say which.

MODULE 3 CHEAT SHEET

EXERCISES WITH SOLUTIONS

E3.1. A model is advertised at 1M tokens. On your task, success is 0.80 at 8k, 0.79 at 32k, 0.74 at 128k, 0.55 at 512k. Compute $L_{\text{eff}}(0.05)$. What layout would you design for?

S3.1. $M(\ell_0)=0.80$. Tolerance 0.05 → threshold 0.75. At 32k, 0.79 ≥ 0.75. At 128k, 0.74 < 0.75. So $L_{\text{eff}}(0.05)=32k$. Design for a 32k budget: system prompt and tool schemas at the front (stable, cached), the task context compacted to fit, the current instruction at the end. Do not construct 128k contexts; the model will accept them and degrade.

E3.2. A compaction policy reduces a 200-step trajectory to 10% of its tokens and loses 8 points of task success. The oracle loses 0 points. Compute the projection error and the compaction inefficiency. Is the policy near-optimal?

S3.2. Projection error $=8$ points (the loss relative to no compaction). Compaction inefficiency $=0-(-8)=8$ points (the gap to the oracle). The policy is not near-optimal: there is 8 points of room. A policy that lost 8 points with an oracle that also lost 8 points would be near-optimal; here the oracle loses nothing, so the entire 8 points is recoverable in principle.

E3.3. A summarisation policy produces summaries that a human rates 4.5/5 on faithfulness and completeness, but loses 15 points of task success. Explain the discrepancy, and say what you would measure instead.

S3.3. The human is judging the summary against a general standard of what a good summary is, not against the task’s information needs. The summary can be faithful (it does not hallucinate) and complete (it covers the trajectory) and still omit the one fact that determines the next action, because the summariser did not know which fact was decisive. Measure task success under the compaction, and compare it to the oracle summary’s success; the gap is the information that mattered and was lost. The summary-quality rating is a proxy that is uncorrelated with the decision-relevant loss.

E3.4. A task requires reading 50 files. Pre-loading retrieves the top 50; the relevant file is in the top 50 with probability 0.9. JIT retrieves the relevant file with probability 0.85, at an expected cost of 3 extra steps (0.3 s each). Pre-loading costs 40,000 tokens (0.012); JIT’s steps cost 0.01. The value of a success is 1.0. Which is preferred?

S3.4. Pre: $0.9\times1.0-0.012=0.888$. JIT: $0.85\times1.0-0.01=0.84$. Pre-loading is preferred by this calculation, because its retrieval-quality advantage (0.05×1.0 = 0.05) is less than its cost advantage (0.012 vs 0.01 — actually pre-loading costs slightly more, but the quality advantage dominates: 0.05 > 0.002). If JIT’s retrieval quality were 0.95, JIT would win (0.95−0.01=0.94 > 0.888). The criterion is the quality-vs-cost comparison, not a preference for either strategy.

E3.5. A context layout puts the conversation history first and the system prompt last. What are the two problems, and how would you fix them?

S3.5. (1) U-curve problem: the system prompt — the stable instruction — is in the recency position, which is high-attention, but the history is in the primacy position, and the current instruction (if placed after the system prompt) is in the middle. The model attends most to the history and the system prompt, and least to the current instruction. (2) Cache problem: the history is volatile, so putting it first means the cache misses from the first turn, and the stable system prompt at the end is not cached as a prefix. Fix: system prompt first (stable, cached, primacy), history in the middle (partially cached), current instruction last (volatile, recency). Both problems are solved by the canonical layout.

E3.6. A DSPy-optimised prompt improves task success from 0.70 to 0.78 on the validation set, but on a held-out test set it achieves 0.69. What happened, and what would you do?

S3.6. Overfitting. The optimiser searched a large prompt space on a finite validation set and found a configuration that exploits the validation set’s idiosyncrasies. On held-out data it does no better than the baseline. Remedies: (1) use a larger validation set or cross-validation; (2) restrict the search space (fewer prompt templates, fewer examples); (3) pre-register the test set and do not use it for selection; (4) check whether the metric is a proxy that the optimiser is exploiting. The lesson is the same as in any optimisation: the search is only as good as the held-out evaluation, and the evaluation must be protected from the search.

E3.7. A MemGPT-style agent pages a fact out to external memory at step 10 and needs it again at step 50. It searches external memory and finds the fact. Is this a success or a failure of the memory architecture? What would tell you?

S3.7. It is a success of the memory architecture in the sense that the fact was recoverable — the paging out was not lossy, and the search recovered it. But it is a cost in the sense that the agent spent steps paging and searching, which a policy that had retained the fact in the main context would not have. Whether the architecture is net-positive depends on the frequency of such reacquisition events: if they are rare, the paging saves main-context space at low cost; if they are common, the agent is thrashing, and a larger main context or a different policy would be better. The diagnostic is the page-in rate — how often the agent reacquires paged-out facts — and a high rate is a signal that the main context is too small for the task.

E3.8. You are designing a harness for a task where the relevant content is identifiable from the task description (a document question-answering task). Should retrieval be a tool or a pipeline stage? Justify with the decision criterion.

S3.8. A pipeline stage (pre-loading). The relevant content is identifiable from the task description, so $p_{\text{pre}}$ is high; the content is a single document, so the distractor cost is low; the model does not need to discover anything mid-run, so $p_{\text{jit}}$ offers no quality advantage. By the criterion $(p_{\text{jit}}-p_{\text{pre}})V>c_{\text{jit}}-c_{\text{pre}}$, the left side is near zero (the qualities are similar) and the right side is positive (JIT costs steps), so pre-loading is preferred. Making retrieval a tool here would add latency and cost for no benefit — the same reasoning as Module 2’s protocol decision procedure, applied to retrieval.

EXIT TEST FOR MODULE 3

You are ready for Module 4 when you can, without notes:

1. State the Lost in the Middle result and derive the three layout consequences (system prompt first, current instruction last, retrieved content near the instruction).
2. Explain distractor sensitivity and why “include less” is the default heuristic.
3. Distinguish nominal from effective context, and describe the procedure for measuring $L_{\text{eff}}(\epsilon)$.
4. Define a compaction policy formally and prove the loss bound $I(b_t;c_t)\le I(b_t;h_t)$.
5. Define the projection error and the compaction inefficiency, and explain why the latter requires an oracle.
6. Compare truncation, summarisation, and structured extraction on the five properties in §3.4.4.
7. Explain the summarisation quality trap and why downstream success is the correct measure.
8. Describe MemGPT’s two-tier architecture and the conditions under which it is preferable to truncation.
9. Explain the evolving-playbook mechanism and why it is a hypothesis (tier P), not a recommendation.
10. Apply the retrieval decision criterion and distinguish pre-loading from JIT by the three decisions (when, what, how much).
11. Compute the prefix-caching cost saving and lay out a context for cache efficiency.
12. Explain why DSPy optimises the prompt but not the context, and state its four limits.

End of Module 3. Say “next” to receive Module 4 in full.