Context Language Models vs Summarization: What Hands-On Testing Found
Hands-on tests compare Pi's default summarization to context language models, exposing hallucination rates, cache costs, and prompt injection risks.
What is a context language model, and how is it different from summarization?
A context language model (CLM) is an agent architecture where the model edits its own context window directly, instead of relying on a harness to summarize old turns when the window fills up. Standard agent harnesses like Claude Code, Codex, and Pi handle long sessions by triggering a summary at a fixed point (often around 75% of the context window), keeping recent turns and replacing everything older with a condensed text block. CLMs, introduced in a paper from Meta and the University of Washington, replace that process entirely: before every request, the harness writes the context to a file, and the model can open that file and edit it with a small script, the same way it would edit code. Whatever remains in the file becomes the next prompt. The model decides, turn by turn, what to keep, shrink, delete, or rewrite.
TL;DR
- Standard summarization can silently drop critical details: in one test, Pi’s own summarization hallucinated 11 of 12 tracked codes on the first pass and 14 of 18 by the third summary, while still presenting the result with confident-looking checklists.
- Context language models edited a running notes block instead of summarizing, and in the same tracking task got all 18 codes exactly right, missing only one answer across a batch of related tasks.
- The harness configuration determines whether CLM even activates: with default parallel tool calls, the model skipped self-editing in three of four runs and fell back to a built-in safety net that swaps old tool outputs for short notes.
- Prefix caching cost depends on where in the context file the model edits: changes near the end of the file preserve most of the cached prefix, but edits in the middle invalidate everything after that point and force expensive reprocessing.
- CLM is not automatically cheaper: one test run processed roughly twice as many tokens as standard summarization, and another saw only 10% of the prompt served from cache.
- Model capability matters: the paper’s own benchmark showed a 9-billion-parameter model trailing clean summaries by six points until it was fine-tuned with reinforcement learning, and the CLM advantage shrank to a tie with summarization at a 128,000-token budget.
- Self-editable memory creates a new attack surface: a prompt injection that gets written into the context file can persist across every subsequent turn.
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
Why does context summarization fail in practice?
The core problem with fixed-point summarization is timing. A harness writes a summary at a set threshold, before anyone knows which piece of information will turn out to matter later. If a value read 40 turns earlier gets compressed out of that summary, the information is gone, and the agent has no way to know something is missing. It either re-derives the answer at extra token cost or simply gets it wrong.
Hands-on testing against Pi’s default summarization showed this directly. In a synthetic task built around 16 days of warehouse shift logs, where the agent had to track truck seal codes and determine which had been voided, the first summary invented 11 of 12 seal codes that did not exist anywhere in the source logs. By the third round of summarizing summaries, 14 of 18 codes were fabricated, including an invented sequence placed suspiciously close to a real one. In a separate test, the agent summarized its way into concluding that deploy information wasn’t present in the logs, despite having already read the line containing it.
The agent in the seal-code test still landed on the correct final answer, but not because the summary preserved the right information. It worked only because the model ignored its own summary, went back to the raw logs, and re-read everything with grep, despite being told not to. That is a workaround, not a fix. It burns tokens and time that a reliable memory system wouldn’t need.
How well did the context language model perform on the same tasks?
Running the same warehouse log tasks through Pi’s CLM extension produced a different outcome. Instead of compressing history into a lossy summary, the model maintained a single running notes block tracking every seal code applied, every code voided, and what still needed review. All 18 codes came out exact, and across a batch of seal and incident tasks, the model got every answer right except one, with no need for Pi’s “re-read everything with grep” fallback.
The underlying paper backs this up at larger scale. On BrowseComp Plus, a deep research benchmark, a CLM scored 59.4% accuracy using a 20.6B-parameter model, about 11% higher than the best summarization approach while using roughly a fifth less compute. The paper also describes CLMs inventing their own memory conventions when given freedom: creating a dedicated “notes” role, writing helper functions to compact old search results, and maintaining a running scoreboard across dozens of sub-agents, updating it in place over a hundred times in one run.
What gotchas should you watch for before adopting this?
The results are good, but hands-on testing surfaced four real limitations worth understanding before building on this approach.
The harness configuration matters more than expected. In three of four default test runs, the model never edited its own context at all. The reason was parallel tool calls: when the model reads several files in a single turn, it can blow past the context budget before getting a chance to edit anything, and Pi’s built-in safety net (which swaps old tool outputs for a short note) ends up doing all the work instead of the model’s own editing logic. Getting the model to actually self-edit required switching to one tool call per turn and forcing a sizing note after every result, matching how the paper’s own harness was configured.
Prefix caching cost is edit-position-dependent. Inference servers like VLLM cache the processed prompt so repeated prefixes don’t need reprocessing. If a CLM edits near the end of its context file, most of that cached prefix stays valid and responses stay fast. If it edits in the middle of the file, everything after that edit point falls out of the cache, forcing the server to reprocess it. That is a direct latency and cost hit, and it is not visible unless you are specifically watching cache hit rates.
CLM is not automatically cheaper. In one seal-tracking run, Pi CLM processed roughly twice the tokens that standard Pi summarization used. In another, only 10% of the prompt was served from cache. The paper proposes a fix called suffix cache reuse, which preserves cached work for unchanged text after an edit, but it currently exists only as an SGLang patch tested on a single model and a single GPU, not something broadly reproducible yet.
Model capability is a prerequisite, not a given. In the paper’s own benchmark, a 9-billion-parameter model trailed clean summarization by six points until it was trained with reinforcement learning specifically for context editing. The headline results also rely on a relatively tight 32,000-token budget; at 128,000 tokens, CLM and summarization performance converge to roughly a tie. Smaller or untrained models may not be ready to manage their own memory reliably.
Is a self-editable context window a security risk?
Yes, and it’s worth taking seriously. If a model can write to its own memory, so can a successful prompt injection. Anything malicious that gets embedded into the context file during one turn persists across every subsequent turn, since the file itself is the model’s memory. This is a concern the paper’s own authors flag directly. Standard summarization has its own failure modes, but it doesn’t give an attacker a persistent foothold inside the agent’s working memory the way a self-editing context file can.
Frequently Asked Questions
What is Pi CLM?
Pi CLM is an extension for the open-source Pi coding agent that implements the context language model approach from the Meta/University of Washington paper. It adds a panel showing context size over time and every self-edit the model makes, and can be installed with a single command inside Pi.
Does a context language model always beat summarization?
No. It performed better in hands-on testing at a 32,000-token budget and on the paper’s BrowseComp Plus benchmark, but the advantage shrinks to roughly a tie at 128,000 tokens, and smaller models need reinforcement learning to close the gap with summarization quality.
Why did Pi’s summarization hallucinate seal codes?
Summarizing a summary compounds information loss. Each pass drops detail that the model has no way of flagging as missing, and the resulting text still reads as confident and complete, so there’s no visible signal that something was fabricated or dropped.
Is context editing more expensive than summarization?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
It can be. Cost depends heavily on where in the context file edits happen. Edits near the end preserve prefix cache hits; edits in the middle invalidate the cache for everything after that point, increasing both token reprocessing and latency.
What setup is needed to get a model to actually self-edit its context?
Testing found that default settings with parallel tool calls often let the model blow past its context budget before it could edit anything, triggering a fallback safety net instead. Limiting the harness to one tool call per turn and forcing a sizing note after each result got the model to reliably perform its own context edits.
