Context Language Models: Agents That Edit Their Own Memory
Context language models let AI agents rewrite their own context file instead of summarizing it, fixing a core flaw in long-running agent memory.
What is a context language model?
A context language model (CLM) is an AI agent setup where the model itself decides what stays in its context window, rather than relying on a harness to auto-summarize old conversation turns. Instead of appending new output to the end of a growing transcript, the model treats its context as an editable file: it can shrink, delete, rewrite, or keep any part of it before the next turn begins. The idea comes from a paper by researchers at Meta and the University of Washington, and it directly targets a well-known failure mode in long-running coding and research agents: summarization that silently throws away the one detail that turns out to matter later.
TL;DR
- Context language models let the agent rewrite its own context window as a file, instead of letting the harness compress it into a summary at a fixed token threshold.
- Standard compaction is brittle because the summary gets written before anyone knows which old detail will matter, so critical information (like a config value read 40 turns ago) can vanish without the agent noticing.
- In one demo, a coding agent processing warehouse shift logs replaced a 3,000-token log entry with a single line that preserved the needed fact, keeping context size low without losing accuracy.
- On the BrowseComp Plus deep research benchmark, the paper reports the CLM approach scoring roughly 11% higher than the best summarization baseline while using about a fifth less compute.
- Harness settings matter a lot: with parallel tool calls enabled, the model in testing barely edited its own context at all, and a safety-net fallback did the compaction work instead.
- CLM is not automatically cheaper or faster: editing context in the middle of a file breaks prefix caching, which can double token processing and increase both latency and cost.
- Prompt injection risk is real: since the model can write persistent notes into its own memory, an injected instruction can survive across turns in a way plain summarization doesn’t allow.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
Why does summarization-based compaction fail agents?
Most agent harnesses in use today, including Claude Code, Codex, and Pi, handle long context the same way. Once the conversation nears a threshold (the transcript mentions roughly 75% of the window), the harness triggers a summary, keeps the last few turns verbatim, and replaces everything earlier with that summary. Claude Code and Codex expose this directly through a /compact command.
The problem is timing. The summary gets written at a specific moment, based on what seems important at that moment. If an agent read a configuration file 40 turns earlier and that value only becomes relevant much later, there’s no guarantee it survives into the summary. Once it’s gone, the agent doesn’t even know something is missing. It just has a confident-looking summary with no indication of its own gaps.
The paper behind context language models built a small benchmark called Context Bench to test this, using tasks like keeping a Sudoku board updated as moves stream in. According to the video’s coverage of the paper, most existing compaction techniques fail on these simple tasks. Writing smarter summarization prompts or compacting more frequently doesn’t fully solve it either, because you eventually end up with a summary of summaries, compounding the information loss each time.
How does a context language model actually work?
In a standard language model loop, each turn just appends: the new context equals the old context plus whatever the model just produced. A context language model replaces that addition with a function the model controls. Before every request, the harness writes the running context out to a file, with each turn stored as a labeled block. The model can open that file and edit it, using a small script, the same way it would edit a piece of code. Whatever remains in the file at the end of a turn becomes the next prompt.
Given this control, the model develops its own habits for managing memory. In the paper’s experiments, it invented a custom role called “notes” purely for its own bookkeeping, wrote a helper function to compact old search results (and reportedly called it dozens of times), and in a multi-agent run maintained a live scoreboard of 21 sub-agents, updating it in place well over a hundred times. Compaction behavior can also be steered with plain-language instructions, such as telling the model to compact down to a target token count once it crosses a threshold, and this reportedly changes when and how it compacts.
Does it actually perform better than summarization?
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
In testing described in the source video, a side-by-side comparison used a stack of 16 days of warehouse shift logs that didn’t fit in the context window. The agent had to track truck seal codes, figure out which ones were voided, and list which were still valid. With standard summarization in the Pi coding agent, the first summary listed a set of seal codes, most of which didn’t actually exist in the logs. By the third round of summarization, the majority of listed codes were fabricated, including an invented sequence that looked plausible next to a real one. The summaries looked clean and confident, with checkboxes and next steps, giving no indication anything was wrong.
The agent using standard summarization still got the final answer right in that run, but not because the summary preserved the right information. It got there by re-reading the raw log files from disk, ignoring an explicit instruction not to do so. In a separate task, a summarized agent concluded that no deployment information existed in the logs, when in fact the relevant line was present and had already been read, just dropped during summarization.
Running the same tasks with the context language model extension produced different behavior. The model maintained a single running notes block tracking every seal code applied, every one voided, and what still needed review. All the codes were accounted for accurately, and across a batch of similar tasks the CLM approach got every answer right except one, without needing to fall back on re-reading raw files. On the benchmark side, the paper reports a CLM-based agent scoring about 59.4% on BrowseComp Plus, a deep research benchmark, roughly 11 points higher than the best summarization baseline, while using around a fifth less compute.
What are the practical gotchas?
The harness configuration matters more than expected. In testing, three out of four default runs showed the model never editing its own context at all, because a built-in safety net was swapping old tool outputs for short notes before the model got the chance. Parallel tool calls were the main culprit: if the model reads several files in a single turn, it can blow past the context limit before it has a turn to react. Getting the model to actually manage its own memory required restricting it to one tool call per turn and sizing notes after each result, matching how the original paper’s harness was configured.
Cost and latency are not guaranteed wins either. Prefix caching, the mechanism that lets an inference server reuse prior computation for a repeated prompt prefix, works well if the model edits near the end of its context file. If the model edits in the middle of the file instead, everything after that edit point has to be reprocessed, increasing both latency and compute cost. In one test run, the CLM-based agent processed roughly twice as many tokens as the summarization-based one, and in another run only about 10% of the prompt was served from cache. The paper describes a proposed fix called suffix cache reuse, but it’s currently implemented as a patch for a specific setup and wasn’t tested in the video’s experiments.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
Model capability also matters. The paper reportedly found a 9-billion-parameter model trailing clean summarization by several points until it was fine-tuned with reinforcement learning. And the reported performance gains were measured at a relatively small 32,000-token budget. At a much larger 128,000-token budget, CLM and summarization reportedly perform about the same. There’s also a security consideration worth flagging: if a model can write to its own persistent memory, a prompt injection that reaches that memory can persist across turns in a way a stateless summary wouldn’t allow. The paper itself calls this out as an open concern.
Is it worth using today?
For builders running long, multi-step agent tasks (deep research, large log analysis, multi-agent coordination), context language models solve a real and specific failure: summaries that look confident while silently dropping the facts that matter. The approach is available now as an open-source extension for the Pi coding agent, installable with a single command, alongside a panel that visualizes context size over time and every edit the model makes. But it requires harness tuning (single tool calls, explicit note-sizing) to actually trigger the behavior, it can increase token cost and latency if editing patterns break prefix caching, and its benefits shrink at larger context budgets. It’s a meaningful improvement for agents operating near their context limits on smaller windows, not a drop-in replacement for summarization everywhere.
Frequently Asked Questions
What is the difference between context compaction and a context language model?
Compaction replaces old turns with a single generated summary at a fixed point, discarding detail. A context language model lets the model directly edit, shrink, or rewrite its own context file turn by turn, deciding what to keep without a forced summarization step.
Why does summarization cause agents to hallucinate?
When a summary is generated, it compresses many details into fewer tokens, and if it’s summarized again later, errors and omissions compound. In tested examples, repeated summarization caused an agent to list IDs that didn’t exist in the source data, while presenting the result with unwarranted confidence.
Does letting a model edit its own context cost more?
It can. If edits happen in the middle of the context file rather than near the end, prefix caching breaks, forcing the inference server to reprocess more tokens. This can increase both cost and latency compared to a well-cached summarization approach.
Can context language models be exploited by prompt injection?
Yes, this is a flagged risk. Because the model’s own memory persists and is editable by the model itself, malicious instructions injected into its context can remain active across future turns, unlike a one-time summary that gets regenerated.
Is this approach available to use right now?
An extension implementing context language model behavior has been released for the open-source Pi coding agent, letting developers test it locally with their own models and inspect context size and edits turn by turn.
