Run Context Language Models Locally: Pi CLM + VLLM Setup Guide
How to install the Pi CLM extension and run context language models locally with Qwen3 27B on VLLM, plus the settings that actually make it work.
What is a context language model, and why run one locally?
A context language model (CLM) is a language model that edits its own context window instead of letting a harness summarize it when space runs out. Rather than appending every turn to the end of a growing conversation, the model treats its context as a file it can open, rewrite, shrink, or delete from, turn by turn. The idea comes from a Meta and University of Washington paper, and it’s implemented as an open-source extension for Pi, an open-source coding agent. Running it locally with a model like Qwen3 27B served through VLLM lets you watch exactly how the model manages its own memory, and lets you test whether it actually holds up better than standard summarization on tasks that don’t fit in the context window.
TL;DR
- Context language models replace the usual “append and summarize” loop with a function the model controls, letting it keep, shrink, or rewrite any part of its own context instead of relying on a fixed compaction point.
- The Pi CLM extension installs with a single command and adds a
/clmpanel inside Pi showing context size over time and a side-by-side view of every edit the model makes. - Standard summarization can hallucinate badly under pressure: in one test, a third-round summary invented 14 of 18 tracked codes while still looking confident with checkboxes and next steps.
- One tool call per turn is a required setting, not optional. Parallel tool calls can blow past the context budget before the model gets a chance to edit anything, which defeats the whole mechanism.
- Prefix caching costs depend on where in the file the model edits. Edits near the end of the context preserve cache and stay cheap; edits in the middle force the server to reprocess everything after that point.
- CLM is not automatically cheaper or faster. In testing, one run processed roughly twice the tokens of standard Pi summarization, and another saw only 10% of the prompt served from cache.
- Model quality still matters. The underlying paper found a 9B model trailed clean summaries until it was fine-tuned with reinforcement learning, and the CLM advantage shrank to a tie at larger context budgets like 128,000 tokens.
How does a context language model actually work?
In a standard agent loop, every turn adds to the end of the context: old context plus new output. A CLM replaces that “plus” with a function the model itself controls. Before each request, the harness writes the current context out to a file, with every turn stored as a labeled block. The model can then open that file and edit it with a small script, the same way it would edit a code file. Whatever remains in the file at the end of the turn becomes the next prompt.
This gives the model full discretion over what survives. In the paper, the model invented its own chat role called “notes” to track information it wanted to persist, wrote a helper function to compact old search results and called it dozens of times in a single run, and in a multi-agent setting maintained a running scoreboard of 21 sub-agents that it updated in place over 160 times. Compaction behavior can also be steered in plain English: telling the model to compact down to a target token count once it crosses a threshold is enough to change when and how aggressively it cleans up.
On BrowseComp Plus, a deep research benchmark, the paper reports a CLM-based approach scoring 59.4% with a 27B model, about 11 percentage points above the best summarization baseline, using roughly a fifth less compute.
How do you install and run the Pi CLM extension?
The authors released the CLM capability as an extension for Pi, installable with a single command inside the Pi environment. Once installed, typing /clm opens a dedicated panel that shows context size climbing as the agent reads files and dropping each time it cleans up after itself, along with a page showing every edit side by side with the original content.
For local testing, Qwen3 27B served through VLLM is a practical choice. The original paper used an earlier version of this model family; running the current release still works, but be aware you’re not reproducing the paper’s exact numbers. The paper used a 32,000-token context budget for its headline results, which is worth matching if you want comparable behavior, since the CLM advantage narrows considerably at larger budgets.
Which settings make or break Pi CLM?
Two settings determine whether the model ever actually edits its own context:
- One tool call per turn. With parallel tool calls enabled, the model can read multiple files in a single turn and overshoot its context budget before it gets a chance to react. In testing, three out of four default runs never triggered any self-editing at all, because Pi’s built-in safety net (which swaps old tool outputs for a short note when context is about to overflow) did all the work instead of the model.
- A size note after every tool result. This gives the model visibility into how much context it’s accumulating after each step, which is necessary for it to decide when to compact.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
With both settings active, the model started editing its own context file the way the paper’s harness intended.
Does context language model performance beat standard summarization?
In hands-on testing with synthetic tasks (tracking truck seal codes across 16 days of warehouse logs, where codes get voided days after being issued), standard Pi summarization showed clear failure patterns. The first summary listed 12 seal codes, 11 of which didn’t exist in the source logs. By the third round of summarizing summaries, 14 of 18 codes were fabricated, including an invented sequence placed suspiciously close to a real code. The summaries still looked fully confident, complete with checkboxes and next steps, giving no signal that anything was wrong. The agent eventually got the right answer, but only because it ignored instructions and re-read the raw logs with grep rather than trusting its own summary.
Running the same tasks with Pi CLM, the model kept a single notes block tracking every seal code applied, every code voided, and what still needed reading. All 18 codes came back exact, and across multiple tasks it got every answer right except one, where Pi’s grep fallback was needed instead of the standard summarizer’s hallucinated memory.
What are the real costs and limitations of running this locally?
Four gotchas stood out in testing:
Cost and latency depend on prefix caching. VLLM and similar servers cache the processed prompt so repeated prefixes don’t need reprocessing. If the CLM edits its context near the end of the file, the cached prefix stays valid and responses stay fast. If it edits in the middle, everything after that point falls out of the cache and has to be reprocessed, increasing both latency and compute cost.
It’s not automatically cheaper. One test run saw Pi CLM process roughly double the tokens of standard Pi. Another saw only 10% of the prompt served from cache. The paper describes a fix called suffix cache reuse to preserve unchanged text after an edit, but as of this testing it only exists as a patch for a specific setup (SGLang, one model, one GPU) and wasn’t broadly testable.
Model quality still matters a lot. The paper’s 9B model trailed clean summarization by six points until it was trained with reinforcement learning specifically for this task. And the gap between CLM and summarization shrinks to roughly a tie once the context budget grows to 128,000 tokens, since there’s less pressure to compact aggressively at that size.
Prompt injection risk is real. If a model can write to its own memory, an injected instruction can persist across every subsequent turn instead of fading out. The paper itself flags this as an open concern.
Frequently Asked Questions
What is the Pi CLM extension?
It’s an open-source extension for the Pi coding agent that implements context language models, letting the model edit its own context file directly instead of relying on the harness to summarize it at a fixed threshold.
Do I need a specific model to use Pi CLM?
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
No specific model is required, but model capability affects results. Testing used Qwen3 27B via VLLM; the original paper used an earlier version of the same family plus a 9B model that needed reinforcement learning to match summarization quality.
Why does one tool call per turn matter so much?
Parallel tool calls let the model read multiple large files in a single turn, often exceeding the context budget before the model can intervene. With that setting off, Pi’s own overflow safety net handles cleanup instead of the model, which defeats the purpose of testing CLM behavior.
Is running CLM locally cheaper than standard summarization?
Not necessarily. Token processing can roughly double compared to standard summarization depending on how often and where the model edits its context, and cache hit rates can drop sharply if edits land outside the end of the file.
What’s the biggest risk with letting a model edit its own context?
Prompt injection persistence. Because the model’s self-edits carry forward into every future turn, an injected instruction that makes it into the context file can stay active far longer than it would under a standard summarization cycle.