LMCache vs vLLM Caching: What Actually Speeds Up Coding Agents?
LMCache and vLLM's built-in KV caching both cut agent latency, but they solve different problems. Here's when each one actually matters.
Why do coding agents get slower the longer they run?
Every turn in an agent session makes the model reread the entire conversation so far, unless something is cached. Without caching, a 30-turn coding session can balloon from a quick first reply to tens of seconds of silence before the model writes a single word, because the prompt it has to process keeps growing. Multiply that across several agents sharing one GPU, and the slowdown compounds instead of staying flat. That’s the core problem KV caching solves, and it’s why the choice between vLLM’s built-in caching and a separate layer like LMCache matters for anyone running agents on their own hardware.
TL;DR
- KV cache stores the model’s “notes” (the key/value states from processing a prompt) so it doesn’t have to reread text it already saw on a previous turn.
- vLLM ships with a prefix cache on by default, and in testing it cut prompt rereading by roughly 14 times compared to no cache at all, with similar gains in raw latency.
- vLLM also supports CPU offload, spilling cached notes into system RAM when GPU memory fills up, and in testing it matched LMCache’s speed in most live-serving scenarios.
- LMCache’s real advantage shows up after a restart. Because it stores notes in a separate process, it survives a vLLM crash or reboot, while vLLM’s own cache (GPU or CPU offload) is wiped clean and has to rebuild from scratch.
- With multiple agents sharing a GPU, caching stops being a nice-to-have. Testing showed up to roughly 5.6x to 7.5x more completed turns per minute with eight agents running cached versus uncached.
- LMCache adds operational cost: extra memory held during restarts, slower restart times in some tests, and rougher support on newer or non-standard model architectures.
- The practical rule from testing: leave vLLM’s prefix cache on always, add CPU offload if your cache outgrows GPU memory, and reach for LMCache only if you need the cache to survive restarts or outgrow RAM entirely.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
What does KV cache actually save you?
Before a language model generates any output, it processes the entire input prompt and builds an internal representation of it, commonly called the KV cache (key and value tensors). This is computationally expensive and happens every single time the model sees a prompt, unless the result is cached and reused.
In one test using a 30-turn agent conversation, a setup with no caching at all had to reread the full prompt on every turn, climbing to around 115,000 tokens of rereading by the final turn, and over 2 million tokens cumulatively across the session. With caching enabled, the model never had to read more than about 10,000 tokens per turn, a roughly 14x reduction in repeated work.
The catch is that this cache isn’t free. It lives in GPU memory alongside the model weights themselves, and GPU memory is finite. Once it fills up, the oldest cached entries get evicted, forcing the model back into full rereads on subsequent turns.
How does vLLM’s built-in caching work?
vLLM, the open source inference server widely used to run large language models, includes its own prefix caching by default. It reuses cached notes as long as a new prompt starts with the same text as a previously cached one, and it keeps that cache inside GPU memory, inside the vLLM process itself.
vLLM also offers a CPU offload mode, which lets cached data spill out of GPU memory into regular system RAM when the GPU runs out of room, while still being managed entirely within the vLLM process.
In practice, testing on a model served from a document-heavy prompt showed both of these cutting response time from roughly 1.35 seconds (no cache) down to around 112 to 122 milliseconds, an 11x to 12x speedup. When the workload was scaled up to overflow available GPU cache (48 large documents against a GPU that could hold notes for far fewer), vLLM’s GPU-only prefix cache saw its hit rate collapse to 0%, performing about as poorly as no caching at all. Switching on CPU offload fixed that, recovering hit rates above 99% and bringing latency back down to roughly 164 milliseconds.
What does LMCache add that vLLM doesn’t already do?
LMCache is a separate caching layer that sits outside the vLLM process. Instead of keeping cached notes inside vLLM’s own memory space, it stores them in a standalone server, which can use system RAM and, in some configurations, NVMe solid-state drives for overflow.
In steady-state serving, with everything fitting in GPU memory or RAM, LMCache performs roughly on par with vLLM’s own CPU offload. In one comparison, LMCache measured around 140 milliseconds against vLLM’s offload at 164 milliseconds, essentially a tie. Under concurrent load (16 simulated clients generating roughly 144 concurrent requests), LMCache and vLLM’s CPU offload again finished close together, both well ahead of vLLM’s GPU-only prefix cache and dramatically ahead of no caching, which took over three minutes to complete a job the cached versions finished in roughly a minute.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
The real separation appears after a restart. Because vLLM’s prefix cache and CPU offload both live inside the vLLM process, restarting vLLM wipes them out entirely. In testing, both fell back to full, uncached latency (around 1.35 seconds) immediately after a restart. LMCache, because its cache lives in a separate server that keeps running independently, reconnected after the restart and immediately resumed near-instant responses (around 141 milliseconds with a 99.9% hit rate).
Is LMCache worth the added complexity?
Not automatically. Running LMCache introduces real operational overhead that shows up clearly in testing.
When vLLM was restarted while LMCache still held onto roughly 89 gigabytes of GPU memory from the dead process, the new vLLM instance failed to start due to insufficient memory. There’s a configurable wait time for releasing that memory, but it has a floor: a 30-second minimum. Even at that minimum, the restart was roughly 20 to 30 seconds slower overall than plain vLLM. Counting full restart time, LMCache actually finished about 20 seconds behind vLLM’s own cache in one head-to-head session-restart comparison, even though LMCache was dramatically faster per-turn once running.
Support across newer model architectures was also inconsistent in testing. One mode (cache blending, which reuses document chunks even when they appear in a different order in the prompt) had no vLLM support available at test time. A hybrid model architecture caused vLLM’s own CPU offload to fail to start entirely, and LMCache itself required rebuilding against a newer version before it would run on that same model.
How much faster are coding agents with caching turned on?
This is where the gains compound rather than just add up. In agent-specific testing using real coding sessions (scripted, with actual files and logs replayed identically across each caching configuration), a single agent’s late-session turns went from around 104 seconds of wait time with no cache down to roughly 2.1 seconds with caching enabled, about 5.6 times more completed turns per minute.
Scaling to eight agents sharing one GPU made the gap larger. With no cache, overall throughput stayed around eight turns per minute, since the GPU spent most of its time rereading full prompts for every agent. With caching on, throughput climbed to nearly 60 turns per minute. Split evenly, that moves each agent from roughly one turn per minute to about seven. The same pattern held across two different machines and model combinations, with roughly 5.6x and 7.5x improvements in turns per minute at eight concurrent agents.
One nuance from testing: a cached request sharing a GPU with a large, uncached request can still get stuck waiting, because the serving layer may batch requests together. Caching reduces how much work the GPU has to do per request, but it doesn’t bypass GPU scheduling or contention.
Frequently Asked Questions
What is KV cache in the context of LLMs?
KV cache refers to the key and value tensors a model computes while processing a prompt. Storing and reusing them means the model doesn’t have to reprocess text it has already seen, which is the main lever for speeding up multi-turn conversations like agent sessions.
Do I need LMCache if I’m already using vLLM?
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Not necessarily. vLLM’s built-in prefix caching should stay on by default in nearly all cases, and its CPU offload feature handles cases where the cache outgrows GPU memory. LMCache becomes worth the added complexity mainly when you need the cache to survive a vLLM restart or crash, or when your workload outgrows available RAM and needs disk-backed storage.
Does caching make the model produce better answers?
No. Caching only affects speed, not output quality. It prevents redundant reprocessing of text the model has already read; it doesn’t change what the model knows or how it reasons.
What happens to the cache when vLLM restarts?
With vLLM’s own prefix cache or CPU offload, a restart clears everything because the cache lives inside the vLLM process. LMCache stores its data in a separate server process, so it can survive a vLLM restart and let the agent resume without rebuilding its cache from scratch, though this comes with its own memory-release delays during restart.
Does adding more agents make caching more or less valuable?
More valuable. Each additional agent sharing a GPU adds its own growing conversation history that would otherwise need full rereading every turn. Testing showed the relative speedup from caching grew larger as more agents ran concurrently, since uncached GPU time scales with both conversation length and agent count.
