What Is KV Cache? Why It Makes AI Coding Agents Dramatically Faster
KV cache explained: how key-value caching stops LLMs rereading old prompts, and why the speedup multiplies as agent conversations grow longer.
What is KV cache, in plain terms?
KV cache is the memory a large language model keeps so it doesn’t have to reread a conversation from scratch every time it replies. “KV” stands for keys and values, the internal representations a model computes for every token it reads. Once computed, those values can be stored and reused. Without that cache, a model has to reprocess the entire prompt, including everything said in earlier turns, before it can write a single new word. With it, the model only has to process whatever is new since the last turn.
A useful way to think about it: the KV cache is the model’s notes from reading a document. If you keep your notes, you don’t reread the whole book next time, you just read the new chapter. Lose the notes, and you start over from page one.
TL;DR
- KV cache stores the model’s per-token internal state so it doesn’t have to reprocess the whole prompt on every turn of a conversation.
- Without caching, a 30-turn agent session can end up rereading over 2 million tokens in total, with each turn taking longer than the last.
- With caching, the same session never reprocesses much more than a small fraction of the growing conversation, cutting reread volume by roughly an order of magnitude.
- The speedup compounds with agent count: running eight agents on one GPU went from about one turn per minute each with no cache to around seven turns per minute each with caching on.
- Where the cache lives matters: a cache stored inside the inference server (like vLLM’s built-in prefix cache) disappears on restart, while a cache stored in a separate process can survive restarts and reconnect almost instantly.
- Tools like LMCache add that persistence, plus support for reusing cached chunks even when documents appear in a different order, at the cost of extra setup complexity and occasionally slower restarts.
- Caching doesn’t make a model smarter or more accurate. It only eliminates redundant work, so the benefit is pure latency and throughput, not quality.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
Why does the slowdown get worse the longer an agent runs?
An agent’s conversation history keeps growing. Every tool call, every file read, every reply gets appended to the context that has to be fed back into the model on the next turn. Without caching, the model reprocesses that entire growing history every single time.
In one tested session, turn-by-turn response time crept up from roughly a tenth of a second on turn one to nearly 19 seconds by turn 25, simply because there was more and more prior conversation to read before the model could respond. Scale that out: a 30-turn session with no caching ended up rereading about 115,000 tokens on the final turn alone, and over 2 million tokens cumulatively across the whole session. Every one of those tokens gets reprocessed from scratch, every single turn.
This is the core reason KV cache matters more for agents than for one-off chatbot questions. A single question-and-answer exchange only pays the “reread” cost once. An agent loop pays it on every turn, and the cost grows as the conversation grows. That’s a multiplying effect, not a flat one.
How much faster does caching actually make things?
Tested across multiple models and hardware setups, the difference between no cache and cache was consistently large, not marginal.
On a single large document-based query, no caching took around 1.35 seconds, while any of three caching approaches dropped that to roughly 112 to 122 milliseconds, about 11 to 12 times faster. That gap widens dramatically under concurrent load: with 16 clients generating around 144 concurrent requests, no caching took 187 seconds, the default GPU prefix cache brought that to 107 seconds, and CPU memory offload got it down to 69 seconds.
The agent-specific numbers are the most telling. In a scripted 30-turn coding agent session where the conversation grew to almost 100,000 tokens, late-turn latency with eight agents sharing one GPU averaged 104 seconds per turn with no caching. With caching enabled, that dropped to about 2.1 seconds, roughly 5.6 times more completed turns in the same timeframe. On a different GPU and model pairing, the throughput gain for eight concurrent agents reached 7.5 times. The pattern held: more agents sharing a GPU means a bigger relative payoff from caching, because the GPU’s reading workload without a cache scales up fast while the cache’s workload barely scales at all.
Where does the cache actually live, and why does that matter?
The KV cache has to sit in memory physically close to the GPU for fast access, and that memory isn’t free. The model itself occupies a large chunk of GPU memory, and whatever is left over becomes the cache budget. This is part of why running a model that nearly fills a GPU’s memory capacity on its own leaves no room for caching, and performance suffers as a result.
Four setups were compared in testing:
No cache rereads everything every turn, with no reuse at all.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
A built-in GPU prefix cache (the kind that ships enabled by default in inference servers like vLLM) reuses notes as long as the new prompt starts identically to a previous one, and keeps those notes in GPU memory inside the serving process.
CPU offload lets the same serving process spill cache contents into regular system RAM when GPU memory runs out, still tied to that one process.
A separate caching layer (tested here using the open-source project LMCache) keeps notes in an external server, outside the inference process, in RAM or even on NVMe drives.
That last distinction turned out to be the deciding factor in one specific scenario: process restarts. When the inference server dies or gets restarted, for example after a crash, a config change, or a routine deploy, any cache stored inside that process is gone. In testing, right after a restart, the built-in prefix cache and CPU offload both dropped back to roughly 1.35 seconds of cold reprocessing, as if caching had never existed. The externally-stored cache, unaffected by the restart because it lived in a different process entirely, delivered a 141-millisecond response immediately after reconnecting, with the vast majority of tokens still served from cache.
Is a separate cache layer worth the added complexity?
Not always. If the entire cache fits comfortably in GPU memory, the default built-in prefix caching that ships with servers like vLLM is enough, and it’s already on by default. If the cache occasionally overflows GPU memory but still fits in system RAM, built-in CPU offload performs about as well as an external cache layer in most throughput and latency tests.
The case for an external, persistent cache layer shows up in more specific conditions: frequent server restarts where you don’t want every agent starting cold again, caches too large to fit in RAM at all (where spilling to NVMe disk still kept response times fast and consistent even after a restart), or workloads like retrieval-augmented generation where the same documents get reused but in different orders across requests, something plain prefix caching can’t handle well since it only matches prompts that start identically.
The tradeoffs are real. An external cache layer adds a separate running service, more configuration, and in testing occasionally caused slower restarts when the dead server’s memory wasn’t released quickly. Support for newer or architecturally unusual models (hybrid attention designs, for instance) also lagged behind, requiring rebuilds or workarounds that may or may not be fixed by the time you read this. Caching never changes what the model knows or how accurate it is. It only removes redundant computation, so the entire benefit is speed and cost, nothing else.
Frequently Asked Questions
What do the “K” and “V” in KV cache stand for?
Keys and values. They’re internal numerical representations the model computes for each token as it processes a prompt. Storing them lets the model skip recomputing that work for tokens it has already seen.
Does KV cache make an AI model smarter or more accurate?
No. It has no effect on output quality. It only prevents the model from redoing identical computation it has already done, which saves time and compute, not correctness.
Why does caching matter more for AI agents than for simple chatbot queries?
One coffee. One working app.
You bring the idea. Remy manages the project.
A single question-answer exchange only benefits from caching once. An agent running many turns in a loop reprocesses its growing conversation history on every turn without caching, so the wasted work compounds turn after turn, and the time saved by caching compounds right along with it.
What happens to the KV cache if the server restarts?
If the cache lives inside the inference server’s own process (like a default built-in prefix cache), a restart wipes it and the next request has to reprocess everything from scratch. A cache stored in a separate external process can survive the restart and be reused immediately once the server reconnects.
Is enabling KV caching difficult to set up?
Built-in prefix caching in common inference servers like vLLM is typically already enabled by default and requires no extra setup. External, persistent caching layers require running an additional service and more configuration, and may lag in support for newer or non-standard model architectures.
