Speculative Decoding in Llama.cpp: How to Actually Speed Up Local LLMs
How draft models, NGL layer tuning, and quantization choice combine in Llama.cpp to speed up local LLM inference, based on real hardware tests.
Why local LLMs run slower than you’d expect
Running a large language model locally is rarely limited by raw compute. It’s limited by memory bandwidth. Every token a model generates requires streaming its entire set of weights past the chip once. A 27 billion parameter model quantized to around 18GB means that generating a single word involves moving 18GB of data through memory. Do that dozens of times a second and you hit a wall fast, especially on integrated GPUs that share memory with the CPU instead of having dedicated VRAM.
This is why a model that “runs” isn’t the same as a model that’s usable. A 27B coding model outputting four tokens per second is technically working, but nobody sticks with that pace when a cloud API responds instantly. The real question with local inference is never “does it run,” it’s “how do I get it to run fast enough to actually use.”
There are three practical levers for this in Llama.cpp: speculative decoding with a draft model, tuning how many layers get offloaded to GPU (NGL), and picking the right quantization format for what actually fits in VRAM. None of these require new hardware. They’re configuration choices, and testing on a small Panther Lake mini PC showed each one produces measurable, repeatable gains.
TL;DR
- Speculative decoding pairs a large model with a small, fast draft model that guesses several tokens ahead, and the big model verifies them in a single pass, which can more than double generation speed since verifying multiple tokens costs the same as verifying one.
- Draft depth matters, and going too aggressive backfires: pushing the guess window (Nmax) beyond the default of 3 made generation slower than not using speculative decoding at all in one test.
- NGL (number of GPU layers) controls how much of the model gets offloaded to VRAM, and letting a model slightly overflow available VRAM by even a couple gigabytes can crater speed by an order of magnitude or more.
- Auto-fitting in Llama.cpp is decent but not optimal: manually tuning NGL beat the automatic setting by a wide margin in a case where the model almost, but didn’t quite, fit in VRAM.
- Choosing a smaller quantization format that actually fits in VRAM outperformed manual layer-splitting entirely, roughly doubling both prompt processing and token generation speed with no measurable quality loss on a perplexity test.
- Perplexity testing is a useful but limited quality check: it can catch overall coherence loss but won’t reliably detect degradation in code correctness, arithmetic, or long reasoning chains, so quantization choices should be validated against your actual workload.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
What is speculative decoding and how does it work?
Speculative decoding uses two models instead of one: a large “target” model (the one you actually want output from) and a smaller, faster “draft model.” The draft model runs ahead and guesses several tokens at once. The target model then checks all of those guessed tokens in a single forward pass. Because a pass through the big model costs roughly the same whether it’s verifying one token or several, every guess that turns out correct is essentially free additional throughput.
Many popular model families on Hugging Face now ship a compatible draft model alongside the main weights, often adding only a small amount of extra disk space (in the neighborhood of 1 to 2GB). If a draft model is available for the model you’re running, there’s little reason not to use it. In one test on an integrated GPU, enabling a draft model took token generation from about 5 tokens per second to roughly 11, more than doubling speed for close to zero added cost.
How much draft depth should you actually use?
Llama.cpp exposes a flag for how many tokens ahead the draft model is allowed to guess, often referred to as Nmax or draft depth. It’s tempting to assume more guessing is always better, but that’s not how it plays out. Beyond a certain depth, the target model spends verification passes rejecting guesses that never had a real chance of being correct, and those wasted passes cost real time.
Testing across two very different devices, an integrated GPU and a discrete GPU roughly five times faster, both peaked at the same draft depth of 3, which happens to be Llama.cpp’s own default. Pushing the depth further made things worse: one test using a depth of 8 dropped generation speed below what it was without speculative decoding at all. The practical takeaway is simple: if a draft model is available, turn it on, but leave the depth setting at its default rather than assuming a bigger number helps.
How does NGL layer tuning affect speed, and why is it risky?
Large language models are structured as a stack of dozens of layers (for example, around 62 in a 27B-class model). Llama.cpp’s NGL flag (“number of GPU layers”) controls how many of those layers get copied into VRAM versus left on system memory or CPU. The common default advice in tutorials is to set NGL to a very high number (often written as 99) to force everything onto the GPU.
That advice breaks down the moment the model doesn’t fully fit. In one test, a model that was 1.7GB too large for available VRAM showed a smooth, expected speed increase as layers were added, climbing steadily up to around 14 tokens per second at 56 offloaded layers. Then, at 60 layers, throughput collapsed to under 1 token per second, a drop of roughly 17x from just four additional layers. There’s no gradual slope into that failure. It’s a cliff, caused by the driver having to shuffle a small number of overflow layers in and out of VRAM on every single token.
Letting Llama.cpp auto-fit the layer count avoided the worst of the collapse but still left meaningful performance on the table: manual tuning to 56 layers outperformed the automatic setting by a wide margin. The lesson is that “max out NGL” is safe advice for a model that comfortably fits, and actively dangerous advice for one that doesn’t.
Is downloading a smaller quantization better than tuning layers?
In direct comparison, yes, and by a wide margin. Rather than fighting to fit an oversized model into VRAM through careful layer tuning, simply dropping to a more efficient quantization format solved the problem outright. Moving from a Q4KM quantization (around 18GB) to an IQ4XS quantization (around 13GB), both still four bits per parameter but using a newer, more tightly packed encoding scheme, let the entire model live inside 16GB of VRAM with room to spare.
The results: prompt processing roughly doubled, and token generation roughly doubled as well, both improvements exceeding anything achieved through the entire layer-splitting exercise. The conclusion is straightforward: fitting the whole model in VRAM beats any workaround for a model that’s too big, and checking whether a smaller quantization exists should come before spending time tuning NGL by hand.
Does dropping quantization actually hurt output quality?
This is the obvious objection, and it deserves a real test rather than a gut feeling. Perplexity measurement offers one way to check: it feeds a model text it hasn’t seen and scores how “surprised” the model is by each word, with lower scores indicating better predictive fit.
Testing both quantization levels against a standard English-text benchmark, both models scored close to the same perplexity value, with the smaller IQ4XS version scoring marginally better, a difference small enough to be noise rather than a real signal. That means, for this comparison, the smaller file was measurably faster with no detectable quality cost.
The caveat matters: this test compared two specific quantized files from potentially different quantization pipelines, not “IQ4 versus Q4” as a general rule, and it only measured next-token prediction on plain prose. It says nothing about how quantization choice affects code generation, arithmetic, or long multi-step reasoning, all of which can degrade in ways perplexity on prose won’t catch. Anyone making this tradeoff for a coding or agentic workload should test against that actual workload rather than relying on a general prose benchmark.
Frequently Asked Questions
What is a draft model in speculative decoding?
A draft model is a smaller, faster model paired with a larger target model. It predicts several upcoming tokens, and the larger model verifies them in one pass, speeding up generation when guesses are correct.
What does the NGL flag do in Llama.cpp?
NGL sets how many of a model’s layers get loaded onto the GPU versus kept elsewhere. Setting it too high for available VRAM can cause severe slowdowns if the model doesn’t fully fit.
Is a higher draft depth always faster in speculative decoding?
No. Increasing draft depth too far leads to wasted verification passes on incorrect guesses. Testing showed the default depth of 3 outperformed much higher settings.
Does a smaller quantization always mean worse output quality?
Not necessarily. A perplexity test comparing two quantization levels of the same model showed nearly identical scores, though perplexity on plain text doesn’t reliably reveal quality loss in code or complex reasoning tasks.
Why does memory bandwidth matter more than GPU compute for local LLMs?
Every generated token requires streaming the entire model’s weights through memory. On systems where GPU and CPU share memory, this bandwidth becomes the primary bottleneck, not raw processing power.
