Running a 27B Model on a Mini PC: What the Benchmarks Show
Real benchmarks of a 27B model on a tiny mini PC and GPU dock, covering quantization, VRAM fit, NGL tuning, and speculative decoding gains.
What happens when you run a 27B model on a mini PC?
A 27 billion parameter model can run on a palm-sized mini PC, but “run” and “run well” are different questions. On integrated graphics alone, a 27B model quantized to 4 bits generates text at roughly 5 tokens per second, slow enough that most people would give up and go back to a cloud API. The real story is what happens next: speculative decoding, careful GPU layer offloading, and a smarter quantization format can more than double or even quadruple that speed, turning a device that fits in one hand into something that handles a serious coding model at usable speeds.
This was tested on the Kadas Mind Pro, a mini PC built around Intel’s Panther Lake Core Ultra X7 358H chip with 64GB of shared memory and an ARC B390 integrated GPU. The test model was Qwen’s 27B variant, quantized down from 16-bit weights to 4-bit formats (Q4KM at about 18GB, and IQ4XS at about 13GB). The results expose exactly where bottlenecks show up when you push a large model onto small, constrained hardware, and they’re relevant to anyone trying to run large language models locally instead of in the cloud.
TL;DR
- Integrated GPU throughput on this hardware topped out around 5 to 6 tokens per second for text generation across Vulkan, SYCL, and OpenVINO backends, while prompt processing (reading the prompt) hit as high as 446 tokens per second on OpenVINO.
- Memory bandwidth, not compute, is the real bottleneck for token generation on integrated graphics, since every generated token requires streaming the entire set of model weights past the chip once.
- Speculative decoding using a small draft model more than doubled generation speed for free, taking the integrated GPU from about 5 tokens per second to 11, and the default draft depth of 3 tokens ahead (Llama.cpp’s default) turned out to be the sweet spot on both the integrated GPU and a discrete GPU.
- Manually tuning the NGL (number of GPU layers) flag beat the “load everything” default by a wide margin when a model didn’t quite fit in VRAM, with hand-tuned layer counts hitting 14 tokens per second versus 10.5 for auto-fit, and forcing all layers on (NGL 99) collapsing to under 1 token per second.
- Dropping to a smaller quantization format fixed the VRAM fit problem more cleanly than any layer-splitting trick, cutting the model from 18GB to 13GB and roughly doubling both prompt processing and generation speed with no measurable quality loss on a perplexity test.
- The high-speed PCIe dock connection was massively overprovisioned for this workload, since steady-state token generation only used about 2 megabytes per second of a link capable of 15.8 gigabytes per second.
One coffee. One working app.
You bring the idea. Remy manages the project.
Why is memory bandwidth the bottleneck, not the chip itself?
Every token a language model generates requires reading the entire set of model weights from memory at least once. For an 18GB model, that means moving 18 gigabytes through the memory pipeline for every single word produced. On integrated graphics, the CPU and GPU share one pool of system memory, so the GPU spends most of its time waiting on data rather than computing.
The math bears this out directly: model size (17.66GB) divided by measured token rate (5.42 tokens per second) works out to roughly 100GB per second of real, measured memory bandwidth on this device. That’s the actual throughput ceiling, not a marketing spec. No backend, whether Vulkan, SYCL, or OpenVINO, can generate tokens faster than that pipe allows, because the constraint is architectural, not a software inefficiency.
This is the core lesson for anyone evaluating a mini PC or laptop for local LLM inference: raw compute numbers matter less than memory bandwidth once you’re generating tokens at any meaningful model size.
What is speculative decoding and why does it help?
Speculative decoding uses a small, fast “draft model” to guess several tokens ahead, then has the large model verify all those guesses in a single pass. Because a forward pass through the big model costs roughly the same whether it checks one token or five, every guess that turns out correct is effectively free throughput.
On the integrated GPU, this technique took generation speed from about 5 tokens per second to 11, more than doubling it without any hardware change. Many quantized models on Hugging Face already ship a compatible draft model alongside the main weights, often adding only around 1.5GB, making this close to a free upgrade for anyone already downloading GGUF files.
The catch is tuning the draft depth (how many tokens ahead the draft model is allowed to guess). Both the integrated GPU and a discrete GPU (an RTX 5060 Ti tested here) peaked at a draft depth of 3, even though one device was roughly five times faster than the other. Pushing the depth higher didn’t help, it hurt: a test at draft depth 8 dropped speed to about 3.5 tokens per second, slower than not using speculative decoding at all. Llama.cpp’s own default draft depth is already 3, so the practical advice is simple: turn speculative decoding on and leave the depth setting alone.
How much does GPU layer offloading (NGL) actually matter?
A transformer model isn’t one solid block, it’s a stack of layers (62 in this case), and those layers can be split between system RAM and GPU VRAM independently. Llama.cpp controls this with the NGL flag (number of GPU layers), and how many layers get offloaded to VRAM has an outsized effect on speed once a model is close to, or slightly over, the VRAM ceiling.
In this test, a Q4KM quantized model at roughly 18GB didn’t fit inside a 16GB VRAM discrete GPU (the RTX 5060 Ti in the dock). Sweeping the NGL value revealed a smooth climb in performance, from 4.54 tokens per second at zero offloaded layers, to nearly 11 at 48 layers, to almost 14 at 56 layers. Then, at 60 layers, performance collapsed to 0.82 tokens per second, seventeen times slower than the 56-layer result, with no gradual slope, just a wall.
The common tutorial advice to set NGL to 99 (offload everything) actually overrides Llama.cpp’s built-in autofit logic and can badly backfire when a model doesn’t fit. Letting Llama.cpp autofit produced 10.48 tokens per second, but manually tuning to 56 layers pushed that to nearly 14. The practical takeaway: NGL 99 is not automatically the fastest setting, and for models that are borderline oversized for available VRAM, a manual sweep is worth the time.
Does a smaller quantization hurt output quality?
The cleaner fix for a model that doesn’t fit in VRAM turned out to be shrinking the model itself. Dropping from Q4KM (18GB) to IQ4XS (13GB), a newer 4-bit packing scheme rather than a lower bit depth, let the entire model live inside 16GB of VRAM. The results: prompt processing roughly doubled (474 to 943 tokens per second), and generation speed roughly doubled as well (14 to 27 tokens per second).
To check whether that speed came at a quality cost, a perplexity test was run using Wikitext-2 plain English text, a standard method that measures how “surprised” a model is by real text it hasn’t seen, where lower scores indicate a better fit. Both quantizations scored about 6.8, with the smaller model measuring marginally better, a difference small enough to be noise rather than a real gap.
That result comes with real caveats. The two models were built by different quantizers, so this specific comparison isn’t a clean IQ4-vs-Q4 test in general. And perplexity on English prose doesn’t capture how quantization affects code generation, arithmetic, or long reasoning chains, the tasks people actually care about with a 27B coding-capable model. Anyone using a quantized model for production coding work should test their own workload before assuming smaller is safe.
Is the high-speed GPU dock connection worth it?
The mini PC’s discrete GPU dock connects over a proprietary link that turns out to be PCIe 4.0 at 8 lanes, roughly 15.8 gigabytes per second of bandwidth. That link matters a lot when loading a model (moving 17GB of weights in about 3 seconds), but almost not at all during actual text generation. Once weights are resident in VRAM, steady-state token generation peaked at 19 megabytes per second and averaged around 2 megabytes per second, a small fraction of a percent of the link’s total capacity.
In practical terms, the connection would need to be roughly 8,000 times slower before it became a bottleneck for generation. Even Thunderbolt, which is about 5 times slower than this proprietary link on paper, would still be nowhere near a limiting factor for inference workloads. The high-speed dock connection matters most for fast model loading and switching, not for generation throughput itself.
Frequently Asked Questions
Can a 27B parameter model run on a mini PC without a discrete GPU?
Yes. On integrated graphics alone (no external GPU), a 27B model quantized to 4 bits generated text at roughly 5 to 6 tokens per second depending on backend, and processed prompts at up to several hundred tokens per second using OpenVINO. It’s usable but slow compared to a discrete GPU setup.
What is NGL and why does it matter for local LLM performance?
NGL stands for number of GPU layers, a Llama.cpp setting controlling how many of a model’s transformer layers get loaded into GPU VRAM versus system RAM. Setting it too high for available VRAM can cause a severe performance collapse rather than a gradual slowdown, so tuning it manually often beats defaults for models near the VRAM limit.
Does speculative decoding reduce output quality?
No. Speculative decoding uses a small draft model to propose tokens that the full model then verifies, so the final output still comes from the large model’s own predictions. It changes speed, not the model’s answers, provided the draft depth is set reasonably (around 3, matching Llama.cpp’s default).
Is a smaller quantization always safe to use?
Not universally. In this specific test, perplexity scores on plain English text showed no meaningful quality loss when moving from an 18GB to a 13GB quantization. But perplexity on prose doesn’t reliably predict performance on code, math, or long reasoning chains, so quantization choices should be validated against the actual workload.
Does a fast GPU dock connection speed up token generation?
Not much. Model loading benefits from a fast connection, but once weights sit in VRAM, generating text used only a tiny fraction of the available bandwidth on a PCIe 4.0 x8 link. Even a much slower connection like Thunderbolt would likely be sufficient for generation alone.
