How to Run Qwen3.8-Flash-Next Locally with llama.cpp
A practical guide to downloading, quantizing, and serving Qwen3.8-Flash-Next locally with llama.cpp, covering VRAM needs on an H100 and quad 3090 rig.
What is Qwen3.8-Flash-Next and why does it matter for local AI?
Qwen3.8-Flash-Next is an open-weight preview of the architecture that will become Qwen 4. It’s a 125 billion parameter model, but thanks to a mixture-of-experts style design, only around 6 billion parameters activate per token. That gap between total size and active compute is the whole point: you get a model with the knowledge footprint of something much larger while paying the inference cost closer to a small model. For anyone running local AI on consumer or workstation GPUs, that tradeoff is what makes a 125B model even worth attempting outside a datacenter.
TL;DR
- Qwen3.8-Flash-Next activates roughly 6 billion of its 125 billion parameters per token, which is what makes local inference on 1-4 GPUs plausible at all.
- The architecture mixes gated delta layers for cheap running memory with occasional Qwen sparse attention layers that scan chunks of the full context instead of every token, cutting the usual cost of long conversations.
- An engram embedding layer acts like a giant lookup table of common word patterns sitting in RAM, adding knowledge without slowing down the forward pass, and it’s the piece that makes CPU/RAM offloading strategies unusually effective for this model.
- Running it locally requires a specific unmerged llama.cpp build tied to Unsloth’s GGUF pull request, not the mainline release, so the exact commands matter more than usual.
- On a single Nvidia H100 (80GB VRAM), the quantized GGUF from Unsloth loaded and served at just over 61GB of VRAM.
- On a quad RTX 3090 rig (96GB VRAM total), the Unsloth UD-IQ4_XS quant at 93.7GB left almost no headroom, forcing heavy tensor override tricks to offload embeddings to system RAM.
- Benchmarks cited alongside the release show Flash-Next beating Qwen 3.7/3.8-27B and DeepSeek V4 Flash on most language and coding tests, and even edging out Claude Opus 4.6 on several, though it trails on multidisciplinary reasoning (HLE) and repo-level code generation.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
How do you actually install and serve it with llama.cpp?
Two things have to line up before this model will run: the right GGUF quant and a llama.cpp build that understands it.
Because Qwen3.8-Flash-Next uses a new architecture, mainline llama.cpp doesn’t support it out of the box yet. Unsloth published the GGUF conversion along with a pull request against llama.cpp, and that PR has to be built from source rather than pulled from a packaged release. The practical steps look like this:
- Clone and build llama.cpp from the branch tied to Unsloth’s pull request, not the default main branch.
- Install the Hugging Face CLI if you don’t already have it, since the model download command depends on it.
- Pull the quantized GGUF from Hugging Face using Unsloth’s repository.
- Launch
llama-serverpointing at the downloaded GGUF, setting context length, batch size, and GPU layer offload flags to match your hardware.
On the H100 setup, this was close to plug-and-play: one 80GB card, one quant, serve and go, landing at roughly 61GB of VRAM used. On a multi-GPU consumer rig, it’s a different story, which is where most of the tuning work actually happens.
What VRAM and hardware do you need?
The honest answer is “it depends on which quant and how many GPUs,” but two real-world data points help:
Single H100 (80GB VRAM): the Unsloth GGUF quant used in testing consumed just over 61GB of VRAM once served, leaving headroom for context and generation on one card.
Quad RTX 3090 rig (96GB VRAM total, 24GB per card): the UD-IQ4_XS quant weighs in at 93.7GB, which is smaller than the Q4_K_XL quants typically preferred for quality, but was chosen specifically because it barely fits inside four 3090s. With only a few gigabytes of VRAM to spare across the whole system once overhead is accounted for, this setup left almost no margin for error. The rig itself ran a Threadripper Pro 3945WX (12 cores, 24 threads), 8 DIMMs of DDR4-2133 on a WRX80 motherboard, and all four GPUs negotiated at full PCIe Gen4 x16.
The takeaway: an 80GB single-GPU card handles this model comfortably at this quant level, while a four-card consumer setup is right at the edge of its VRAM ceiling and needs deliberate tensor placement to stay stable.
How does the engram layer change local deployment?
The architecture’s engram embedding layer is essentially a large lookup table of common word patterns that sits in memory and feeds the model extra knowledge without adding to the compute cost per token. For local deployment, this matters because that lookup table doesn’t have to live on the GPU to be useful. It can be placed on system RAM using llama.cpp’s tensor override flags, freeing GPU VRAM for the layers that actually benefit from GPU compute.
In practice, on the quad 3090 rig, the per-token embed and engram embed were explicitly routed to CPU RAM using override tensor flags rather than left on GPU by default. That single change was what allowed a 93.7GB quant to run at all inside a 96GB VRAM budget, at the cost of relying on system RAM bandwidth for that portion of the workload. One useful rule of thumb offered from testing: roughly 48-64GB of system RAM, without much else competing for it, should comfortably support a 128K context window with this offloading approach.
What llama.cpp flags actually matter here?
A few flags come up repeatedly as consequential for this specific model, beyond the usual context size and batch size settings:
--override-tensorfor placing engram and embedding tensors onto CPU RAM instead of GPU, which is the key lever for fitting a near-96GB quant into a 96GB VRAM system.fit offrather than the defaultfit on. The default automatic layer-fitting mode caused instability in testing, while manually spreading layers and specifying which tensors go where produced a more predictable (if still tight) result.- KV cache quantization flags (Q8 for both key and value) paired with
--ropehandling, since some early compatibility flags for disabling rotary embedding attention are workarounds tied to this specific quant and may not be needed once the llama.cpp PR is fully merged. --parallel 1, which forces single-sequence serving. This simplifies debugging but makes concurrent agentic workloads harder to run.- Thread counts and batch sizes tuned conservatively (in one case, a 12-core/24-thread CPU was split as 6/12/4 across roles) because these workstation CPUs are not massively multithreaded compared to server-grade chips.
None of these are default-safe settings. They reflect manual tuning against a model and a build that hasn’t been fully merged into llama.cpp yet, and flag names or defaults may shift once mainline support lands.
Is it worth running locally right now?
For a single high-VRAM card like an H100, yes, it runs cleanly and the reported benchmarks are competitive: Flash-Next reportedly outperforms Qwen 3.7/3.8-27B and DeepSeek V4 Flash across most language and coding tests, and even beats Claude Opus 4.6 on a number of them, with the notable exceptions of multidisciplinary reasoning (HLE) and repo-level code generation, where it trails.
For a quad-3090 consumer rig, the honest answer from hands-on testing is “not yet comfortably.” The UD-IQ4_XS quant fits, barely, but pushing past roughly 80-90K tokens of context caused instability, and token generation speed degraded over a long run (dropping from around 28-29 tokens/second down into the low 20s as context filled). This is a preview architecture with an unmerged llama.cpp implementation, so rough edges around stability, especially at long context lengths, should be expected until support matures.
Frequently Asked Questions
What quantization format should I use for Qwen3.8-Flash-Next?
Unsloth’s GGUF conversions are the practical starting point. Two variants show up in testing: a UD-IQ4_XS quant at 93.7GB, chosen specifically to fit inside a 96GB VRAM budget across four GPUs, and standard Q4_K_XL style quants, which are larger but were the usual preference before VRAM constraints forced a smaller option.
Can I run this on a single consumer GPU?
Not at these quant sizes. Both demonstrated setups used either one 80GB datacenter GPU (H100) or four 24GB consumer GPUs (RTX 3090) totaling 96GB. A single 24GB card would need a much smaller quant, likely at a real cost to quality, since even the reduced UD-IQ4_XS quant is 93.7GB.
Why does this model need a special version of llama.cpp?
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
Because Qwen3.8-Flash-Next introduces new architectural components, including gated delta layers, Qwen sparse attention, and the engram embedding layer, that mainline llama.cpp doesn’t yet support. Unsloth’s GGUF conversion ships alongside a pull request that has to be built from source until it’s merged.
What’s the difference between Qwen3.8-Flash-Next and the hosted Qwen3.8 Flash API?
Qwen3.8-Flash-Next is the open-weight preview version, downloadable and quantizable for local use. Qwen3.8 Flash is the production hosted version of the same underlying architecture served through Qwen’s own API, useful for comparing quality against the local quantized build without hardware constraints.
How much system RAM do I need alongside the GPUs?
For a setup offloading engram and embedding tensors to CPU RAM, roughly 48 to 64GB of system RAM was cited as enough to safely support a 128K token context window, assuming that RAM isn’t heavily used by other processes at the same time.
