Run Qwen3.8-27B Faster Locally With Speculative Decoding
How to speed up Qwen3.8-27B locally using DSpine or DFlash2 draft models with SGLang, plus VRAM needs and benchmark numbers.
How do you make Qwen3.8-27B generate text faster without changing its output?
You attach a small draft model to it and serve both through SGLang using speculative decoding. The draft model guesses several tokens ahead in parallel, the big model verifies the guesses in a single pass, and every correct guess is accepted for free. Two draft models currently target Qwen3.8-27B: Tencent’s DSpine and the Z-Lab team’s DFlash2. In independent testing on a single H100 GPU, DSpine pushed throughput from a baseline of roughly 33 tokens per second (no speculative decoding) up to around 150 tokens per second, beating DFlash2’s roughly 129 tokens per second on the same hardware and prompts.
TL;DR
- Speculative decoding pairs a large model with a small “draft” model that guesses multiple tokens ahead, letting the large model verify a whole block in one slow pass instead of generating one token at a time.
- DSpine, built by Tencent, is a five-layer, roughly 2-billion-parameter draft model that drafts in blocks of 16 tokens and passes predicted features between adjacent positions to keep guesses consistent.
- DFlash2, from the Z-Lab team, drafts 8 tokens at a time and uses small convolutions to blend neighboring guesses before picking the best path.
- In a head-to-head test on an H100 GPU with identical prompts and settings, DSpine averaged about 150 tokens/sec versus DFlash2’s 129 tokens/sec, a roughly 16% edge overall.
- VRAM usage differs: DFlash2 consumed about 73GB on an 80GB H100, while DSpine consumed about 78GB due to extra flags for its verification and caching setup.
- On Tencent’s own benchmarks, DSpine’s acceptance length (how many drafted tokens get accepted per round) beats DFlash2 across GSM8K, MATH-500, HumanEval, MBPP, and agent benchmarks like SWE-bench Pro and WorkBuddy.
- Both draft models run through SGLang, and installation is a straightforward git clone and pip install, but the serving command needs several specific flags to match each draft model’s design.
What is speculative decoding and why does it speed things up?
A standard large language model generates one token at a time. Each token requires a full forward pass through the entire network, which is slow, especially for a 27-billion-parameter model like Qwen3.8-27B. Speculative decoding breaks that bottleneck by introducing a much smaller “draft” model that proposes several tokens in advance. The big model then checks all of those proposed tokens in a single pass, accepts the ones that match what it would have generated anyway, and only falls back to normal one-token generation at the first point where the draft guessed wrong.
The final output is identical to what the big model would have produced on its own. Nothing about quality or correctness changes. The only thing that changes is speed, and the speedup scales with how many draft tokens get accepted per round. A draft model that averages 7 accepted tokens per block is doing roughly seven times less “expensive” work for the big model than generating token by token.
How do DSpine and DFlash2 differ?
Both are small helper models trained specifically to draft tokens for Qwen3.8-27B, but they handle the guessing differently.
DSpine, from Tencent, is a five-layer draft model with a block size of 16 tokens and an 8,192-token sliding window. Its trick is passing predicted features between adjacent positions at every layer, so by the time it reaches the end of the network, each guess has been conditioned on its neighbors’ guesses. It then uses a lookup table to pick tokens left to right. According to Tencent’s model card, this gets DSpine to an average acceptance length of 7.20 tokens on GSM8K and 6.49 on MATH-500 at temperature 0, both notably ahead of DFlash2’s 5.69 and 5.49 respectively on the same tasks.
DFlash2, built by the Z-Lab team, drafts a full block of 8 tokens in parallel in one pass, then uses small convolutions to blend neighboring guesses before a final path-picking step selects the best sequence. It’s a different mechanism for solving the same problem: keeping a block of simultaneous guesses internally consistent rather than generating them one by one.
In practice, the official default draft block size for DFlash2 is 8 tokens, while DSpine defaults to 16, which partly explains why DSpine tends to pull ahead on raw acceptance length and throughput.
How much faster is DSpine than DFlash2 in practice?
In a controlled test on a single H100 GPU (80GB VRAM), both models were served through SGLang with matched memory and context settings, then hit with the same 16 prompts (8 short, 8 requiring longer or reasoning-heavy answers), each run twice for 32 total requests at temperature zero for repeatability.
Big news
firstNov 17
DFlash2 produced about 129 tokens per second overall: 133 on short prompts, 128 on longer ones, with individual requests ranging from 91 to 168 tokens/sec. DSpine produced about 150 tokens per second overall: 152 on short prompts, 150 on longer ones, with individual requests ranging from 88 to 227 tokens/sec. That puts DSpine roughly 14-16% faster across the board in this particular test.
Tencent’s own published benchmarks go further, measuring end-to-end agent throughput against plain autoregressive decoding (no speculative decoding at all). On WorkBuddy Code under high-reasoning settings, DSpine hit 3.18x the speed of autoregressive decoding versus DFlash2’s 2.85x. On WorkBuddy Office, DSpine reached 3.55x versus DFlash2’s 3.08x. Across every benchmark Tencent tested, including SWE-bench Pro and tau2-bench, DSpine outperformed DFlash2, though both comfortably beat plain decoding by a factor of roughly 2.5x to nearly 4x.
What does it cost in VRAM, and is it worth the setup?
Both draft models add meaningful memory overhead on top of serving Qwen3.8-27B itself. In testing, DFlash2 consumed about 73GB of VRAM on an 80GB H100 with its standard serving configuration. DSpine consumed a bit more, just over 78GB, because its setup requires extra flags: dynamic verification referencing DeepSpec-style checks, and a feature called ReplaySSM for caching. On an 80GB card, both fit, but there’s very little headroom left over, which matters if you’re also running other processes or want larger context windows.
Whether it’s worth the setup depends on what you’re doing. If you’re running Qwen3.8-27B for anything latency-sensitive (agent loops, coding assistants, interactive chat) the throughput gains are large enough to notice immediately: going from roughly 33 tokens/sec unassisted to 150 tokens/sec with DSpine is nearly a 5x improvement. If you’re running batch jobs where latency doesn’t matter and you’re already VRAM-constrained, the extra memory draw from either draft model may not be worth it.
How do you set it up with SGLang?
Both draft models are installed and served in a similar way. For DSpine, Tencent provides a GitHub repository you clone and install with pip, then launch SGLang with a specific set of flags pointing at both the base Qwen3.8-27B weights and the DSpine draft checkpoint. Key settings include the speculative algorithm flag set to DSpine, 16 draft tokens, BF16 precision, FlashAttention 3 as the backend, and flags enabling the ReplaySSM caching mechanism with a fixed Mamba cache size. Context length, memory fraction, and max concurrent requests are all configurable and should be matched across runs if you want to benchmark fairly against DFlash2.
DFlash2 follows the same general pattern but with its own draft model path and a default of 8 draft tokens instead of 16. The underlying SGLang serving mechanics (attention backend, sampling backend, reasoning and tool-call parsers) stay largely the same across both, since both are designed to slot into the same inference stack without changing anything about Qwen3.8-27B’s own weights or behavior.
Frequently Asked Questions
Does speculative decoding change the model’s output quality?
No. The large model still verifies every token the draft model proposes. Rejected guesses are discarded and regenerated normally, so the final text is identical to what the base model would produce without speculative decoding. Only generation speed changes.
Which draft model should I use, DSpine or DFlash2?
Based on both Tencent’s published benchmarks and independent testing on an H100, DSpine consistently produced higher acceptance lengths and higher throughput than DFlash2 across coding, math, reasoning, and agent tasks. DFlash2 uses slightly less VRAM, which could matter on tighter hardware.
How much VRAM do I need to run Qwen3.8-27B with a draft model?
In testing on an 80GB H100, DFlash2’s setup used about 73GB and DSpine’s used about 78GB, including the base model, draft model, and context/caching overhead. Smaller GPUs would need reduced context length or batch size to fit either configuration.
Do I need special hardware to use these draft models?
They were tested on an Nvidia H100 with 80GB of VRAM. The underlying technique works on any GPU with enough memory to hold both the base Qwen3.8-27B model and the small draft model simultaneously, plus room for the context cache.
What is “acceptance length” and why does it matter?
It’s the average number of drafted tokens the big model accepts per verification round. Higher acceptance length means fewer wasted guesses and more tokens produced per expensive forward pass through the large model, which directly translates to faster generation.
