How to Run Qwen3.8-27B with DFlash2 on vLLM or SGLang
Set up speculative decoding for Qwen3.8-27B with the DFlash2 draft model using vLLM or SGLang, with H200 benchmark numbers included.
What is DFlash2 and why does it speed up Qwen3.8-27B?
DFlash2 is a draft model built specifically for speculative decoding with Qwen/Qwen3.8-27B. It doesn’t answer prompts on its own. Instead, it sits in front of the full 27B model inside a speculative decoding server, guesses a block of upcoming tokens in one pass, and lets the target model verify them in a single forward pass instead of generating them one at a time. On an H200 GPU running SGLang, this pushed throughput up to 3.43x over standard autoregressive decoding on math and coding benchmarks, with no loss in output quality.
TL;DR
- DFlash2 is a drafter, not a chat model: it only works paired with Qwen/Qwen3.8-27B inside a speculative decoding server, and it has no standalone use.
- Decoding stays lossless: greedy output from the speculative pipeline matches what the base 27B model would produce on its own, and sampling preserves the same output distribution.
- The architecture is block diffusion, not next-token drafting: DFlash2 predicts a whole block of candidate tokens at once, keeps multiple options per position, and uses a lightweight selector to pick one coherent sequence through them.
- Benchmarks on a single H200 show real gains: acceptance length (tokens accepted per verification step) reached 5.46 on GSM8K and 5.28 on MATH-500, both ahead of Qwen3.8’s built-in MTP drafter and the community DSpark drafter.
- Speedups shrink as concurrency rises: at concurrency 1, DFlash2 hit 3.11x to 3.43x speedup depending on task; at concurrency 32, gains dropped to 1.01x to 1.45x because the GPU is already compute-bound.
- Setup requires bleeding-edge builds: both the vLLM and SGLang integrations currently need installation from git branches or open pull requests rather than stable pip releases.
- Hardware context matters: the published benchmarks all run on a single NVIDIA H200 with FlashAttention 3, so results on smaller GPUs or multi-GPU setups will differ.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
How do you set up DFlash2 with SGLang?
SGLang has direct support for DFlash2 through a dedicated speculative algorithm flag. Install the SGLang package straight from its GitHub repository (the stable release doesn’t yet include DFlash2 support), then launch the server pointing at both the target model and the draft model:
pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git#subdirectory=python"
python -m sglang.launch_server \
--model-path Qwen/Qwen3.8-27B \
--speculative-algorithm DFLASH \
--speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \
--speculative-num-draft-tokens 8The --speculative-num-draft-tokens 8 flag sets the speculation block size. In the published benchmarks, this configuration produced 7 draft tokens per verification step, which is the setting used across all the acceptance-length and throughput numbers below.
How do you set up DFlash2 with vLLM?
vLLM support for DFlash2 lands through an open pull request rather than a merged, stable release, so the install command points at that specific PR branch:
pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head"
vllm serve Qwen/Qwen3.8-27B \
--speculative-config '{
"method": "dflash",
"model": "incoai/Qwen3.8-27B-DFlash2",
"num_speculative_tokens": 7
}'The configuration is passed as a JSON blob via --speculative-config, specifying the method (dflash), the draft model repo, and the number of speculative tokens per step. Because this depends on an unmerged PR, expect the exact branch reference to change or disappear once the feature merges into vLLM’s main branch. Check the vLLM repository for the current status before pinning this in a production deployment.
What hardware do you need to run this?
The official benchmarks were run on a single NVIDIA H200 with FlashAttention 3 handling both the target model’s attention and the draft model’s attention. Qwen3.8-27B at 27 billion parameters plus a draft model on top means you need enough VRAM to hold both models simultaneously, along with KV cache for whatever context lengths and concurrency you’re planning to serve. The H200’s 141GB of memory gives it headroom for this combination even at higher batch sizes. If you’re working with smaller GPUs, expect to either reduce concurrency, use quantization, or hit memory limits before you hit the H200 numbers shown here. No specific minimum VRAM figure was published, so plan for a generous margin above the base 27B model’s footprint.
How much faster is DFlash2 than standard decoding?
At concurrency 1 (a single request at a time), DFlash2 delivered the largest relative gains: 3.43x on GSM8K, 3.34x on MATH-500, 3.11x on HumanEval, 3.29x on MBPP, and 2.67x on MT-Bench, compared to plain autoregressive decoding. These numbers beat both Qwen3.8’s built-in seven-token MTP drafter and the community DSpark drafter across every task tested.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
As concurrency increases, the advantage narrows. At concurrency 8, DFlash2 speedups ranged from 2.27x to 2.85x. At concurrency 32, they dropped further, to between 1.01x and 1.45x. This pattern is expected: speculative decoding helps most when the GPU has spare compute capacity to verify draft tokens cheaply. At high concurrency, the GPU is already saturated with real work, so there’s less idle capacity for verification to exploit. Notably, at concurrency 32, MTP and DSpark actually underperformed plain autoregressive decoding on some tasks (values below 1.00x), while DFlash2 stayed above the autoregressive baseline across every benchmark.
What makes the block-diffusion approach different from typical draft models?
Most speculative decoding drafters work autoregressively themselves, predicting one token, then the next, conditioned on the first, and so on, before handing a batch to the verifier. DFlash2 instead predicts an entire block of tokens in parallel in a single forward pass, keeping the top candidates at every position within that block. A separate, lightweight selector module then traces through those candidates to pick one consistent, coherent sequence.
The model card also mentions “two-tap dynamic convolutions” built into the backbone, designed to prevent draft quality from degrading toward the end of each block. This is a common failure mode in block-style drafting: predictions further from the start of the block have less context to work with and tend to get less accurate. The convolution mechanism is aimed at keeping acceptance rates high even for later positions in each draft block.
Despite the more aggressive parallel drafting, decoding remains lossless: greedy decoding through the speculative pipeline produces output identical to running the 27B model alone, and sampling-based generation preserves the same probability distribution the target model would produce on its own. Speculative decoding here is purely a throughput optimization, not an approximation.
Frequently Asked Questions
Can I use DFlash2 without Qwen3.8-27B?
No. DFlash2 is trained specifically as a draft model for Qwen/Qwen3.8-27B and has no function as a standalone language model. It only operates inside a speculative decoding server alongside that specific target model.
Does speculative decoding with DFlash2 change the output quality?
No. The setup is lossless: greedy decoding matches the base model’s output exactly, and sampling preserves the target model’s original output distribution. The speedup comes from verifying multiple drafted tokens per step rather than generating one token per forward pass.
Is DFlash2 always faster than the built-in MTP drafter?
In the published H200 benchmarks, yes, across every task and every concurrency level tested (1, 8, and 32), DFlash2 produced higher throughput and higher acceptance length than both Qwen3.8’s built-in MTP and the community DSpark drafter. The margin is largest at low concurrency and narrows as concurrency increases.
Do I need a specific GPU to run this setup?
The official benchmarks used a single NVIDIA H200 with FlashAttention 3. No official minimum hardware spec was published, but running a 27B parameter model plus a draft model requires enough VRAM to hold both plus KV cache, so smaller GPUs may need reduced concurrency or quantization to fit.
Why do speedups drop at higher concurrency?
Speculative decoding gains come from using spare GPU compute to verify drafted tokens cheaply. At low concurrency, the GPU has idle capacity to exploit. At high concurrency (32 concurrent requests in the benchmarks), the GPU is already busy with real generation work, leaving less room for speculative verification to add value, so the relative speedup shrinks.
