DFlash 2: Run Qwen3.8-27B at 2x Speed with Speculative Decoding
DFlash 2 speeds up Qwen3.8-27B inference roughly 2x on a single A100 using speculative decoding in SGLang, with no output quality loss.
What is DFlash 2 and why does it matter?
DFlash 2 is a speculative decoding method built on top of Qwen3.8-27B that roughly doubles token generation speed without changing the model’s output quality. It works by pairing the main model with a small draft model that guesses several tokens ahead in parallel, then lets the big model verify them all in a single forward pass. Released by the z-lab research team on Hugging Face as z-lab/Qwen3.8-27B-DFlash2, it plugs into inference engines like SGLang (with support noted for vLLM and llama.cpp) and requires no retraining or architecture changes to the base model to get the speedup.
It’s also a good example of what happens after a major open weight release. Alibaba shipped Qwen3.8-27B, and within a short window the community had already quantized it, distilled it, and now layered a speculative decoding scheme on top that nearly doubles throughput on the same hardware.
TL;DR
- DFlash 2 is a speculative decoding technique for Qwen3.8-27B that uses a small draft model to propose multiple tokens per step, which the full model verifies in one pass.
- In a live benchmark on a single Nvidia A100, baseline SGLang decoding of Qwen3.8-27B averaged around 28.9 tokens per second, while DFlash 2 pushed the same prompts to about 59.1 tokens per second, roughly 2x faster.
- The technique claims the top correct token is almost always within the top 16 candidates at each position (about 99% of the time at the first slot), which is why keeping 16 options per position and scoring paths between them works so well.
- A lightweight path selector and a small convolutional layer let the draft model’s guesses stay coherent across positions while adding only about 1% latency overhead.
- Running the full setup (Qwen3.8-27B plus the DFlash 2 draft model) consumed just over 77GB of VRAM in the demonstrated configuration, which can be tuned down by reducing KV cache size.
- Installation is done through SGLang from a specific branch, and the model weights are hosted on Hugging Face under an Apache 2.0 license.
- Output quality is reported as identical to standard autoregressive decoding, since DFlash 2 only changes how tokens are proposed and verified, not the model’s actual parameters or generation logic.
How does speculative decoding actually work?
Normal LLM inference generates one token at a time. Every single token requires a full forward pass through billions of parameters, which is expensive and slow when done token by token.
Speculative decoding breaks that pattern. Instead of relying solely on the big model, a smaller, cheaper draft model guesses several tokens ahead in one go. The large model then checks all of those guesses in a single verification pass. If the guesses are correct, you get multiple tokens for the cost of one verification step. If a guess turns out wrong, everything after that point gets discarded, but everything before it is kept. The economics favor you either way: bad guesses cost almost nothing, and good guesses are essentially free tokens.
This is not a new idea in the field, but DFlash 2 pushes it further by making the drafting step itself parallel rather than sequential.
What makes DFlash 2 different from standard speculative decoding?
Standard speculative decoding still has the draft model guess tokens one at a time, just ahead of the main model. DFlash 2 predicts all draft positions in a single pass instead of guessing them sequentially.
The key insight behind DFlash 2 is that even when the single top-ranked guess at a given position is wrong, the correct token is almost always sitting somewhere in the top 16 candidates, about 99% of the time at the first position. Rather than throwing away that information, DFlash 2 keeps all 16 candidates at every slot and runs a lightweight path selector that scores neighboring pairs of candidates to find the most coherent sequence through them.
To keep that parallel structure from producing disjointed guesses, DFlash 2 adds a small convolutional component (described as a “two-tap” convolution) that lets each position take context from the one before it, without breaking the parallelism that makes the whole scheme fast. The net effect: roughly one additional accepted token per verification pass, for about 1% added latency. That’s a strong trade given the throughput gain.
How do you install and run DFlash 2 locally?
The setup demonstrated used an Ubuntu machine with a single Nvidia A100 GPU. The broad steps:
- Create an isolated environment. A
uvvirtual environment was used, though conda works equally well. - Install SGLang from the specific branch that supports DFlash 2’s speculative decoding flags. This is not yet in a default stable release, so pulling from the correct branch matters.
- Launch the SGLang server pointing at the
z-lab/Qwen3.8-27B-DFlash2checkpoint, with speculative decoding enabled and a draft token count configured (eight draft tokens was the setting used, meaning the draft model guesses eight tokens ahead before the full model verifies them in one shot). - First run downloads the weights, after which the server serves the model normally over its API.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
VRAM usage for the full setup, including both the 27B parameter Qwen model and the smaller draft model, came in just over 77GB. Reducing the KV cache allocation is one straightforward way to bring that down if you’re working with less headroom.
How much faster is it in practice?
The benchmark methodology was simple and fair: run the same five prompts twice, once with plain autoregressive decoding in SGLang and once with DFlash 2 speculative decoding enabled, then measure tokens per second for each.
Baseline decoding (no DFlash 2) averaged about 28.9 tokens per second, consistent across all five prompts. With DFlash 2 turned on, the same prompts on the same hardware averaged about 59.1 tokens per second, a bit more than double. One prompt (a “Silk Road” style test) reportedly hit around 72 tokens per second, which reflects a case where the draft model’s guesses ran correct for a long stretch and the verifier accepted nearly all of them without rejection.
That gap between the fastest and average cases is expected with speculative decoding generally: the speedup depends on how predictable the ongoing text is, since easier, more pattern-following continuations let the draft model guess further ahead correctly.
Is DFlash 2 worth using for local inference?
For anyone already running Qwen3.8-27B locally on capable hardware, DFlash 2 looks like a low-risk way to roughly double throughput. It doesn’t touch the base model’s architecture or weights in a way that changes outputs, it doesn’t require retraining anything on your end, and the added latency overhead from the path selection process is small (around 1%) relative to the gains.
The main considerations are practical: you need SGLang from the right branch (this isn’t yet a one-line stable install), you need enough VRAM to hold both the main model and the draft model (about 77GB in the tested setup, tunable via KV cache size), and you’re relying on a community-released research checkpoint rather than an official Qwen or SGLang feature at this stage. For production deployments, it’s worth validating stability and output parity on your own workloads before fully committing, but the demonstrated results suggest the throughput claims hold up under direct, apples-to-apples testing.
Frequently Asked Questions
What hardware do you need to run DFlash 2 with Qwen3.8-27B?
The demonstrated setup used a single Nvidia A100 GPU with just over 77GB of VRAM consumed by the model and draft model combined. Reducing KV cache allocation can lower that requirement if you have less VRAM available.
Does DFlash 2 change the model’s output quality?
No. It changes how tokens are generated and verified, not the underlying model weights or logits used for final acceptance, so output quality is reported as identical to standard decoding.
Is DFlash 2 available in vLLM or llama.cpp, or only SGLang?
The demonstrated install and benchmark used SGLang from a specific development branch. Support in vLLM and llama.cpp has been mentioned as available or in progress, but the hands-on testing here was done specifically through SGLang.
How many draft tokens does DFlash 2 guess ahead?
The configuration used in testing set the draft model to guess eight tokens ahead before the full Qwen3.8-27B model verifies them in a single pass.
Where can I find the DFlash 2 model weights?
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
The checkpoint is hosted on Hugging Face as z-lab/Qwen3.8-27B-DFlash2, released under an Apache 2.0 license, built on top of the base Qwen3.8-27B model.
