JetSpec Local Install: How Much Faster Is Tree-Based Speculative Decoding?
JetSpec's tree-based speculative decoding speeds up LLM inference without quality loss. Here's a local H100 benchmark with Qwen 3.8B and setup notes.
What is JetSpec and why does it matter for LLM inference?
JetSpec is a speculative decoding technique built to make large language model text generation faster without changing what the model actually outputs. It comes from UCST and uses a tree-structured approach to guessing future tokens, which distinguishes it from older speculative decoding methods that guess in flat, independent sequences. The pitch is simple: same output, less waiting.
Standard autoregressive decoding generates one token at a time, and each token has to wait for the previous one to finish before the model can compute the next. That serial dependency is the main bottleneck in LLM inference, especially for long responses. Speculative decoding attacks this by having a small, fast draft model guess several tokens ahead, then letting the large model verify all those guesses in a single forward pass instead of one token at a time. If the guesses check out, generation jumps ahead multiple tokens at once. If they don’t, the large model falls back to normal decoding for that step. Either way, the final text is identical to what the large model would have produced alone.
TL;DR
- JetSpec uses tree-based speculative decoding, where each branch of guessed tokens only depends on its own earlier guesses, so inconsistent branches don’t get wasted the way they do in flat speculative decoding schemes.
- A local test on an H100 GPU with Qwen 3.8B showed baseline autoregressive generation running at roughly 28 tokens per second.
- With JetSpec’s draft head enabled, the same prompt on the same hardware hit about 2.82x speedup, close to 3x, without any change in output quality.
- Increasing the speculation budget (the number of nodes allowed in the guess tree) from 32 to 128 kept pushing speed higher, peaking near 85 tokens per second.
- Pushing the budget to 256 caused speed to drop, likely because the H100 test setup lacked flash attention, which the JetSpec team used to reach their published benchmarks on B200 GPUs.
- The project’s own benchmarks claim up to 9x speedup, but that number appears tied to newer hardware (B200) and flash attention, not a stock H100 setup.
- Installation is straightforward: clone the repo, run the base install, download a supported model plus its matching draft head from Hugging Face, and run the included benchmark scripts.
How does JetSpec’s tree-based speculative decoding actually work?
Classic speculative decoding pairs a small draft model with a large target model. The draft model proposes a short run of tokens, and the target model checks them all in one pass instead of token by token. When the guesses are right, you get a burst of accepted tokens for the cost of one verification step. When they’re wrong, you fall back to normal generation from the point of divergence.
The limitation of older speculative decoding is that when it tries multiple guesses at once, those guesses run in parallel without referencing each other. A later guess in one branch might not connect logically to what happened in a different branch, so a lot of the guessed tokens get thrown away during verification.
JetSpec changes the structure. Instead of independent parallel guesses, it builds a tree where each branch only looks back at its own prior guesses within that branch, not at sibling branches. That keeps each individual path internally consistent, which means more of the tree survives the verification step. More surviving tokens per verification pass translates directly into higher throughput. This is the core architectural difference the JetSpec team highlights over prior speculative decoding approaches, and it’s the reason the method can extract more speed out of the same verification budget.
What does a real local benchmark on an H100 look like?
Testing on an Ubuntu machine with an H100 GPU and Qwen 3.8B, a baseline (no speculative decoding) run produced around 28 tokens per second on a single prompt. That’s the reference point for autoregressive generation on this hardware with no tricks applied.
Enabling JetSpec’s speculative decoding with the draft head pulled from Hugging Face, the same prompt ran at 2.82x the baseline speed, close to 3x faster, with identical output text. That’s a meaningful jump for something that doesn’t touch model weights, quality, or determinism, it only changes how tokens get generated and verified.
The test then varied the “budget,” which is the maximum number of nodes allowed in JetSpec’s guess tree. A bigger budget means more candidate tokens get proposed per round. Running the same prompt at budgets of 32, 64, 128, and 256 showed speed climbing steadily up through 128, topping out around 85 tokens per second, nearly 3x the baseline. At 256, though, speed actually dropped back down instead of continuing to climb.
The likely explanation is flash attention. JetSpec’s published benchmarks reportedly run on B200 GPUs with flash attention compiled in, which is a more efficient attention kernel that scales better with larger speculation trees. Without flash attention, which takes noticeable time to compile and wasn’t used in this H100 test, the bigger tree budgets stop paying off past a certain point and efficiency tapers off.
Is JetSpec worth running on your own hardware?
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
For anyone doing local inference or self-hosting an LLM, a consistent 3x speedup on stock hardware, without flash attention and without a top-tier GPU, is a solid outcome. It suggests the technique is not narrowly tuned to one specific card or software stack. If you do have access to a B200 or you’re willing to compile flash attention into your setup, the ceiling appears meaningfully higher, based on the project’s own claims of up to 9x.
The tradeoff to know about going in: JetSpec needs a matching draft head model for whatever base model you’re running, and those are distributed separately on Hugging Face. You’re not just pip installing a wrapper around your existing checkpoint, you’re pairing a compatible draft model with your target model. That means coverage depends on which base models the project has published draft heads for. As of testing, the JetSpec team had released a set of models but their most recent updates were from mid-July, which raises a fair question about how actively the project is maintained and how quickly it will expand to cover more base models.
The bigger caveat is generality: this speedup was verified for one prompt, one model size (Qwen 3.8B), and one GPU (H100). Real-world gains will vary by prompt type, sequence length, and how well the draft model’s guesses match the target model’s actual behavior for that content.
How do you install and benchmark JetSpec locally?
The basic setup involves cloning the JetSpec repository and running its base installation, which handles the necessary dependencies. From there you download both a base model (in this case Qwen 3.8B) and its corresponding JetSpec draft head from Hugging Face.
A practical benchmarking approach: run the same prompt twice on the same GPU. First run it through plain autoregressive generation to get a baseline tokens-per-second number. Then run the identical prompt through JetSpec’s tree-based speculative decoding and time it the same way. Comparing the two gives a direct, apples-to-apples speedup figure rather than relying on the project’s own published charts.
To go further, you can sweep the speculation budget parameter (the max tree size) across a few values, such as 32, 64, 128, and 256, to see where your hardware’s sweet spot sits. As shown above, more budget isn’t always better without the right attention kernel backing it up.
Frequently Asked Questions
Does speculative decoding change the model’s output quality?
No. Speculative decoding, including JetSpec’s tree-based version, is designed to be lossless. The large model still verifies every proposed token, so the final generated text is identical to what plain autoregressive decoding would have produced. The only thing that changes is how fast you get there.
What speedup can I expect from JetSpec?
On an H100 GPU with Qwen 3.8B and no flash attention, testing showed close to 3x faster generation compared to baseline autoregressive decoding. The project’s own benchmarks claim up to 9x, but that figure appears to depend on more advanced hardware (B200) and flash attention, which weren’t part of the stock local setup tested here.
Do I need a specific GPU to run JetSpec?
JetSpec ran successfully on a consumer-accessible H100, so it’s not restricted to the newest hardware. However, the largest published speedups seem tied to B200 GPUs paired with flash attention, so results on older or smaller cards will likely land closer to the 3x range than the 9x ceiling.
Why did increasing the speculation budget past a point reduce speed?
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
In testing without flash attention, speed peaked at a budget of 128 nodes and then dropped at 256. This suggests that larger guess trees need an efficient attention kernel to process the extra candidate tokens without adding overhead that outweighs the benefit. Flash attention appears necessary to keep scaling gains at higher budgets.
Is JetSpec actively maintained?
At the time of testing, the project’s models had last been updated in mid-July, which is a relatively long gap in a fast-moving AI tooling space. The underlying tree-based speculative decoding concept is considered promising, but ongoing model coverage and updates are worth checking before committing to it for production use.