How to Run Qwen3.8-9B Distill Locally on a Single GPU
Guide to running Empero's Qwen3.8-9B distilled model locally: required kernels, sampling settings, and 262K context setup on one GPU.
What is Qwen3.8-9B and why does it matter?
Qwen3.8-9B is a full-parameter distillation released by Empero that compresses the reasoning behavior of a much larger teacher model, Qwen3.8 2.4T A95B, into a dense 9B model built on the Qwen3.5-9B architecture. The point of the release is straightforward: get frontier-style chain-of-thought reasoning onto hardware that a single GPU can handle, without the mixture-of-experts routing or multi-GPU serving overhead that the teacher model requires.
The model was trained on roughly 70,000 curated teacher traces covering math, code, general reasoning, instruction following, and tool use. Every trace was quality-filtered before training, and the fine-tune touched all parameters rather than adding an adapter layer on top of the base.
TL;DR
- Full distillation, not LoRA: Empero updated every parameter of Qwen3.5-9B using around 70,000 filtered chain-of-thought traces from the Qwen3.8 2.4T A95B teacher.
- Big MMLU jump, small GSM8K dip: the distilled model gains roughly 20 points on flexible-extract MMLU (0.751 vs 0.546) but scores slightly below the base on gsm8k_cot (0.870 vs 0.885).
- Native 262,144-token context carries over unchanged from the Qwen3.5-9B base, so long-document and long-conversation workloads don’t need extra configuration tricks.
- Gated DeltaNet kernels are mandatory for usable speed: without flash-linear-attention and a matching causal_conv1d build, the linear-attention layers fall back to slow PyTorch ops that eat memory and time.
- Recommended sampling is temperature 0.6, top_p 0.95, top_k 20; greedy decoding is a known trigger for repetition loops on long reasoning generations in this model class.
- Every response starts with a
<think>block learned from the teacher’s own reasoning traces, so production code needs to parse and strip that span before showing output to end users. - Function calling works natively per the Qwen3.5 spec, meaning tool use doesn’t require a wrapper or a separate fine-tune.
One coffee. One working app.
You bring the idea. Remy manages the project.
How do you set up Qwen3.8-9B locally?
The quickest path is the Hugging Face Transformers quickstart. You need a recent version of transformers with Qwen3.5 architecture support, since this model reuses that base architecture rather than something older. A minimal load-and-generate script looks like this:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "empero-ai/Qwen3.8-9B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "A snail is at the bottom of a 10-meter well. Each day it climbs 3 meters, each night it slips back 2. How many days until it escapes?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=16384,
temperature=0.6, top_p=0.95, top_k=20, do_sample=True)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))device_map="auto" handles GPU placement, and bfloat16 keeps memory reasonable on a single card. The model is also compatible with vLLM and SGLang for anyone who wants an inference server instead of a raw script, since both runtimes support the Qwen3.5 architecture this model inherits.
Why do you need Gated DeltaNet kernels?
Qwen3.5-9B’s architecture uses linear-attention layers based on Gated DeltaNet, and those layers depend on custom CUDA kernels to run efficiently. Specifically, the model card calls out two dependencies: flash-linear-attention and a CUDA-matched build of causal_conv1d.
If those aren’t installed, or if the causal_conv1d build doesn’t match your CUDA version, the linear-attention layers silently fall back to plain PyTorch operations. That fallback isn’t just slower, it’s also much more memory-hungry, which can turn a comfortable single-GPU deployment into one that runs out of VRAM or crawls at unusable speed. This is the single most common failure point for people trying to run this model class locally: everything loads fine, inference technically works, but it’s an order of magnitude slower than expected because the fast kernel path never activated.
Before troubleshooting anything else, confirm both packages are installed and that causal_conv1d was built against the same CUDA toolkit version as your PyTorch install. Mismatched builds are the usual culprit.
What sampling settings should you use?
Empero’s documentation is specific here: temperature=0.6, top_p=0.95, top_k=20. These are the same settings recommended for the Qwen3.5 base model, and they matter more than usual for this particular fine-tune.
The reason is that Qwen3.8-9B was trained on distilled reasoning traces, which means it tends to produce long chain-of-thought sequences before landing on an answer. Greedy decoding (picking the single highest-probability token every step) is a known failure mode for reasoning models in this class: it tends to push the model into repetition loops during long generations, where it gets stuck restating the same phrase or reasoning step. Sampling with the recommended temperature and top-p/top-k values avoids that trap.
Generous output length matters too. The model card recommends allowing up to 16,384 new tokens, since every response opens with a <think> block before the final answer. If you cap max_new_tokens too aggressively, you risk truncating the reasoning before the model ever reaches its conclusion.
How good are the benchmark results?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
Empero evaluated both the Qwen3.5-9B base and the Qwen3.8-9B distill using EleutherAI’s lm-evaluation-harness, with identical settings and chain-of-thought protocols for both.
On mmlu_flan_cot_zeroshot, which spans all 57 MMLU subjects (about 1,700 questions), the distilled model shows a substantial jump: 0.751 versus 0.546 on flexible-extract accuracy, and 0.511 versus 0.251 on strict-match. That’s a large gain, and it lines up with the training emphasis on dense reasoning traces across varied domains.
On gsm8k_cot, the picture flips slightly. The distilled model scores 0.870 (flexible) and 0.850 (strict), compared to 0.885 and 0.875 for the base. It’s a small regression, not a collapse, but worth noting: distillation isn’t a universal upgrade across every benchmark, and grade-school math is one place where the base model already performed strongly.
Read together, the results suggest the distillation traded a small amount of GSM8K performance for a large gain in broad knowledge and reasoning coverage as measured by MMLU. Whether that trade is worth it depends on what you’re building.
Is Qwen3.8-9B worth running over the base model?
If your workload leans on broad knowledge, general reasoning, or instruction following across varied subjects, the MMLU gains make a strong case for the distilled version. If your use case is narrowly focused on grade-school-style arithmetic word problems, the base model has a slight edge on that specific benchmark.
Beyond raw scores, the practical draw of Qwen3.8-9B is that it packages frontier-style reasoning behavior, learned from a 2.4T-parameter teacher, into a dense 9B model that fits on one GPU. You don’t need the infrastructure to serve a mixture-of-experts model at scale to get some of that reasoning quality. The tradeoffs are the ones inherent to distillation: the model inherits the teacher’s reasoning style, including a tendency to over-deliberate on questions that don’t need it, and the fine-tune is text-only, so any vision capability from the underlying Qwen3.5 base wasn’t touched or evaluated.
Weights are released under Apache-2.0, inherited from the Qwen3.5-9B base, so licensing isn’t a blocker for commercial or research use.
Frequently Asked Questions
What hardware do I need to run Qwen3.8-9B?
The model is designed to deploy on a single GPU, consistent with its 9B parameter count and bfloat16 loading in the quickstart example. Exact VRAM requirements depend on context length used and batch size, but a single modern GPU with enough memory for a 9B bfloat16 model plus KV cache is the baseline expectation.
Do I need special software beyond Transformers?
Yes. Beyond a recent transformers release with Qwen3.5 support, you need flash-linear-attention and a CUDA-matched causal_conv1d build for the Gated DeltaNet layers to run at full speed. Without them, inference falls back to slow PyTorch operations.
Why does every response start with a <think> block?
The model was distilled from chain-of-thought traces produced by the Qwen3.8 2.4T A95B teacher, and it learned to reproduce that same reasoning-then-answer structure. Applications should parse and strip the <think>...</think> span before displaying output to end users.
Can Qwen3.8-9B handle long documents?
It inherits a native 262,144-token context window from the Qwen3.5-9B base, so long documents and extended conversations are supported without extra context-extension configuration.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
Does this model support tool calling?
Yes, native function calling per the Qwen3.5 specification, with no wrapper or separate tool-use fine-tune required.