Qwen3.8-4B Distilled: How Emprius Shrank a 2.4T Model to 4B
How Emprius distilled Qwen3.8's 2.4 trillion parameter model into a 4B student using 45,000 reasoning traces, and what survived the compression.
What is the Qwen3.8-4B distilled model?
It’s a 4 billion parameter model trained to mimic the reasoning behavior of Qwen3.8’s much larger 2.4 trillion parameter model, built by a team called Emprius. Instead of training a small model from scratch and hoping it learns to reason well, Emprius ran the giant teacher model on roughly 45,000 curated prompts, captured its full reasoning traces, and trained the 4B student to reproduce those same thought patterns. The result is a model small enough to run on a consumer GPU that inherits a compressed version of how a trillion-parameter model thinks.
TL;DR
- Emprius took Qwen3.8’s 2.4 trillion parameter teacher model and distilled its reasoning behavior into a 4 billion parameter student, following up on an earlier 9B distillation from the same team.
- The student model started from a Qwen3.5-4B hybrid architecture using gated Delta net layers, not a Qwen3.8 architecture trained from zero.
- Training relied on 45,000 curated reasoning traces: prompts run through the teacher, with the full chain of reasoning captured and used as the training target for the small model.
- The distilled model reportedly gains close to 20% on MMLU over the base 4B model, while GSM8K math scores dip slightly, a tradeoff typical of aggressive distillation at small scale.
- In quantized testing (Q4, Q6, Q8 GGUF formats via llama.cpp), all three versions preserved the underlying reasoning behavior on a hard ethical dilemma prompt, showing the distilled reasoning survives compression down to 4-bit.
- The Q4 quant runs under 3GB and fits on any GPU with more than 4GB of VRAM, making it viable as a lightweight daily driver despite being a fraction of the teacher’s size.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How does knowledge distillation actually work here?
Traditional distillation often just matches output logits or final answers between a teacher and student model. This approach goes a step further by targeting the reasoning process itself. Emprius ran the 2.4 trillion parameter Qwen3.8 model on a set of about 45,000 carefully selected prompts and recorded the full reasoning traces (the intermediate chain of thought the teacher produces before landing on an answer), not just the final output.
The 4B student model was then trained to reproduce those traces directly. That’s a meaningful distinction: the small model isn’t learning to reason independently through reinforcement or from raw pretraining data. It’s learning to imitate a specific reasoning style that a much larger model already discovered. Think of it less as teaching a student to solve problems and more as having them memorize a master’s worked solutions closely enough that they can reproduce similar reasoning patterns on new problems.
This matters because reasoning ability tends to be one of the hardest capabilities to compress. Small models can often match large ones on factual recall or simple pattern completion, but multi-step reasoning, especially on ambiguous or value-laden problems, usually degrades sharply as parameter count drops. Distilling directly from reasoning traces, rather than just final answers, appears to be the mechanism that lets the 4B model retain more of that behavior than it otherwise would.
Why start from a Qwen3.5-4B architecture instead of Qwen3.8?
Rather than building a scaled-down version of the Qwen3.8 architecture from scratch, Emprius used an existing Qwen3.5-4B hybrid model as the student’s starting point. This is a 4 billion parameter architecture that incorporates gated Delta net layers, a design choice associated with more efficient sequence modeling compared to standard attention-only transformer blocks.
Using an already-trained, architecturally distinct base model as the student, rather than training a fresh 4B model from zero, is a common efficiency move in distillation projects. The base model already has functioning language capabilities and general knowledge. Distillation training then focuses specifically on aligning its reasoning behavior with the teacher’s, rather than teaching it language and reasoning simultaneously. This likely explains part of why the MMLU gains are so large relative to the base model: the architecture already had decent general knowledge, and the distillation process sharpened how it applies that knowledge to reasoning-heavy tasks.
What actually improved, and what got worse?
According to the model’s reported benchmark changes, MMLU scores jumped by close to 20% over the base 4B model. That’s a substantial gain for a benchmark that measures broad multitask knowledge and reasoning across subjects like law, science, and history. It suggests the reasoning-trace distillation transferred real generalizable reasoning patterns, not just memorized answers to the specific 45,000 training prompts.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
GSM8K, a benchmark of grade-school math word problems, reportedly dropped slightly. This is a known tradeoff in distillation work: math reasoning tends to require precise, multi-step symbolic manipulation that’s sensitive to exact training data composition. If the 45,000 curated traces skewed toward general reasoning and knowledge-style prompts rather than heavy quantitative problem sets, a small dip in math performance is expected. It doesn’t necessarily mean the model reasons worse overall, just that math-specific capability wasn’t the primary target of the distillation set.
Does the distilled reasoning survive quantization?
Testing across Q4, Q6, and Q8 GGUF quantizations (run locally through llama.cpp) suggests yes, at least on qualitative reasoning tasks. All three quantization levels, when given a deliberately difficult ethical dilemma prompt with no factual right answer, converged on the same underlying decision and used comparable reasoning structures to justify it. That consistency across quant levels is a meaningful signal: it means the reasoning behavior distilled from the teacher model isn’t fragile enough to break down under aggressive weight compression.
That said, quality wasn’t identical. The Q8 quant produced noticeably deeper reasoning, taking longer to produce its first token and generating a longer, more original response that reframed the problem before answering. Q4 and Q6 defaulted to more straightforward utilitarian reasoning, arriving at similar conclusions through more predictable logic. Q4 was fastest and most verbose, Q6 was the most concise (and finished quickest overall despite not being the fastest per-token), and Q8 was the slowest but arguably the most thoughtful. None of the quants broke or produced incoherent reasoning, which is the more important finding for anyone deciding whether quantizing this model for local deployment is safe.
Is a 4B distilled model actually useful compared to the full-size teacher?
For most practical local deployment scenarios, yes, with clear tradeoffs. A 2.4 trillion parameter model is not something most people can run outside of a cloud API or a serious multi-GPU cluster. A 4 billion parameter model, especially at Q4 quantization under 3GB, runs comfortably on a single consumer GPU with modest VRAM. That accessibility is the entire point of distillation work like this.
The tradeoff is real but narrower than raw parameter count would suggest. The distilled model won’t match the teacher’s depth on genuinely novel or highly technical problems, and math performance in particular seems to have taken a small hit. But for general reasoning, broad knowledge tasks, and everyday use cases, the gap is smaller than the 600,000x difference in parameter count implies. That’s the actual value proposition of reasoning-trace distillation: it doesn’t recreate the teacher, but it captures enough of its reasoning style to be useful in a package that fits on hardware most developers already own.
Frequently Asked Questions
What is Emprius’s role in this model?
Emprius is the team that performed the distillation, taking Qwen3.8’s large teacher model and training a separate, much smaller 4B student model to replicate its reasoning behavior. They previously released a 9B distillation using a similar approach.
How many reasoning traces were used in training?
Approximately 45,000 curated prompts were run through the 2.4 trillion parameter teacher model, and the full reasoning traces produced were used to train the 4B student model.
What architecture is the 4B student model based on?
It’s based on a Qwen3.5-4B hybrid architecture that uses gated Delta net layers, rather than a scaled-down version of the Qwen3.8 architecture.
Does quantizing the model hurt its reasoning quality?
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
Testing across Q4, Q6, and Q8 quantization levels showed all three preserved the core reasoning and reached similar conclusions on a difficult test prompt, though Q8 produced deeper, more original reasoning while Q4 and Q6 were more direct and predictable.
What size GPU do you need to run it?
The Q4 quantized version comes in under 3GB, making it runnable on consumer GPUs with as little as 4GB of VRAM.
