Qwen3-8-27B at 11.8GB: Do GSQ and RCO Quantization Actually Hold Up?
ISTA's Das Lab shrank Qwen3.8-27B to 11.8GB with new GSQ and RCO quantization. Here's what that means and how it performs locally via llama.cpp.
What are GSQ and RCO quantization?
GSQ and RCO are two new weight compression methods from Das Lab at the Institute of Science and Technology Austria (ISTA), the same research group behind GPTQ, one of the most widely used quantization techniques in the industry today. The lab applied both methods to Qwen3.8-27B, a 27 billion parameter model, and released versions as small as 8 to 12 GB. The headline claim is that the compressed model is “task lossless,” meaning it performs close to the full precision original on real benchmarks despite the drastic size reduction.
TL;DR
- GSQ (Gumbel Softmax Quantization) examines each weight individually and picks the bit depth, two, three, or four bit, that keeps that specific weight closest to its original value, since some weights tolerate heavy compression while others don’t.
- RCO (Riemannian Constrained Optimization) takes the sensitivity map GSQ produces and allocates a fixed total memory budget across the whole model, giving more bits to sensitive tensors and fewer to tolerant ones, landing on an exact target size like 11.8GB.
- Das Lab released Qwen3.8-27B at multiple compressed sizes, and the 11.8GB IQ3S build is the one the lab recommends and labels task lossless.
- In hands-on testing with llama.cpp, the compressed model consumed just over 28GB of VRAM on a single 48GB RTX card and generated a Canvas based HTML animation (a “doner kebab” render) with correct flame gradients, shading, and drip physics on the first attempt.
- The model correctly caught a deliberately hidden logic bug in a long, nested SQL query (an ascending rank sort where descending was needed), matching the behavior of the original unquantized model.
- On a stress test spanning 75+ languages including low-resource ones and constructed languages, the compressed model showed the same multilingual weaknesses as the base Qwen3.8 model, solid on major languages, shaky on languages like Twi, Burmese, and Balochi, suggesting the quantization itself didn’t introduce the gap.
- The model worked through a multi-step chemistry equilibrium problem (Ksp, common ion effect, mass balance) with correct setup and a numerically accurate final answer, showing no hallucination or reasoning breakdown despite the heavy compression.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Why does quantization matter for running large models?
Every neural network is built from millions or billions of numeric weights, organized into tensors, learned during training. Normally these numbers are stored at high precision, which is why a 27 billion parameter model can require well over 50GB of memory in its original form. Quantization stores those same weights with fewer bits per number, shrinking file size and VRAM needs dramatically. The hard part has always been doing this without wrecking accuracy. Push compression too far uniformly across the model and you start losing reasoning ability, factual recall, or coherence. This is the problem GSQ and RCO are aimed at solving with more precision than older blanket quantization approaches.
How do GSQ and RCO actually work together?
GSQ operates at the level of individual weights. For every weight in the model, it tests different possible bit depths and learns which one preserves that weight’s value most faithfully. Some weights change very little even under aggressive compression. Others are fragile and degrade fast if you don’t give them enough bits. GSQ effectively builds a sensitivity map of the entire model, weight by weight.
RCO then takes that map and does the budgeting. Given a fixed total size target, in this case 11.8GB for the recommended build, RCO decides how to distribute available bits across all the tensors. Sensitive tensors get more bits. Tolerant ones get compressed harder. The result is a model that hits an exact size target while trying to waste as little accuracy as possible in the process. GSQ figures out what each weight needs. RCO figures out how to spend the overall budget given those needs.
Does the compressed model actually perform well locally?
In testing run through llama.cpp on a single Nvidia RTX card with 48GB of VRAM, the 11.8GB quant of Qwen3.8-27B used just over 28GB of VRAM during inference (KV cache included), meaning it’s runnable on hardware well short of what the full precision model would demand. Users with less VRAM can drop to a smaller quant size or reduce the KV cache further.
Across several tests, the model held up in ways that matter for real use:
It generated a self-contained HTML/JavaScript animation of a döner kebab rotating on a spit over a flame, using pure Canvas with no external libraries, on the first attempt. The render included flame gradients, shaded meat with sear marks, drip physics, and steam, details that suggest the model retained fine-grained coding and visual reasoning ability rather than just pattern-matching a simpler output.
It found a deliberately hidden logic bug in a long, deeply nested SQL query where a ranking clause was sorted ascending instead of descending, silently returning the worst customers instead of the best. This is a bug designed to be hard to catch: the query used nested CTEs and buried the ordering mistake next to correct code in a sibling clause. The compressed model caught it and proposed fixes, matching what the original uncompressed model was able to do.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
It also worked through a multi-step chemistry reasoning problem involving solubility equilibrium (Ksp), the common ion effect, and mass balance, a type of problem where an early arithmetic or conceptual slip cascades into a wrong final answer. The model’s reasoning chain tracked units, set up equilibrium expressions correctly, and arrived at a numerically correct result without looping or hallucinating.
Where does the compression fall short?
Multilingual coverage is the weak spot, but it appears to be inherited from the base Qwen3.8 model rather than introduced by quantization. In a stress test covering roughly 75 languages, including major world languages, low-resource languages like Maithili and Balochi, a historical script (Elder Futhark runes), and constructed languages (Klingon and Quenya), results were mixed. Major languages like Mandarin, Spanish, and Arabic came through solid. Several low-resource languages were only barely coherent. This mirrors known behavior in the original, unquantized Qwen3.8 model, and testers have noted that multilingual ability actually regressed compared to the earlier Qwen3.6 generation. The compression didn’t make this worse. It just didn’t fix an existing gap.
On the positive side, the model handled uncertainty honestly in the constructed language tests. Rather than confidently hallucinating fluent Klingon or Quenya, it flagged missing vocabulary and improvised with glosses, the kind of calibrated behavior you want from a model that doesn’t actually know an answer.
Is running the 11.8GB quantized Qwen3.8-27B worth it?
For anyone trying to run a 27 billion parameter model on a single consumer or prosumer GPU, this quant is a meaningful option. Getting VRAM use down to around 28GB while retaining coding accuracy, SQL debugging precision, and multi-step scientific reasoning is a real result, not a marketing claim. The “task lossless” label seems to hold up on the tasks tested: front-end code generation, bug detection in complex queries, and chained chemistry equilibrium problems all came back correct or functionally equivalent to what the full precision model would produce. The main caveat is multilingual performance, which was already a soft spot in Qwen3.8 before quantization and remains one after.
Frequently Asked Questions
What does “task lossless” mean in this context?
It means the compressed model performs essentially the same as the original, full precision model on real benchmark and task tests, despite using a fraction of the memory. It doesn’t mean literally zero difference in every weight, it means the practical output quality on real tasks doesn’t measurably drop.
How much VRAM do you need to run this model?
The 11.8GB quant of Qwen3.8-27B used just over 28GB of VRAM in testing on an RTX card with 48GB total, including KV cache. Users with less VRAM available can select a smaller quant size released by Das Lab or reduce the KV cache allocation.
What’s the difference between GSQ and older quantization methods like GPTQ?
GPTQ, also from Das Lab, was an earlier widely adopted quantization approach. GSQ adds a per-weight sensitivity analysis (testing multiple candidate bit depths per weight) paired with RCO’s model-wide budget allocation, aiming for a more precise, non-uniform distribution of bits than blanket quantization schemes.
Does quantization hurt multilingual performance?
In this case, the compressed model showed the same multilingual weaknesses (strong on major languages, weak on low-resource ones) as the original uncompressed Qwen3.8 model. The gap appears to be a pre-existing training characteristic rather than something introduced by GSQ or RCO.
Can I run this with tools other than llama.cpp?
The testing here used llama.cpp specifically, serving the model locally and interacting through its browser interface and through an agent pointed at local files. Since the released files are standard quantized model formats, other inference engines that support similar quant formats should be able to load them, though compatibility depends on the specific tool.
