How to Run Qwen3.8-27B-OBLITERATED Locally with GGUF Quants
A practical guide to running the uncensored Qwen3.8-27B-OBLITERATED model locally: GGUF quant sizes, VRAM needs, and the exact settings it requires.
What is Qwen3.8-27B-OBLITERATED?
Qwen3.8-27B-OBLITERATED is an abliterated (refusal-removed) version of Alibaba’s Qwen3.8-27B model, built by a developer going by Pliny the Prompter using a toolchain called OBLITERATUS. Abliteration is a weight-surgery technique that identifies the internal “refusal directions” a model uses to trigger safety declines, then projects them out of the weight space. The result is a model that answers questions the stock version would refuse, while trying to keep everything else, reasoning, coding, general knowledge, close to unchanged. It’s distributed under Apache 2.0, the same license as the base model, and is available as full safetensors weights or as GGUF quantized files for local inference through llama.cpp, Ollama, or LM Studio.
TL;DR
- Three surgery iterations (V1, V2, V3) trade off capability loss against how thoroughly refusals are removed, with V3 the current recommended release.
- V3 scores 82.33% on MMLU versus 84.46% for stock Qwen3.8-27B, a 2.12 percentage point cost for what the model card calls “genuine liberation,” meaning both hard refusals and softer safety-lecture deflections are gone.
- GGUF quants range from about 11 GB (Q2_K) to 27 GB (Q8_0), so the model can run on anything from a single high-VRAM consumer GPU down to more constrained setups depending on quality tolerance.
- Correct decoding settings are not optional: temperature 0, repetition_penalty 1.15, and no system prompt are all required to avoid degraded or looping output.
- The chat template matters as much as the weights: V3’s template prefills an empty thinking block, so llama.cpp needs
--jinjaand Ollama or LM Studio need to use the model’s built-in template rather than a generic one. - STEM tasks take the biggest capability hit (-3.3pp) while humanities are barely touched (-1.0pp), and a few subjects like philosophy actually score higher than stock.
- Agentic use requires different settings than single-turn chat, including slight temperature and periodic context summarization to avoid tool-call loops.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
How does the abliteration process work?
Abliteration doesn’t fine-tune a model in the traditional sense. It finds directions in the weight space statistically associated with refusal behavior and removes or dampens them. The OBLITERATUS project went through three rounds of refinement to get this right.
V1 used a single aggressive SVD (singular value decomposition) pass with five directions. It killed hard refusals completely but cost 6 percentage points of MMLU, a noticeable capability drop.
V2 introduced what the model card calls “complementary abliteration blending.” Two different surgery methods, aggressive SVD and LEACE (a technique that minimizes mutual information rather than greedily targeting refusal signal), were applied separately and then blended at roughly 60/40 in weight space. Because the two methods fail in different ways, one damaging capability and the other leaving weaker refusal removal, blending canceled out some of each one’s weaknesses. V2 landed at just -0.3pp MMLU versus stock, but it still deflected some queries with safety lectures rather than direct refusals.
V3 built on V2 rather than restarting from the stock model, an approach the developer calls iterative stacking. It applied a gentle refinement pass, then a second surgery using a narrower, targeted corpus aimed specifically at the categories where soft deflections remained. The two results were blended 50/50, and multimodal and vision-related tensors were restored from the stock model. The outcome: zero hard refusals, zero soft deflections in testing, and a smaller MMLU cost of 2.12 points.
What are the GGUF quant options and VRAM requirements?
GGUF is the file format used by llama.cpp and compatible tools like Ollama and LM Studio to run quantized models efficiently on consumer hardware. Quantization reduces the precision of the model’s weights to shrink file size and memory use, at some cost to output quality. The available quants for Qwen3.8-27B-OBLITERATED, per the official and community (mradermacher) GGUF repositories, span:
- Q8_0 (~27 GB): maximum quality, closest to the full-precision model.
- Q6_K (~21 GB): strong balance of quality and size.
- Q5_K_M (~18 GB): solid general-purpose option.
- Q4_K_M (~16 GB): commonly considered the sweet spot between size and fidelity.
- Q3_K_M (~13 GB): for lower-VRAM setups, with a visible quality tradeoff.
- Q2_K (~11 GB): minimum viable quality, mainly for hardware-constrained runs.
- IQ4_XS (~14 GB): an experimental compact quant.
The community GGUF repo also includes additional intermediate quants (Q3_K_L, Q3_K_S, Q4_K_S, Q5_K_S) along with mmproj files for the vision components. As a rule of thumb, pick a quant whose file size fits comfortably in your GPU’s VRAM with some headroom left for context; a 24 GB card handles Q4_K_M to Q5_K_M well, while 16 GB cards are better matched to Q3_K_M or Q4_K_M. Full bfloat16 safetensors weights are also published (about 54 GB across 29 shards) for anyone running through Transformers on multi-GPU or high-memory setups instead of a quantized local build. MLX support for Apple Silicon is listed as pending, waiting on upstream mlx_lm to add support for the Qwen3.5 architecture family this model derives from.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
What settings are actually required for correct output?
This is the part most people get wrong, and the model card is explicit that these settings “matter.” Running the model with defaults from a generic chat UI will produce degraded or broken output.
- Temperature: 0. Greedy decoding gives the most complete and accurate outputs, especially for code. Anything above 0.5 measurably degrades quality.
- Repetition penalty: 1.15. This is described as essential, not optional. Without it, greedy decoding gets stuck looping on import statements and boilerplate. Lower it to 1.10-1.12 if you want tighter, shorter responses.
- Max new tokens: 2048 or more. Complex code generation or multi-step reasoning needs room to finish.
- System prompt: none. The developer notes this was A/B tested; adding a system prompt can reintroduce refusal-like behavior, so an empty system prompt performs better.
- Thinking mode: off (recommended). V3 ships with a chat template that prefills an empty thinking block, skipping the reasoning chain by default. Thinking mode can be turned on and works without refusals, but off gives more direct answers.
- Sampling parameters (top_p, top_k, min_p): not needed. Greedy decoding plus the repetition penalty is reported to outperform sampling-based approaches for this model.
For GGUF users specifically, the chat template needs to travel with the weights. In llama.cpp that means launching with the --jinja flag so the bundled template is honored. In Ollama or LM Studio, make sure the tool is configured to use the model’s built-in template rather than a generic Qwen or ChatML template, otherwise the thinking-block prefill won’t apply and output quality suffers.
Is it worth running for agentic or coding workflows?
For single-turn use, the model card reports 20 out of 20 correct, functional implementations on a set of code generation tasks, and 7 out of 8 on a broader set of “advanced real-world” tasks including async code refactoring, JSON schema extraction, Kubernetes debugging scenarios, and security code review, matching stock Qwen3.8-27B’s own 7/8 score on the same tasks. Both stock and abliterated versions failed the same multi-tool chaining task, suggesting that particular limitation is inherited from the base model rather than introduced by the surgery.
For agent harnesses (coding agents, automated pentest frameworks, anything with multi-turn tool calls), the recommended settings shift. Repetition penalty stays critical at 1.15 to stop the model looping on repeated tool calls. Temperature moves up slightly to 0.1-0.3, since pure greedy decoding can get stuck in deterministic loops in agentic contexts where a single point of randomness helps break out. Per-turn token limits should be trimmed to 1024-2048 to keep the agent focused rather than over-generating, and context should be summarized after roughly every 10 turns since repeated tool-call history fills the context window fast and can degrade coherence.
Frequently Asked Questions
What is “abliteration” and how is it different from fine-tuning?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
Abliteration edits a model’s existing weights to remove directions associated with refusal behavior, rather than training on new examples. It’s a surgical, mathematically targeted intervention (using SVD or LEACE-style projections) rather than a gradient-based retraining process, which is why it can be done with relatively small capability cost.
How much VRAM do I need to run this model?
It depends on the quant. The smallest usable GGUF quant (Q2_K) is around 11 GB, while the highest quality quant (Q8_0) is about 27 GB. A 16 GB GPU comfortably handles Q4_K_M, and a 24 GB card can run Q5_K_M or Q6_K with room for context.
Why does the model loop or repeat text without the right settings?
Greedy decoding (temperature 0) alone tends to fall into repetitive loops on boilerplate like import statements. The repetition_penalty of 1.15 specifically counteracts this and is described in the model documentation as essential, not a minor tuning knob.
Does removing refusals hurt the model’s general capability?
Yes, but the cost has shrunk across iterations. V1 lost 6 percentage points of MMLU, V2 lost about 0.3, and the current V3 release costs about 2.1 points versus stock Qwen3.8-27B, with the loss concentrated in STEM subjects like abstract algebra and formal logic rather than spread evenly.
Can I run this on Apple Silicon with MLX?
Not yet, according to the model card. MLX support is pending upstream changes in the mlx_lm library to support the Qwen3.5 architecture family that this model is based on.