OrcaSAQ2 27B: 3-Bit Quantization That Barely Loses Fidelity
OrcaSAQ2 27B shrinks Qwen3.8-27B from 54GB to 12.3GB at 3-bit precision, holding perplexity loss to +0.02% for agent tasks.
What is OrcaSAQ2 27B?
OrcaSAQ2 27B is a 3-bit quantized version of Qwen3.8-27B, built by OrcaRouter to run large-model reasoning on far less GPU memory. It shrinks the original 54GB BF16 checkpoint down to 12.3GB, a 77.2% reduction in storage, while reporting only a +0.02% increase in perplexity and 93.2% token-level agreement with the full-precision model. It’s designed specifically for long-horizon agent work: coding, tool use, and multi-step task execution where small errors compound over time.
TL;DR
- OrcaSAQ2 27B compresses Qwen3.8-27B from a 54GB BF16 checkpoint to 12.3GB using a proprietary sensitivity-aware mixed-precision quantization method that averages 3.21 bits per weight.
- The model reports near-identical fidelity to BF16: perplexity moves from 5.6468 to 5.6482 (+0.02%) on WikiText-2, with a mean KL divergence of 0.031 and 93.2% top-1 token agreement.
- On public benchmark tables, OrcaSAQ2 scores 70.0% on SWE-bench Verified and 58.4% on Terminal-Bench 2.1, landing ahead of Qwen3-Coder-480B-A35B on the former and close to several frontier agent stacks on the latter.
- The checkpoint fits inside a 16GB GPU with roughly 3.7GB of headroom left for KV cache, making it one of the few 27B-class models deployable on consumer or single-workstation hardware.
- MTP speculative decoding boosts single-stream throughput by 38% (65.3 to 90.1 tokens/sec) but trades away batched throughput and KV pool capacity, so the two serving modes suit different workloads.
- The model supports a 262K token context window, thinking mode, and structured tool calling, and ships for production serving through vLLM with OpenAI-compatible APIs.
- OrcaRouter frames the release around a long-horizon fidelity argument: perplexity measures next-token similarity, but agent tasks require the model to stay coherent across dozens of sequential decisions, which is a harder and different bar.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
How does OrcaSAQ2’s quantization work?
OrcaSAQ2 uses what OrcaRouter calls a “sensitivity-aware mixed-precision quantization system.” In plain terms, this means not every weight in the model is compressed to the same bit width. Parameters that matter more for output quality get allocated more bits, and less sensitive parameters get compressed harder, landing at an average of 3.21 bits per weight (bpw) across the decoder. This is different from a flat 3-bit or 4-bit quantization scheme applied uniformly across all layers.
The specific calibration strategy, precision allocation rules, and packing format are not publicly disclosed. OrcaRouter’s model card explicitly states that “detailed quantization methodology, calibration strategy, precision allocation and packing techniques are not currently disclosed,” so the technique itself is proprietary. What is disclosed is the outcome: a 12.3GB checkpoint (packaged across four safetensors shards) that behaves almost identically to the 54GB original on standard fidelity tests.
The base model, Qwen3.8-27B, uses a Qwen3_5ForCausalLM architecture with 64 layers, a hidden size of 5120, and a hybrid attention design: 48 Gated DeltaNet layers paired with 16 full-attention layers. That hybrid structure is part of why the model can support a 262,144-token context window without every layer paying full quadratic attention cost.
What do the fidelity numbers actually mean?
Three metrics anchor OrcaSAQ2’s fidelity claims, all measured against the BF16 reference on WikiText-2 (16,376 predicted tokens):
Perplexity moved from 5.6468 (BF16) to 5.6482 (OrcaSAQ2), a difference of +0.02%. Perplexity measures how well the model predicts held-out text; a near-zero delta means the quantized model’s overall language modeling quality is essentially unchanged.
Top-1 token agreement sits at 93.2%. This measures, token by token, how often OrcaSAQ2’s top prediction matches BF16’s top prediction. It’s a stricter test than perplexity because it looks at exact decisions rather than aggregate distribution similarity. A 93.2% agreement rate means roughly 1 in 14 tokens would come out differently than the full-precision model, even though the overall language quality (measured by perplexity) barely moves.
Mean KL divergence of 0.031 quantifies how different the full probability distributions are between the two models at each step. Low KLD alongside high top-1 agreement suggests that when OrcaSAQ2 disagrees with BF16, it’s usually disagreeing between similarly plausible tokens rather than producing wildly different outputs.
OrcaRouter is explicit that these numbers don’t guarantee identical downstream behavior. The model card notes that 93.2% agreement means “some token decisions differ from BF16” and that perplexity fidelity “does not guarantee identical downstream performance.” That’s why the benchmark section exists at all.
How does it perform on agent benchmarks?
OrcaRouter reports two long-horizon benchmark results, both framed as public reference points rather than strict head-to-head comparisons (since agent benchmarks depend heavily on the scaffold, tools, and reasoning budget used):
SWE-bench Verified, which tests whether a model can resolve real GitHub issues end-to-end:
- Claude Sonnet 4.6: 79.6
- Claude Sonnet 4.5: 77.2
- Gemini 3: 76.2
- OrcaSAQ2 27B: 70.0
- Qwen3-Coder-480B-A35B: 69.6
- Gemini 2.5 Pro: 63.8
- GPT-4.1: 54.6
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
OrcaSAQ2 edges out Qwen3-Coder-480B-A35B, a model roughly 18 times larger in parameter count, while running from a 12.3GB checkpoint.
Terminal-Bench 2.1, which measures agentic performance in terminal environments:
- Gemini 3.1 Pro / Terminus 2: 70.7
- Claude Opus 4.6 / Claude Code: 70.1
- Claude Opus 4.6 / Terminus 2: 63.8
- Claude Sonnet 4.6 / Claude Code: 58.5
- OrcaSAQ2 27B: 58.4
- Gemini 3 Flash / Gemini CLI: 56.9
- GPT-5.4 / Terminus 2: 54.8
- Claude Sonnet 4.6 / Terminus 2: 51.5
Here OrcaSAQ2 lands just behind Claude Sonnet 4.6 paired with Claude Code, and ahead of several frontier model and agent-harness combinations. Given the caveat that these scores use different agent stacks, the results should be read as evidence OrcaSAQ2 is competitive in its class rather than as a definitive ranking.
Why does long-horizon stability matter more than single-turn accuracy?
Short benchmark prompts can mask small quality degradation. A single wrong token in a one-shot answer might not change the outcome. In an agent loop, it can.
Agent execution follows a repeating cycle: plan, act, observe, decide, recover, repeat. If a quantization-induced error changes a single tool call, that changes the state of the environment the agent is operating in. Every subsequent decision inherits that changed state. Over a long enough trajectory, small per-step errors can compound into behavior that looks nothing like the original model’s output.
This is why OrcaRouter treats long-horizon benchmarks as a stress test that plain perplexity can’t substitute for. Perplexity asks how similar the next-token distribution is. Long-horizon evaluation asks whether the model can still complete a multi-step task after dozens of interdependent decisions. Both matter for judging whether a compressed model is actually usable in production agent systems, not just accurate on a language modeling test set.
Is OrcaSAQ2 27B worth deploying?
For teams that need a capable coding or tool-use agent but can’t fit a 54GB checkpoint on their hardware, OrcaSAQ2 is a practical option. The 12.3GB footprint fits on a single 16GB GPU with roughly 3.7GB left over for KV cache and runtime overhead, which puts a 27B-class reasoning model within reach of workstation and prosumer GPU setups that couldn’t otherwise run it.
The tradeoffs are real, though. Vision is not supported (this is a text-only checkpoint). Deployment requires a specific vLLM integration (OrcaSAQ2-kernel) rather than standard off-the-shelf quantization tooling. And the 262K architectural context limit doesn’t mean that much context will actually fit in a 16GB memory envelope; OrcaRouter suggests around 32K tokens of interactive context as a practical starting point on that hardware tier, with usable context scaling up on larger GPUs.
Serving configuration also matters. With MTP (multi-token prediction) speculative decoding enabled, single-stream throughput hits 90.1 tokens/sec, a 38% improvement over the 65.3 tokens/sec baseline, which benefits interactive use cases like coding assistants. But MTP reduces KV pool capacity (14,563 tokens vs. 29,354 tokens) and underperforms on heavily batched workloads, where turning MTP off yields over 330 tokens/sec across 8 or 16 concurrent streams. Teams should benchmark both configurations against their actual traffic pattern before choosing one.
Frequently Asked Questions
What base model does OrcaSAQ2 27B use?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
It quantizes Qwen/Qwen3.8-27B, a 27-billion-parameter model with a Qwen3_5ForCausalLM architecture, 64 layers, and hybrid Gated DeltaNet plus full-attention design supporting up to 262,144 tokens of context.
How much smaller is OrcaSAQ2 than the original model?
The checkpoint drops from 54GB in BF16 to 12.3GB, a 77.2% storage reduction, achieved through mixed-precision quantization averaging 3.21 bits per weight rather than a uniform bit width across all layers.
Does 3-bit quantization hurt accuracy?
On standard fidelity metrics, the loss is minimal: perplexity rises just 0.02% and top-1 token agreement with the BF16 model sits at 93.2%. On agent benchmarks like SWE-bench Verified (70.0%) and Terminal-Bench 2.1 (58.4%), it performs competitively against much larger and higher-precision models, though OrcaRouter notes these public scores use different agent harnesses and aren’t a strict apples-to-apples comparison.
What hardware do I need to run OrcaSAQ2 27B?
The 12.3GB checkpoint fits on a 16GB GPU, leaving about 3.7GB for KV cache and runtime overhead. A practical starting context length on that hardware tier is around 32K tokens; larger GPUs can push closer to the model’s 262K architectural maximum.
Is OrcaSAQ2 open source?
Yes. It’s released under Apache-2.0, the same license as the base Qwen3.8-27B model, and is available on Hugging Face with weights, a vLLM serving kernel, and generation samples included.
