How to Run OrcaSAQ2 27B on a 16GB GPU
OrcaSAQ2 27B compresses a 54GB model to 12.3GB with near-BF16 fidelity, letting a single 16GB GPU run vLLM agents at up to 90 tok/s.
What is OrcaSAQ2 27B?
OrcaSAQ2 27B is a quantized version of Qwen3.8-27B, compressed from a 54GB BF16 checkpoint down to 12.3GB using a proprietary sensitivity-aware mixed-precision quantization method built by OrcaRouter. The compressed model runs on a single consumer GPU with 16GB of VRAM, retains the 262K context window of the base model, and reports a perplexity increase of only 0.02% against the full-precision original. It ships with thinking mode, tool calling, and MTP speculative decoding, and it’s served through vLLM.
TL;DR
- OrcaSAQ2 27B shrinks a 54GB checkpoint to 12.3GB, a 77.2% storage reduction, while keeping perplexity within +0.02% of the BF16 reference model.
- The model averages 3.21 bits per weight across its decoder layers instead of a uniform bit-width, which is how it preserves fidelity better than a flat 3-bit or 4-bit quantization would.
- On a 16GB GPU with a 15.7GiB memory cap, vLLM serves it at 65.3 tok/s single-stream with MTP off, or 90.1 tok/s with MTP on, a 38% speedup for interactive use.
- It scores 70.0% on SWE-bench Verified and 58.4% on Terminal-Bench 2.1, landing in the range of much larger frontier models on public leaderboards, though those scores use different agent scaffolds and aren’t strict apples-to-apples comparisons.
- Multi-stream throughput actually drops with MTP enabled (219 to 220 tok/s at 8 to 16 streams versus 332 to 333 tok/s with MTP off), because speculative decoding trades KV-cache capacity for single-stream speed.
- A practical starting point on 16GB hardware is around 32K tokens of context, even though the architecture supports up to 262,144 tokens.
- The checkpoint is text-only (no vision tower) and requires a dedicated vLLM integration package to run the custom kernel.
How does OrcaSAQ2 fit a 27B model into 12.3GB?
The core technique is what OrcaRouter calls sensitivity-aware mixed-precision quantization. Instead of quantizing every weight to the same bit-width, the method identifies which parts of the network are more sensitive to precision loss and allocates bits accordingly. The result is a decoder that averages 3.21 bits per weight rather than a flat 16-bit (BF16), 8-bit, or even uniform 3-bit or 4-bit scheme.
The measured outcome, evaluated on WikiText-2 across 16,376 predicted tokens, shows:
- Perplexity of 5.6482 versus 5.6468 for BF16 (a 0.02% delta)
- 93.2% top-1 token agreement with the BF16 model
- A mean KL divergence of 0.031
Those numbers matter because perplexity alone can mask degradation that shows up later in a task. A model can predict similar token distributions on a static text corpus and still make a different tool call or planning decision in an agent loop. OrcaRouter’s documentation is explicit about this gap: perplexity asks whether the next-token distribution looks similar, but agent tasks ask whether the model can still finish the job after dozens or hundreds of decisions. That’s why the model card pairs static fidelity metrics with long-horizon benchmark results rather than reporting perplexity in isolation.
What hardware do you need to run it?
OrcaSAQ2’s checkpoint is 12.3GB, which fits inside a single 16GB consumer GPU with room left over for vLLM overhead, the KV cache, and CUDA graph structures. Under a 15.7GiB memory cap (leaving a small buffer below the 16GB ceiling), the model runs with a KV pool of roughly 14,563 tokens with MTP enabled or 29,354 tokens with MTP disabled.
The practical implication: full use of the model’s 262K context window isn’t realistic on a single 16GB card. The published guidance suggests starting around 32K tokens of interactive context and tuning from there based on batch size, concurrency, and whether MTP is active. Larger context windows are reachable on GPUs with more VRAM or through multi-GPU serving, but the headline claim, running a 27B-class model locally at all, is specifically about the 16GB tier.
What throughput can you expect from vLLM?
OrcaRouter published two serving configurations, both measured under the same 15.7GiB cap:
| Configuration | 1 Stream | 8 Streams | 16 Streams | KV Pool |
|---|---|---|---|---|
| vLLM, MTP off | 65.3 tok/s | 332 tok/s | 333 tok/s | 29,354 tokens |
| vLLM, MTP on | 90.1 tok/s | 220 tok/s | 219 tok/s | 14,563 tokens |
MTP (multi-token prediction) is a speculative decoding technique where the model predicts several tokens ahead and verifies them in a batch, reducing the number of full forward passes needed per generated token. On this checkpoint it delivers a 38% single-stream speedup, which matters most for interactive use cases like a coding assistant or a single agent working through a task step by step.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
The tradeoff shows up under concurrency. Because MTP consumes extra compute and KV-cache capacity to support the speculative heads, batched throughput at 8 and 16 concurrent streams is actually lower with MTP on than with it off. If you’re serving many simultaneous requests, MTP off and a larger KV pool is the better choice. If you’re running one agent or one user at a time, MTP on is faster. The model card’s own advice is to benchmark both configurations against your actual workload rather than assume one is universally better.
How does it perform on agent and coding benchmarks?
OrcaRouter reports two public benchmark results for OrcaSAQ2 27B:
- SWE-bench Verified: 70.0%, ahead of Qwen3-Coder-480B-A35B (69.6%) and GPT-4.1 (54.6%), and behind Gemini 3 (76.2%) and Claude Sonnet 4.5/4.6 (77.2% and 79.6%).
- Terminal-Bench 2.1: 58.4%, close to Claude Sonnet 4.6 running Claude Code (58.5%) and ahead of GPT-5.4 paired with Terminus 2 (54.8%).
These numbers come with an important caveat that OrcaRouter itself states: agent benchmarks depend heavily on the scaffold, tool access, reasoning budget, and timeout settings around the model, not just the model’s raw weights. A 27B model scoring near models many times its size doesn’t mean the underlying capability is equivalent, it means the specific benchmark harness produced comparable pass rates. Treat the table as a reference point for what’s achievable, not a certified head-to-head ranking.
How do you actually deploy it?
The setup uses vLLM plus a custom kernel package for the quantization format:
pip install -U vllm huggingface_hub
pip install git+https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel
hf download orcarouter/OrcaSAQ2-27B --local-dir ./OrcaSAQ2-27B
vllm serve ./OrcaSAQ2-27B \
--served-model-name OrcaSAQ2-27B \
--max-model-len 262144 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'Once running, it exposes an OpenAI-compatible endpoint, so existing code using the openai Python client just needs base_url pointed at the local server. Recommended sampling settings are temperature 1.0, top_p 0.95, top_k 20, and thinking mode is on by default. For production agent use, it’s worth benchmarking with the actual tool schema and reasoning budget your application uses, since the published numbers reflect OrcaRouter’s own test harness.
Is it worth running locally?
For anyone who needs a 27B-class reasoning model on a single 16GB GPU, OrcaSAQ2 solves a genuine capacity problem: the unquantized model simply doesn’t fit. The measured fidelity numbers (93.2% top-1 agreement, +0.02% perplexity) suggest the compression is unusually well-targeted rather than a blunt round-to-3-bit approach, and the SWE-bench and Terminal-Bench scores back up that it retains real coding and agent capability. The tradeoffs are real too: it’s text-only, it requires a nonstandard kernel package rather than running out of the box in vanilla vLLM, and the full 262K context is aspirational on 16GB hardware rather than something you’ll actually use. For coding agents, terminal agents, and other tool-using workloads on a budget GPU, it’s a reasonable fit. For vision tasks or workloads needing very large context windows on modest hardware, it isn’t.
Frequently Asked Questions
How much VRAM does OrcaSAQ2 27B actually need?
The checkpoint itself is 12.3GB. Running it comfortably with vLLM, including KV cache and overhead, was tested under a 15.7GiB memory cap, so a 16GB GPU is the practical minimum.
Does OrcaSAQ2 support the full 262K context window on a 16GB GPU?
The architecture supports up to 262,144 tokens, but that full window won’t fit in 16GB of VRAM alongside the model weights. A realistic starting point on that hardware is around 32K tokens of context.
What’s the difference between running it with MTP on versus off?
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
MTP (multi-token prediction / speculative decoding) speeds up single-stream generation by about 38% but reduces the KV-cache pool and lowers throughput under heavy concurrency. Use it for single-user or single-agent workloads and disable it for high-concurrency serving.
Can OrcaSAQ2 handle images or other modalities?
No. This checkpoint is text-only. The vision tower from the base model is not included in the quantized release.
How does its coding performance compare to larger models?
It scored 70.0% on SWE-bench Verified and 58.4% on Terminal-Bench 2.1, putting it close to some much larger models on those specific public leaderboards. These scores depend on the agent scaffold used during testing, so they shouldn’t be read as a definitive ranking against frontier models.
