Underdog Saluki 27B: How Much Does Compression Cost You?
Underdog Saluki 27B shrinks Qwen3.8-27B to under 8GB. Here's what the benchmarks say about tool calling, coding, and math losses.
What is Underdog Saluki 27B?
Underdog Saluki 27B is a compressed version of Qwen3.8-27B, shrunk from 54 GB down to 7.89 GB, a reduction of roughly 85%. It’s packaged as a standard GGUF file that runs on stock llama.cpp, with tool calling as the specific capability the compression was tuned to protect. On the benchmark Underdog built to measure that goal (120 tasks drawn from the Berkeley Function Calling Leaderboard v4), the compressed model actually scored higher than the full-size original: 88 out of 120 versus 84. That result is the headline, but it’s not the whole story. Across nine benchmarks covering coding, instruction-following, and competition math, Saluki retains about 96% of the full model’s performance on average, with the math benchmarks taking the biggest hit.
TL;DR
- Tool calling held up and even improved, with Saluki scoring 88/120 on Underdog Bench versus 84/120 for the full-size Qwen3.8-27B, and 42 versus 35 on parallel tool-call tasks.
- The file size drops from 54 GB to 7.89 GB, an 85% reduction, using an IQ2-mix quantization that runs in stock llama.cpp without custom runtimes.
- Competition math is the weak spot, with AIME 2025 scores falling from 96.7 (public) to 79.2, meaning Saluki keeps only about 82 to 85% of the full model’s score there.
- Instruction-following benchmarks actually edged upward, with IFEval at 93.5 versus a public score of 91.5, though the model struggles specifically with letter-level puzzles like palindromes and alphabetical ordering.
- Coding and reasoning benchmarks landed in between, with SWE-bench Verified at 30 versus 33 fixed issues and MuSR dropping more sharply from 79.6 to 67.5.
- A vision add-on is optional and separate, adding 629 MB to 928 MB on top of the text-only base model for image support.
- The comparison isn’t perfectly apples-to-apples, since the public scores for the full-size model come from a different benchmark harness than the one Underdog used for its own testing.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
How was Underdog Saluki 27B compressed?
Saluki is built on top of Qwen3.8-27B, with an intermediate step through ISTA-DASLab’s Qwen3.8-27B-GSQ-RCO-GGUF, before Underdog applied its own quantization mix (labeled IQ2-mix) to get the file down to 7.89 GB. All three components, the base model, the intermediate quantization, and Underdog’s own work, are released under Apache 2.0.
The practical result is a model that loads and runs through the same llama-server setup as any other GGUF, with the standard --jinja flag handling the Qwen3.8 chat template for tool calls and thinking mode. There’s no special runtime or proprietary serving stack involved, which matters for anyone trying to deploy a 27B-class model on hardware that can’t fit 54 GB in memory.
Why does tool calling score higher after compression?
This is the most counterintuitive result in the release: a model that’s 85% smaller outscoring its full-size parent on tool calling (88 vs 84 on single calls, 42 vs 35 on parallel calls). The model card doesn’t claim this is a general law of quantization, it’s specific to how Underdog tuned the compression, presumably weighting the quantization process to preserve the parts of the model responsible for structured output and function-call formatting at the expense of other capabilities.
It’s worth noting the test set size: 120 tasks for the main benchmark and 100 for parallel calls. The model card itself flags this as “a modest test” and notes that a gap of a few tasks could be run-to-run variation rather than a real signal. A four-point edge (88 vs 84) sits in a range where some caution is warranted, even if the trend (tool calling survives compression well, possibly even benefits from targeted tuning) looks real across both the single and parallel call tests.
How much does compression hurt coding and reasoning?
Coding and general reasoning benchmarks show a more typical compression story: small, consistent losses rather than gains.
- SWE-bench Verified: 30 vs 33 out of 50 issues fixed. A real but modest drop.
- MBPP+: 78.0 vs a public score of 83.9.
- MuSR: 67.5 vs a public score of 79.6, the largest non-math drop in the set.
- IFBench: 72.7 vs a public score of 71.0, another case where the compressed model nudges ahead.
The pattern suggests compression doesn’t hit all capabilities evenly. Structured, format-heavy tasks (tool calls, instruction-following) held up or improved, while tasks requiring broader contextual reasoning (MuSR, which tests multi-step reasoning through narratives) degraded more. SWE-bench sits in the middle, which makes sense given that fixing real GitHub issues requires both structured output and genuine code comprehension.
What happens to math performance under compression?
Math is where the compression cost is clearest and largest. AIME 2025 (average of 4 runs) dropped from a public score of 96.7 to 79.2, and AIME 2026 dropped from 94.6 to 80.0. The model card states plainly that Saluki keeps “about 82 to 85%” of the full model’s competition math score, which is the steepest retention loss of any benchmark listed.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
This tracks with a broader pattern seen across the quantization literature: multi-step symbolic reasoning, where small errors compound over long chains of computation, tends to be more fragile under aggressive compression than pattern-matching or format-following tasks. A model can still produce fluent, well-structured text while losing precision on problems that require many consecutive correct logical steps.
Is Underdog Saluki 27B worth using over the full model?
It depends entirely on what you’re building. If you need a model for agentic workflows, tool-calling pipelines, or API orchestration, and you’re memory-constrained, Saluki’s results suggest you won’t lose much, and might even see better tool-calling consistency, while using roughly a seventh of the storage and memory footprint of the full model.
If your workload leans on competition-level math or complex multi-hop reasoning (the kind of problems AIME and MuSR are built to test), the full-size Qwen3.8-27B is the safer choice. The retention numbers there (82 to 85% on math, a roughly 12-point drop on MuSR) are large enough to matter for anything where precision is the point.
The model card is also upfront about rough edges: around a fifth of parallel tool-call replies have minor formatting slips, and the model is specifically weak on letter-level puzzles like palindrome checks or alphabetical ordering. Thinking mode is on by default and can run long before producing an answer, which is worth knowing if latency matters for your use case. Vision support requires a separate add-on file (629 MB to 928 MB) since the base model is text-only.
Frequently Asked Questions
What base model is Underdog Saluki 27B built on?
It’s built on Qwen3.8-27B, released by the Qwen team, routed through an intermediate quantization (Qwen3.8-27B-GSQ-RCO-GGUF by ISTA-DASLab) before Underdog’s own compression pass. All components are Apache 2.0 licensed.
How big is Underdog Saluki 27B compared to the full model?
The compressed version is 7.89 GB versus 54 GB for the full-size Qwen3.8-27B, a reduction of about 85%. It ships as a single GGUF file (IQ2-mix quantization) runnable in stock llama.cpp.
Does compression hurt tool-calling accuracy?
In this case, no. Saluki scored 88 out of 120 on Underdog’s tool-calling benchmark versus 84 for the full-size model, and 42 versus 35 on parallel tool-call tasks. The benchmark is modest in size (120 and 100 tasks respectively), so treat the exact margins with some caution, but the overall direction (tool calling surviving or improving under compression) is consistent across both tests.
What’s the biggest weakness of the compressed model?
Competition math. AIME 2025 and AIME 2026 scores dropped to roughly 79 to 80 from public scores in the mid-90s for the full model, meaning Saluki retains only about 82 to 85% of the full model’s math performance. It’s also specifically weak on letter-level instruction puzzles like palindromes and alphabetical ordering.
Can Underdog Saluki 27B handle images?
Yes, but only with an optional add-on file (either 928 MB or 629 MB depending on precision). The core model file is text-only; vision support has to be loaded separately via the --mmproj flag in llama.cpp.