Underdog Saluki 27B: An 8GB Model That Beats Its 54GB Base on Tool Calls
Underdog Saluki 27B compresses Qwen3.8-27B to under 8GB and still beats the full model on BFCL tool-calling benchmarks. Here's the data.
What is Underdog Saluki 27B?
Underdog Saluki 27B 1.0 is a compressed version of Qwen3.8-27B, shrunk from 54 GB down to 7.89 GB, built specifically to preserve tool-calling ability during quantization. On Underdog Bench, a 120-task slice of the Berkeley Function Calling Leaderboard (BFCL v4), Saluki scores 88 out of 120 against 84 for the full-size base model, and it holds a wider lead on parallel tool calls (42 vs 35 on 100 BFCL v4 tasks). It runs on stock llama.cpp with no custom runtime required.
TL;DR
- Saluki beats its own base model on tool calling despite being roughly 7x smaller, scoring 88/120 on Underdog Bench versus 84/120 for the full 54GB Qwen3.8-27B.
- The gap widens on parallel tool calls, where Saluki hits 42/100 against 35/100 for the full model, using the official BFCL v4 checker.
- Quality holds up broadly: Saluki retains about 96% of the full model’s performance averaged across nine benchmarks, including small wins on IFEval and IFBench.
- Math reasoning takes the biggest hit, with Saluki landing around 79 to 80 on AIME 2025/2026 (avg@4) versus roughly 95 to 97 for the full-size model, a drop of 15 to 17 points.
- Coding and some structured tasks also slip slightly, with SWE-bench Verified at 30/50 fixed issues versus 33/50, and MBPP+ at 78.0 versus 83.9.
- The file size makes it runnable on modest hardware, at 7.89 GB for the main weights plus an optional 0.6 to 0.9 GB vision add-on, versus 54 GB for the uncompressed original.
- It ships under Apache 2.0, built on Qwen3.8-27B and a prior compression recipe (GSQ-RCO) from ISTA-DASLab, with benchmark tasks sourced from BFCL.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
How does an 8GB model outscore its 54GB base?
This is the detail worth sitting with: Saluki isn’t a different, newer model beating an old one. It’s a quantized derivative of Qwen3.8-27B beating Qwen3.8-27B itself, on the same task set, in the same harness.
The explanation isn’t that compression adds capability. It’s that standard quantization degrades specific skills unevenly, and tool calling is one of the skills that typically degrades hardest, because it depends on the model reliably producing exact syntax (function names, argument structures, JSON formatting) rather than just “sounding right.” A model can lose a little fluency almost everywhere and still read fine to a human, while the same compression scrambles the precise token sequences a tool-calling parser needs.
Underdog’s approach targets that weak point directly. Rather than quantizing uniformly and hoping tool calling survives, the model card describes a process tuned specifically to keep tool calling intact, likely by weighting the compression or calibration data toward function-calling patterns. The result lands in the quirky but real territory where the compressed model outperforms its source on the exact behavior it was tuned to protect, even though it’s worse than the source on other axes like competition math.
It’s also worth noting the test size: 120 tasks for Underdog Bench and 100 for the parallel-call set are modest samples. The model card itself flags that a few tasks of difference “is run-to-run variation,” which is a reasonable caveat. An 88 vs 84 split on 120 tasks is a real, reproducible gap in the reported runs, but it’s not the same statistical weight as a benchmark with thousands of tasks.
What do the benchmark numbers actually show?
Breaking down the reported scores by category:
Tool calling and agentic use, where Saluki was specifically tuned, it leads:
- BFCL-based tool calling: 88 vs 84 (out of 120)
- Parallel tool calls: 42 vs 35 (out of 100)
- IFEval (instruction following, prompt-loose): 93.5 vs 91.5
- IFBench: 72.7 vs 71.0
General capability, where compression costs show up, it trails:
- SWE-bench Verified: 30 vs 33 (out of 50 issues fixed)
- MBPP+ (coding): 78.0 vs 83.9
- MuSR (multi-step reasoning): 67.5 vs 79.6
- AIME 2025 (avg@4): 79.2 vs 96.7
- AIME 2026 (avg@4): 80.0 vs 94.6
The pattern is consistent: tasks that reward precise, structured output (tool calls, instruction compliance) hold steady or improve, while tasks that reward sustained multi-step reasoning or competition-level math drop meaningfully. AIME is the clearest casualty, losing 15 to 17 points depending on the year. MuSR drops about 12 points. The model card is direct about this, stating Saluki keeps roughly 82 to 85% of the full model’s competition math score.
One caveat baked into the numbers: the “full size” scores for MBPP+, MuSR, and both AIME runs are listed as public scores, meaning they come from a different evaluation harness than the one used for Saluki, SWE-bench, IFEval, and IFBench. Cross-harness comparisons add noise, so those particular gaps should be read as directional rather than precise.
Where does Saluki still fall short?
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
The model card lists its own limitations plainly, which is useful since it saves the work of guessing:
- Competition math is the weak point. The AIME drop is the largest in the entire benchmark table, and it’s the clearest tradeoff for the size reduction.
- Letter-level puzzles struggle. Tasks like palindrome checks, vowel-counting rules, or alphabetical ordering are called out specifically as a weak spot, likely because fine-grained character-level tracking is sensitive to quantization noise.
- Parallel calls aren’t perfectly clean. About one in five parallel-call responses has a small formatting slip, even though the overall parallel-call score beats the full model.
- Thinking mode runs long. With reasoning turned on (the default), Saluki tends to produce lengthy chains of thought before answering, which affects latency even if it doesn’t affect correctness.
- Vision isn’t included by default. The base 7.89 GB file is text-only. Image support requires downloading a separate projector file (either 928 MB or 629 MB depending on precision).
None of these are hidden in fine print. They’re stated directly in the model’s own documentation, which is a reasonable signal that the benchmark claims are being presented with their limits attached rather than oversold.
Is Underdog Saluki 27B worth using over the full model?
It depends entirely on what the workload needs. If the use case is tool-calling-heavy (agents, function-calling pipelines, structured API orchestration) and hardware is constrained, Saluki is a straightforward upgrade: better tool-calling accuracy, a fraction of the memory footprint, and compatibility with stock llama.cpp rather than a custom runtime. Running a 7.89 GB file instead of 54 GB opens the door to consumer GPUs and smaller deployment boxes that couldn’t practically host the full model at all.
If the workload leans on competition-level math, dense multi-step reasoning (MuSR-style tasks), or precise character-level manipulation, the full-size Qwen3.8-27B is still the stronger choice. The AIME and MuSR gaps are large enough that they’d matter for math tutoring tools, formal reasoning chains, or anything close to a research-math context.
For general coding work, the gap is smaller but still present: SWE-bench Verified drops from 33 to 30 out of 50, and MBPP+ drops about six points. That’s a real but modest tax for a model one-seventh the size.
Frequently Asked Questions
What base model is Underdog Saluki 27B built on?
It’s built on Qwen3.8-27B from the Qwen team, using a compression approach that also draws on Qwen3.8-27B-GSQ-RCO-GGUF from ISTA-DASLab. Both underlying components are released under Apache 2.0, as is Saluki itself.
How is Saluki able to outperform its own base model?
Standard compression typically degrades tool-calling accuracy disproportionately because it depends on precise syntax rather than general fluency. Saluki’s tuning process specifically targets preserving tool-calling behavior, which is why it scores higher than the uncompressed model on BFCL-based tasks even though it loses ground on math and reasoning benchmarks.
What hardware does Saluki need to run?
The main GGUF file is 7.89 GB, small enough to run on consumer GPUs that couldn’t host the 54 GB full-size model. It runs in stock llama.cpp with standard flags (--jinja, -ngl, -fa on), no custom runtime needed.
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
Does Saluki support image input?
Not by default. The core model is text-only. Vision support requires downloading a separate projector file, available in a 928 MB (F16) or 629 MB (Q8_0) version, and loading it alongside the main model with the --mmproj flag.
What’s the biggest tradeoff for the smaller size?
Competition math. Saluki retains only about 82 to 85% of the full model’s AIME performance, the largest gap across every benchmark reported on the model card. Multi-step reasoning (MuSR) and letter-level instruction puzzles are the other notable weak points.