Bonsai 2 27B Tested: Does the 98% Benchmark Claim Hold Up?
Bonsai 2 27B claims 98.2% of full-precision performance at a ninth of the size. Real coding, vision, and language tests find the gaps.
What is Bonsai 2 27B?
Bonsai 2 27B is a ternary-quantized version of Qwen3.8-27B, a 27 billion parameter model that normally needs about 54 GB to load at full precision. Bonsai 2 compresses it down to under 8 GB depending on the packing format, claiming to retain 98.2% of the original’s benchmark performance while running roughly nine times smaller. In hands-on testing across bug-fixing, code generation, vision, and multilingual tasks, the model performed unevenly: strong on vision description, shaky on debugging and multi-language output, and inconsistent on straightforward code generation.
TL;DR
- Bonsai 2 27B shrinks Qwen3.8-27B from roughly 54 GB down to somewhere between 5.95 GB and 8.60 GB on disk, depending on whether you use the GGUF or MLX packing, by storing weights as ternary values (-1, 0, +1) instead of 16-bit floats.
- The model card claims 84.78 average across 14 thinking-mode benchmarks, which it reports as 98.2% of the full-precision baseline’s score and well above a comparable IQ2_XXS quantization.
- A Hadamard rotation is baked into the weights before ternary compression, a mathematical reshuffling step the model card says prevents the accuracy collapse that usually comes with squeezing weights this hard.
- In a real hands-on test, the model missed the actual planted bug in a multi-service Docker application, instead reasoning its way right up to the correct answer before talking itself out of it and fixing an unrelated issue.
- A simple single-file HTML code generation task (a latte-making animation) failed to produce a working reset button even after three rounds of follow-up correction requests.
- Vision description and creative writing held up well, producing an accurate and appropriately over-the-top response to an image-based prompt.
- The multilingual translation test broke down entirely, with the model getting stuck in a repetitive reasoning loop on languages like Tamil and never completing the task.
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
How does Bonsai 2 achieve a 9x size reduction?
The compression rests on three combined techniques, according to the model card and its accompanying whitepaper. First, every weight in the language backbone gets reduced to one of three values: -1, 0, or 1. This is “ternary” quantization, and it’s why the model card describes an effective rate of about 1.72 bits per weight instead of the usual 16, an idealized 9.3x reduction versus FP16. In practice, the packaging format used adds some overhead: the GGUF PTQ1_0 packing lands at 1.75 bits/weight (5.95 GB), while the MLX 2-bit format used on Apple Silicon comes in at 2.25 bits/weight (7.67 GB for the language model, 8.60 GB including the vision tower).
Second, before the ternary assignment happens, each weight matrix is transformed using a Hadamard rotation, a fixed mathematical reshuffling applied in blocks. The idea is that no single weight ends up carrying too much information, which is what typically causes accuracy to collapse when models are pushed into very low bit-widths. The rotation costs nothing extra to store because it’s folded permanently into the weights offline, but it does mean the model needs matching custom kernels at inference time. Stock llama.cpp can’t run these files; you need the maintainer’s fork, and ordinary MLX loaders will produce silently wrong output rather than erroring out if you skip the required runtime.
Third, the model inherits a hybrid attention architecture from Qwen3.8, roughly 75% linear attention and 25% full attention. Linear attention is cheaper to compute, which is what allows the model to support a 262,000 token context window without the usual attention bottleneck that makes long-context inference expensive.
What are the real hardware requirements?
Despite the claimed sub-6 GB footprint for the most compact GGUF packing, actual VRAM use in testing ran higher. Served through llama.cpp with default settings, the model consumed just over 12 GB of VRAM, largely because llama.cpp reserves KV cache space for four parallel request slots by default. Dropping to a single parallel slot cuts that down to roughly 8 GB. The raw model size on disk was measured at just over 7 GB in that test.
On Apple Silicon, the model card reports around 47 tokens per second on an M5 Max laptop and about 28-29 tokens per second on an M5 Pro, with prompt processing dropping off significantly on older M4 Pro hardware where prefill becomes the bottleneck for long prompts rather than generation speed. On the GPU side, throughput varies by packing format and card generation, with the two GGUF packings (PTQ1_0 and PQ2_0) trading places depending on whether the hardware is memory-bandwidth-limited or compute-limited.
Does the model actually find and fix bugs?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
This is where the gap between the benchmark claim and real usage showed up most clearly. The test used a full Docker-based application (Postgres, FastAPI, an auto-refreshing dashboard) simulating a wildlife migration monitoring tool for East African parks. A deliberately planted bug caused a 24-hour activity gauge to compute its cutoff using local server time instead of UTC, silently dropping the most recent three hours of real data and under-reporting activity.
Given an open-ended instruction to find and fix a bug in the application, with no hints about where it was, the model’s reasoning trace actually worked through the correct logic: it noticed the time zone mismatch, walked through the math, and then talked itself out of the conclusion, deciding the code was fine when it wasn’t. It ended up fixing an unrelated discrepancy in a separate 14-day trend chart instead. After more than an hour of processing (comparable to the time the full-precision model would take), the planted bug remained unfixed, confirmed by directly querying the running containers afterward.
Is Bonsai 2 reliable for code generation?
A simpler test asked the model to generate a single self-contained HTML file simulating a barista making a latte, with a multi-step animation sequence and a reset button. The visual output looked reasonable on the first pass, with all the expected steps present. But the reset/start button didn’t work. Across three separate rounds of pointing out that clicking the button did nothing, the model claimed each time to have found and fixed the issue, but the button remained non-functional through the final attempt. This is a basic interactivity bug that a fully capable coding model would typically catch and resolve quickly, and it’s a useful signal that compression can affect not just raw benchmark scores but the kind of careful, iterative self-correction that agentic coding workflows depend on.
How well does it handle vision and multilingual tasks?
Vision performance was the strongest result in testing. Given an image and asked to describe it accurately and then write an over-the-top complaint letter in the voice of a homeowners association member, the model produced a coherent, detailed, and appropriately funny response, correctly identifying the scene’s details and matching the requested tone. It wasn’t judged as good as the uncompressed Qwen3.8 model on the same task, but it was described as a clear pass.
Multilingual output was the clear failure point. Asked to translate a simple financial sentence into multiple languages, the model’s reasoning process got stuck in a repetitive loop on languages like Tamil, never producing usable output. This result was notably worse than the previous Bonsai model version, which had performed well on similar tasks in earlier testing.
Is Bonsai 2 27B worth using?
The honest answer depends entirely on the workload. The model card’s own benchmark table, which it explicitly frames as the vendor’s claim rather than independently verified results, reports scores close to the full-precision baseline across agentic tool use, coding, instruction following, reasoning, math, and vision, with math at 96.57 and coding at 89.42 out of a possible 100-point scale implied by the 84.78 average. Real-world testing didn’t fully replicate that consistency. The model handled vision-language description well and showed genuine reasoning ability on a hard debugging task, even if it ultimately reached the wrong conclusion. But it struggled with a basic self-correction loop in code generation and broke down completely on multilingual translation, a regression from the previous Bonsai version.
For anyone considering it, the size and context window are real advantages: a 27B-class model with a 262K context window running under 8 GB of memory is a meaningful capability shift for local and edge deployment. But the “98.2% retention” figure should be read as a benchmark average, not a guarantee that every task category holds up equally, and multilingual or fine-grained debugging work appears to be where the compression costs show up most.
Frequently Asked Questions
What base model is Bonsai 2 27B built on?
It’s derived from Qwen3.8-27B, a 27 billion parameter hybrid-attention model, with the architecture unchanged and the weights compressed through ternary quantization.
How much smaller is Bonsai 2 than the original model?
The full-precision FP16 baseline needs about 54 GB. Bonsai 2 comes in between 5.95 GB and 8.60 GB depending on the packing format used, roughly a 7x to 9x reduction.
Can I run Bonsai 2 27B with standard llama.cpp?
No. It requires a custom fork of llama.cpp (or MLX with a bundled loader on Apple Silicon) because the model uses custom kernels to handle its rotated ternary weight format. Stock runtimes either fail to load the file or produce incorrect output.
Does Bonsai 2 really retain 98% of the original model’s performance?
That figure comes from the model’s own benchmark table, run under the vendor’s own evaluation setup. Independent hands-on testing found the model excelled at some tasks like vision description but underperformed noticeably on debugging accuracy and multilingual translation.
What’s the practical benefit of the ternary compression approach?
Beyond the smaller file size, it allows a 27B-class model with a 262,000 token context window to run on consumer hardware, including laptops, that couldn’t otherwise fit a model of this scale in memory at all.
