Bonsai 2 27B: A 27B Model That Runs in Under 6GB on a Laptop
Bonsai 2 27B uses ternary quantization to shrink a 27B model to under 6GB while keeping 98% of FP16 performance. Here's how it works.
What is Bonsai 2 27B?
Bonsai 2 27B is a quantized version of Qwen3.8-27B, released by Prism ML, that stores its weights as ternary values (-1, 0, +1) instead of the usual 16-bit floats. The result is a 27-billion-parameter model that fits in roughly 6 to 8.6GB depending on packaging, runs at about 47 tokens per second on an Apple M5 Max laptop, and retains 98.2% of the original FP16 model’s benchmark performance across 14 reasoning tests. That combination, near-full intelligence at under a tenth of the original size, is what makes it notable.
TL;DR
- Ternary quantization represents each weight as one of three values (-1, 0, +1) with a shared scale per group of 128 weights, cutting the theoretical cost to about 1.72 bits per weight versus 16 for FP16.
- The language model shrinks from roughly 54GB to under 8GB, with the full package (language model plus vision tower) landing at 8.60GB in the MLX build, or as low as 5.95GB in the GGUF PTQ1_0 packing.
- Benchmark retention is unusually high for sub-4-bit quantization: Bonsai 2 scores 84.78 average across 14 thinking-mode benchmarks, compared to 72.59 for a conventional IQ2_XXS build and within 0.4 points of a 4-bit build three times its size.
- A blockwise Hadamard rotation is folded into the stored weights before ternary assignment, which is a key reason the model avoids the usual collapse in reasoning ability seen in low-bit quantization.
- Throughput varies by GPU generation and packing format: newer cards (RTX 5090, Blackwell, H100) favor the denser PQ2_0 packing, while Ada-generation cards and the L4 favor the PTQ1_0 packing that moves less data per step.
- The model needs custom kernels, not stock loaders. Ordinary MLX or llama.cpp builds will either fail to load it or silently produce wrong output.
- It’s released under Apache 2.0, with separate MLX and GGUF packages for Apple Silicon and CUDA/Metal/CPU respectively.
How does ternary quantization work?
Standard quantization reduces the precision of each weight, going from 16-bit floats down to 8, 4, or sometimes 2 bits. Ternary quantization goes further conceptually: each weight can only be one of three values, -1, 0, or +1. Information theory puts the cost of a ternary value at log2(3), about 1.585 bits. Bonsai 2 groups weights into blocks of 128 and stores one shared FP16 scale factor per group, which amortizes the overhead of that scale across many weights. The math works out to roughly 1.71 bits per weight for the ternary-coded tensors, and 1.72 bits per weight once a small set of tensors (about 0.0976% of the language model, covering things like normalization weights and the recurrent state path in the linear-attention layers) are counted at full precision.
That 1.72-bit figure is a theoretical floor. In practice, the format a model ships in adds overhead. The GGUF PTQ1_0 packing, which stores trits densely, lands close to that floor at 1.75 bits per weight. GGUF’s PQ2_0 packing, which stores each trit in a 2-bit slot for cheaper unpacking, comes in at 2.13 bits. The MLX build used on Apple Silicon stores both a scale and a bias per group (a quirk of MLX’s grouped low-bit container), pushing it to 2.25 bits per weight, translating to a 7.67GB language model.
Why does a ternary weight need a rotation trick?
One detail in Bonsai 2’s design is a “blockwise Hadamard rotation” applied to the weights before they’re reduced to ternary values. Each weight matrix is transformed in blocks of 1024 using a fixed orthogonal rotation, and the same transform is applied to activations at inference time. Because the rotation is folded into the stored weights ahead of time, it costs nothing in extra bits or extra computation on the weight side, it just requires the runtime to also apply the matching transform to activations, or the model produces garbage.
The purpose of this rotation is to spread out the values within a weight matrix so they’re better suited to being squashed into just three levels. Extreme quantization schemes tend to break down when a matrix has a few outlier values that dominate a group’s scale factor. Rotating the basis first tends to even out the distribution, which is part of why Bonsai 2 avoids the sharp accuracy cliff that typically shows up in sub-4-bit models.
How much of the original model’s intelligence does it keep?
According to the model’s published benchmark results, Bonsai 2 scores an 84.78 average across 14 thinking-mode benchmarks, which the model card states is 98.2% of the FP16 baseline’s performance. For comparison, a conventional IQ2_XXS quantization of the same base model scores 72.59 on the same suite, despite taking up more space. Bonsai 2 also lands within 0.4 points of a UD-Q4_K_XL build that is roughly three times its size.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
Breaking that down by category, math performance sits at 96.57 (within half a point of full precision), coding performance is roughly level with the unquantized baseline at 89.42, and agentic tool-calling performance comes in at 74.92. These are the categories that typically suffer most under aggressive quantization, since multi-step reasoning and tool use tend to be sensitive to small errors compounding over a long generation. The fact that they hold up is the main claim worth scrutinizing if you’re evaluating whether this approach generalizes to other base models.
What hardware do you need to run it?
The core appeal is that it runs on hardware that couldn’t hold the original model at all. The FP16 version of the 27B base model is roughly 54GB, which doesn’t fit in memory on most consumer laptops or even many workstation GPUs. Bonsai 2’s language model alone is 7.67GB in the MLX build, and the full package with the vision tower included is 8.60GB. The GGUF PTQ1_0 packing brings the language model down to 5.95GB.
On an Apple M5 Max laptop, Prism ML reports about 47 tokens per second for token generation and 765 tokens per second for prompt processing, running on the llama.cpp Metal backend. On an M5 Pro, measured decode speed streams about 204 GB/s of weight data, which points to the model being memory-bandwidth bound rather than compute bound, a property the ternary format is specifically designed to exploit since less data has to move per token.
On discrete GPUs, an RTX 5090 hits about 130 tokens per second with the PQ2_0 packing, while older Ada-generation cards like the RTX 4090 and L40S actually do better with the denser PTQ1_0 packing, because it moves 17% less data per step even though it costs more arithmetic to unpack. Which packing wins depends on whether the specific GPU generation is limited by memory bandwidth or by instruction throughput, so there’s no single “best” format across all hardware.
Is Bonsai 2 27B worth using over a standard quantized model?
For anyone trying to run a 27B-class model locally without a multi-GPU workstation, it’s a meaningful option. The benchmark gap between Bonsai 2 and a conventional 2-bit quantization of the same base model (84.78 versus 72.59) is large enough to matter for tasks involving reasoning, math, or tool calling, and it comes at a smaller footprint. The tradeoff is that it requires non-standard tooling. The MLX version needs a custom loader because ordinary MLX loaders skip the activation transform required by the Hadamard rotation, and will return incorrect output without any error message. The GGUF version similarly requires a forked build of llama.cpp, since stock llama.cpp doesn’t support the packing.
That’s a real cost. Anyone deploying this needs to pin the exact runtime version documented by Prism ML rather than assuming compatibility with whatever mainline build they already have installed. For experimentation or local development this is a minor hurdle. For production systems it means an added maintenance dependency on a fork that isn’t part of the standard llama.cpp or MLX release cycle.
Frequently Asked Questions
What does “ternary” mean in Bonsai 2 27B?
It means each weight in the model can only take one of three values: -1, 0, or +1. A shared scale factor, computed per group of 128 weights, converts these three symbolic values back into meaningful magnitudes during inference.
How big is Bonsai 2 27B compared to the original model?
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
The original Qwen3.8-27B model in FP16 is roughly 54GB. Bonsai 2’s language model alone is 7.67GB in the MLX build or as small as 5.95GB in the GGUF PTQ1_0 format, and the full package with vision support included is 8.60GB.
Does Bonsai 2 27B lose much accuracy compared to the full-precision model?
The model card reports 98.2% retention of FP16 performance across 14 benchmarks, with math and coding scores close to the unquantized baseline. This is notably better than typical 2-bit quantizations of the same base model, which the card reports at roughly 86% retention.
Can I run Bonsai 2 27B with standard llama.cpp or MLX?
No. The model requires a custom loader for MLX and a forked build of llama.cpp for CUDA, CPU, and Metal, because standard loaders don’t apply the activation transform tied to the model’s weight rotation. Using a standard loader will silently produce incorrect output rather than an error.
What license is Bonsai 2 27B released under?
It’s released under the Apache 2.0 license, based on Qwen3.8-27B.
