Ternary-Bonsai-2-27B: A 27B Model That Fits in 8.6GB
Bonsai 2 27B shrinks a 27B model to 8.6GB using ternary quantization, keeping 98.2% of FP16 benchmark performance intact.
What is Ternary-Bonsai-2-27B?
Ternary-Bonsai-2-27B is a 27-billion-parameter language model, built on Qwen3.8-27B, that stores nearly all its weights as one of three values: -1, 0, or +1. Prism ML packages the result at 8.60GB on disk (7.67GB for the language model plus 0.92GB for an unquantized vision tower), down from roughly 54GB in FP16. The model card reports 84.78 average across 14 thinking-mode benchmarks, which the authors calculate as 98.2% of the FP16 baseline’s performance.
TL;DR
- Ternary quantization restricts nearly every weight to {-1, 0, +1}, giving each weight about 1.585 bits of raw information versus 16 bits in FP16, with a shared scale factor applied per group of 128 weights.
- The packed model lands at 8.60GB total, combining a 7.67GB ternary language model with a 0.92GB FP16 vision tower carried over unmodified from Qwen3.8-27B.
- Benchmark retention is the headline claim: 84.78 average across 14 thinking-mode tests, described as 98.2% of FP16 performance and outperforming a same-class IQ2_XXS build (72.59) at a smaller footprint.
- A blockwise Hadamard rotation is folded into the stored weights before ternary assignment, and matching activation transforms happen at runtime, which the model card says helps ternary weights hold up in the sub-4-bit range where conventional quantization typically degrades.
- Storage format matters more than the “bit” label suggests: the MLX container’s scale-and-bias-per-group scheme pushes effective storage to 2.25 bits/weight, versus 1.75 bits/weight for the GGUF PTQ1_0 packing of the identical ternary values.
- Throughput trade-offs between the two GGUF packings (PTQ1_0 and PQ2_0) depend on hardware: PTQ1_0 wins on Ada-generation and lower-end GPUs where memory bandwidth is the bottleneck, while PQ2_0 wins on H100, A100, and Blackwell cards where decode is limited by compute and launch overhead instead.
- On an Apple M5 Max laptop, the model reportedly generates around 47 tokens per second, a case the model card frames not as a speed comparison but as evidence a 27B-class model can run interactively on consumer hardware that couldn’t hold the FP16 version at all.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How does ternary quantization work in this model?
Standard quantization techniques compress weights to formats like 4-bit or 2-bit integers, each still capable of representing many distinct values. Ternary quantization is more aggressive: every weight becomes one of exactly three values, -1, 0, or +1. Since log₂(3) is about 1.585, each ternary weight theoretically carries under 1.6 bits of information rather than the 4, 8, or 16 bits used by other formats.
Because a single scale factor can’t capture the full dynamic range of a 27B model’s weights on its own, Bonsai 2 uses “g128” grouping: one FP16 scale value is shared across every 128 weights in a group. That scale amortizes the cost of full precision across many parameters, so the practical storage rate comes out to about 1.71 bits per weight for the ternary portion, and 1.72 bits per weight once you count the small slice of components (roughly 0.0976% of the language model, covering recurrent-state paths and normalization weights) that stay in higher precision.
The model card also describes a blockwise Hadamard rotation applied to each weight matrix before ternary assignment. This is an orthogonal transform, applied in fixed 1024-element blocks with ±1 signs, that reshapes the weight distribution before it gets rounded down to three values. The transform is folded into the stored weights ahead of time, so it adds no runtime cost on the weight side, but it does require the runtime to apply a matching transform to activations at inference. That’s why the model needs custom loading code rather than working with a generic MLX or llama.cpp checkpoint loader.
Coverage matters too. The ternary representation applies to embeddings, attention projections, MLP projections, and the LM head, which the model card calls “no high-precision escape hatches.” The vision tower, at 0.92GB, is left in FP16 and untouched by the rotation, since it’s not part of the ternary language backbone.
How does the file size break down, and why do formats differ?
The 8.60GB figure people will see for Ternary-Bonsai-2-27B covers two separate components: a 7.67GB ternary language model and a 0.92GB FP16 vision tower. That’s the number for the MLX release specifically, and it’s not the only packing available.
Prism ML also ships a GGUF version for llama.cpp, in two variants:
- PTQ1_0: 5.95GB, storing ternary values densely at 1.75 bits/weight, close to the theoretical minimum.
- PQ2_0: 7.21GB, storing each ternary value in a 2-bit slot (2.13 bits/weight), which trades some size for cheaper unpacking during inference.
The MLX format used in the main repo comes out largest of the three at 2.25 bits/weight for the language model (7.67GB), because MLX’s grouped low-bit container stores both a scale and a bias value per group of 128 weights, where the ternary format only strictly needs one. The model card is explicit that this doesn’t change the underlying values: the packed weights decode to the same three levels as the GGUF versions, verified by comparing group scales directly. It’s a container overhead, not a different quantization approach.
The model card frames this transparency as a contrast with other low-bit releases, citing an unnamed “2-bit” build of Qwen3.8-27B that actually averages 2.8 bits/weight at 9.4GB, arguing that Bonsai’s labeling matches its real bit-width more closely.
How does it compare to FP16, Q4_K_XL, and IQ2_XXS on benchmarks?
The model card’s central performance claim is a 14-benchmark average of 84.78 in thinking mode, evaluated with EvalScope and vLLM on NVIDIA H100 hardware under matched decoding and scoring conditions. Framed against other quantization tiers:
- Against the FP16 baseline (~54GB), Bonsai 2 retains what the card calls 98.2% of the average score, a gap the authors attribute mostly to genuine precision loss at ternary bit-widths rather than to the Hadamard rotation or grouping scheme.
- Against a comparably-sized IQ2_XXS build, Bonsai 2 scores 84.78 versus 72.59, a sizeable margin, while occupying less than two-thirds the disk footprint.
- Against UD-Q4_K_XL, which runs about three times the size of Bonsai 2’s language model, the gap narrows to under half a point, suggesting the ternary approach captures most of the accuracy typically associated with 4-bit quantization while using a fraction of the storage.
Category-level numbers reinforce the overall average: math benchmarks land at 96.57 (within half a point of full precision, per the card), coding hits 89.42 (described as level with the FP16 baseline), and agentic tool-calling scores 74.92. The general framing is that ternary weights, when built with grouping and a rotation step, don’t collapse the way some sub-4-bit formats do on reasoning-heavy tasks.
Is Ternary-Bonsai-2-27B practical to run locally?
For anyone without a datacenter GPU, the practical question is whether an 8.6GB, 27B-class model actually runs at usable speed on normal hardware. The throughput table in the model card, measured with llama.cpp on the GGUF packs at batch size 1, gives a range of answers depending on platform.
On consumer and prosumer GPUs, token generation (128-token decode) ranges from about 74 to 130 tokens/second depending on card and packing choice. An RTX 5090 hits nearly 130 tok/s with PQ2_0; an RTX 4090 does better with PTQ1_0 at roughly 91 tok/s. Server-class GPUs like the H100 and A100 show a similar split, generally favoring PQ2_0 because batch-1 decode there is limited by compute and launch overhead rather than memory bandwidth.
On Apple Silicon, the picture is different but arguably more notable: a 27B FP16 model (~54GB) simply doesn’t fit on a laptop. The ternary version does, and the card reports around 47 tokens/second on an Apple M5 Max, 28.7 on an M5 Pro, and 18.0 on an M4 Pro, with the M5 Pro figure additionally confirmed at 28.1 tok/s in a separate cross-platform table. Power draw on the M5 Pro is measured at 27.5W on the GPU rail during decode, far below the 300 to 455 watts of board power drawn by the NVIDIA cards in the same table, though the two platforms’ power instrumentation doesn’t measure identical components, so a direct efficiency comparison isn’t offered.
The model requires custom runtime code rather than stock MLX or llama.cpp loaders, since ordinary loaders won’t apply the Hadamard activation transform the format depends on. Prism ML points to a separate demo repository as the maintained, tested setup path for each backend.
One coffee. One working app.
You bring the idea. Remy manages the project.
Frequently Asked Questions
What does “ternary” mean in the context of LLM quantization?
It means each weight in the model is restricted to one of three values: -1, 0, or +1, rather than the wider range of values used in formats like 4-bit or 8-bit integer quantization. A shared scale factor, applied per group of weights, recovers the actual magnitude needed for computation.
How much smaller is Ternary-Bonsai-2-27B than the original model?
The packed model is 8.60GB total (language model plus vision tower), compared to roughly 54GB for the FP16 version of the same Qwen3.8-27B based architecture, an approximate 6.3x reduction in the shipped package, or closer to 9x when comparing the language model alone against its FP16 equivalent.
Does ternary quantization hurt model quality?
According to the model card’s own benchmarks, yes, but modestly: it reports retaining 98.2% of FP16 average performance across 14 thinking-mode benchmarks, with math and coding scores close to the full-precision baseline and a somewhat lower score on agentic tool-calling tasks.
What’s the difference between the PTQ1_0 and PQ2_0 GGUF packings?
Both store the same ternary weight values, but PTQ1_0 packs them densely near the theoretical bit-rate minimum (1.75 bits/weight, 5.95GB), while PQ2_0 stores each value in a fixed 2-bit slot (2.13 bits/weight, 7.21GB) for cheaper unpacking. Neither wins across all hardware. PTQ1_0 tends to perform better on memory-bandwidth-limited GPUs, PQ2_0 on compute-bound ones.
Can this model run on a laptop?
The model card reports it running on Apple Silicon laptops via MLX and llama.cpp’s Metal backend, with roughly 47 tokens/second on an Apple M5 Max and lower speeds on M5 Pro and M4 Pro chips. The FP16 version of the same base model would not fit in memory on typical laptop hardware at all.
