Run Bonsai 2 27B Locally on a Mac: Ternary Quantization Explained
Bonsai 2 27B compresses a 27B reasoning model to 8.6GB with ternary weights, hitting ~47 tok/s on an M5 Max MacBook via MLX.
What is Bonsai 2 27B and why does it matter for local AI?
Bonsai 2 27B is a ternary-quantized version of Qwen3.8-27B, a 27-billion-parameter hybrid-attention reasoning model, compressed down to 8.60GB on disk while retaining 98.2% of the intelligence of the full FP16 model. That combination is unusual. Most low-bit quantization schemes trade a lot of capability for size. Bonsai 2 uses full ternary weights (values restricted to -1, 0, +1) across the entire language backbone, paired with a rotation trick that keeps accuracy close to the original, letting a 27B-class model run interactively on a laptop instead of a data center GPU.
TL;DR
- Bonsai 2 27B is a ternary-weight quantization of Qwen3.8-27B that shrinks a ~54GB FP16 model to 7.67GB for the language backbone (8.60GB with the vision tower included).
- The model scores 84.78 average across 14 thinking-mode benchmarks, versus 72.59 for a conventional IQ2_XXS build at a larger footprint, and comes within 0.4 points of a 4-bit build three times its size.
- Ternary weights carry an idealized 1.72 bits per weight, but the MLX container used on Apple Silicon stores a scale and bias per group, pushing the practical rate to 2.25 bits/weight (7.67GB for the language model).
- On an Apple M5 Max MacBook, the model (measured on an earlier build) generates around 47 tokens per second with 765 tok/s prompt processing, at a 7.2GB footprint.
- Running it correctly on MLX requires a custom loader, not a stock MLX or llama.cpp build, because the model uses a Hadamard rotation baked into the weights that ordinary loaders don’t know how to undo.
- A separate GGUF release exists for llama.cpp users on CUDA, Metal, or CPU, packed in two formats (PTQ1_0 at 5.95GB, PQ2_0 at 7.21GB) that trade off differently depending on hardware.
- The base architecture keeps a 262K-token context window and roughly 75% linear attention, which is what makes long-context inference practical on consumer hardware in the first place.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How does ternary quantization actually work?
Standard quantization schemes squeeze weights into 4-bit or 8-bit integers, each capable of representing 16 or 256 distinct values. Ternary quantization goes further: each weight can only be -1, 0, or +1. That’s just log₂3, about 1.585 bits of information per weight, versus 16 bits for FP16. A single FP16 scale factor is shared across each group of 128 weights, so the group as a whole (not each weight individually) carries the magnitude information.
The math works out to roughly 1.71 to 1.72 bits per weight for the whole model once a handful of tensors that stay in higher precision (about 0.1% of the language model’s parameters, covering things like normalization weights and the recurrent state path in the linear-attention layers) are factored in. That’s close to a 9.3x reduction from FP16.
Bonsai 2 also applies something called a blockwise Hadamard rotation to the weights before ternarizing them. Rotating the weight matrices into a different basis first, then applying the matching inverse transform to activations at inference time, spreads out the information in a way that ternary rounding handles better. The rotation is folded into the stored weights offline, so it adds no extra size or runtime cost, but it does mean a generic MLX or llama.cpp loader that doesn’t know about the rotation will produce wrong output rather than throwing an error. This is why the model requires a dedicated loader.
Why is the Mac version 8.6GB instead of 5.8GB?
The idealized ternary rate for the language model alone is about 5.8GB (1.72 bits/weight). The actual MLX file for Bonsai 2 27B lands at 8.60GB, and there are two separate reasons for the gap.
First, MLX’s grouped low-bit format stores both a scale and a bias value per group of 128 weights, where the ternary representation strictly only needs one (the scale). The bias adds no new information, it’s just how the container is structured, but it does cost extra bits: 2.25 bits/weight in MLX’s packing versus 2.13 for the equivalent GGUF PQ2_0 packing and 1.75 for the denser PTQ1_0 packing. That accounts for the 7.67GB language-model figure on Mac versus 5.95 to 7.21GB in the leaner GGUF formats.
Second, the 8.60GB total includes a 0.92GB vision tower, the original Qwen3.8-27B vision component, carried in full FP16 and left unquantized. That tower isn’t touched by the Hadamard rotation either since the rotation only applies to the language model’s projections.
The practical takeaway: on Mac via MLX, budget for roughly 8.6GB of unified memory just to load the model, before accounting for context and generation overhead.
How fast does it run on Apple Silicon?
The most concrete number available is from an Apple M5 Max MacBook, where the model (a 7.2GB footprint, measured on an earlier pre-rotation build via llama.cpp’s Metal backend) generated tokens at about 47 tokens per second, with prompt processing around 765 tokens per second over a 512-token input. That prompt-processing speed matters if you’re feeding in long documents or big context windows before generation starts.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
A separate measurement on an Apple M5 Pro (a step down from the Max) showed 28.1 tokens per second for generation and 387 tokens per second for prompt processing, with decode measured at a low 27.5 watts on the GPU rail, 34.1 watts across CPU and GPU combined. For comparison, the same weights on desktop NVIDIA cards (RTX 5090, H100, RTX PRO 6000) all draw between 300 and 455 watts of board power to hit generation speeds in the 74 to 130 tokens per second range. The Mac numbers are slower in absolute terms, but the relevant comparison isn’t speed, it’s that the FP16 version of this model (roughly 54GB) simply doesn’t fit in a laptop’s memory at all. Ternary quantization is what makes a 27B model a laptop-class option in the first place.
What do you need to run it, and is it worth it?
To run the MLX version, you download the model from Hugging Face (prism-ml/Ternary-Bonsai-2-27B-mlx-2bit), install the runtime requirements bundled with the download, and use the provided loader rather than a stock mlx-vlm pipeline, since the model declares a custom model_type (prism_hadamard_qwen35) that ordinary loaders don’t recognize. Sampling parameters matter too: mlx-vlm and mlx-lm don’t automatically read the model’s recommended generation settings, so without explicitly passing temperature=1.0, top_p=0.95, top_k=20 for thinking mode, generation defaults to greedy decoding, which will hurt output quality.
Whether it’s worth it depends on what you’re optimizing for. If you need a 27B-class reasoning model that fits comfortably in a MacBook’s memory alongside other apps, and you’re willing to accept generation speeds in the tens of tokens per second rather than the hundreds you’d get on a high-end GPU, this is a genuinely practical option. The benchmark numbers suggest coding and math capability stay close to the full-precision baseline, which is the harder problem low-bit quantization usually fails at. If you have access to an RTX 4090 or better, the GGUF release will run faster and the tradeoffs between the PTQ1_0 and PQ2_0 packings become relevant to squeeze out more throughput.
Frequently Asked Questions
What base model is Bonsai 2 27B built on?
It’s derived from Qwen3.8-27B, a 27.36-billion-parameter hybrid-attention causal language model with roughly 75% linear attention and 25% full attention, plus a vision tower. The architecture itself is unchanged; only the weight representation differs.
Can I run Bonsai 2 27B on Windows or Linux with an NVIDIA GPU?
Yes, but not through the MLX package. NVIDIA users should use the GGUF release (Ternary-Bonsai-2-27B-gguf) through a llama.cpp build that supports the custom low-bit kernels, since a stock llama.cpp build cannot run the packed ternary weights correctly.
Why does the model need a custom loader instead of standard MLX or llama.cpp?
The weights are stored in a rotated basis using a Hadamard transform, and the runtime must apply a matching inverse transform to activations during inference. A generic loader that skips this step will load the file without erroring but will produce incorrect output, since it silently ignores the transform.
How does Bonsai 2 27B compare to a standard 2-bit quantization of the same base model?
The model card reports 84.78 average across 14 thinking-mode benchmarks for Bonsai 2, compared to 72.59 for a conventional IQ2_XXS quantization of the same base model, despite Bonsai 2 using less than two-thirds of that build’s footprint.
Does the vision tower also get quantized?
No. The 0.92GB vision tower is carried over unrotated and unquantized in FP16 from the original Qwen3.8-27B release. Only the language model’s embeddings, attention projections, MLP projections, and LM head go through the ternary quantization and rotation process.
