How to Run Bonsai 2 27B Locally: Full Install Guide
Install and serve Bonsai 2 27B with llama.cpp locally. Covers VRAM usage, slot configuration, GGUF formats, and real-world test results.
What is Bonsai 2 27B?
Bonsai 2 27B is a ternary-quantized version of Qwen3.8-27B, a 27 billion parameter model that normally needs around 54 GB of memory at FP16 precision. Bonsai 2 compresses that down to roughly 6 to 8 GB depending on format, while its maker, Prism ML, claims it retains about 98% of the original model’s benchmark performance. It ships under an Apache 2.0 license, supports a 262,000 token context window, and requires a custom llama.cpp fork (or MLX fork for Apple Silicon) since the ternary kernels aren’t in mainline llama.cpp.
TL;DR
- Bonsai 2 27B shrinks Qwen3.8-27B from ~54 GB down to under 8 GB using ternary weights (values of -1, 0, or +1) instead of standard 16-bit precision.
- Running it requires a custom llama.cpp fork from Prism ML’s model card, since stock llama.cpp cannot decode the ternary kernels or apply the required Hadamard rotation.
- Real-world VRAM usage lands around 12 GB with default settings (four parallel slots) or under 8 GB if you drop to a single slot (
--parallel 1), since the raw model is about 7 to 8 GB and the rest is KV cache overhead. - Two GGUF packings exist: PTQ1_0 (5.95 GB, dense trit packing, close to the theoretical 1.72 bits/weight) and PQ2_0 (7.21 GB, 2-bit slots, faster to unpack on some hardware).
- On Apple Silicon, the MLX build reaches roughly 47 tokens/second on an M5 Max laptop, and the model fits on machines that couldn’t otherwise load a 54 GB FP16 model at all.
- Independent testing found mixed real-world results: the model handled a vision and multilingual complaint-letter test well but missed a planted bug in a full-stack application and failed a basic HTML/JS interactivity test (a broken reset button) even after several fix attempts.
- The model uses a hybrid attention architecture (about 75% linear attention, 25% full attention), which is what lets it support the long 262K context window without the usual computational blowup.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
How does Bonsai 2 achieve such a small size?
The compression rests on three techniques stacked together.
First, every weight in the model gets reduced to one of three values: -1, 0, or +1. This is the “ternary” part of the name, and it drops the effective storage cost from 16 bits per weight down to somewhere between 1.72 and 2.25 bits per weight, depending on the packing format. According to Prism ML’s model card, the idealized ternary representation costs about 1.72 bits per weight, the GGUF PTQ1_0 packing lands at 1.75, GGUF PQ2_0 at 2.13, and the MLX 2-bit container (which stores an extra bias value per group) at 2.25.
Second, before the ternary conversion happens, the weights go through a Hadamard rotation, essentially a mathematical reshuffling that spreads information more evenly across the matrix so no individual weight is doing too much work. This rotation is folded into the stored weights permanently and costs zero extra storage. At inference time, the same transform gets applied to the live activations flowing through the model, so everything stays mathematically consistent. This step is why an ordinary MLX or llama.cpp loader will produce garbage output on this model: it skips the activation transform and the inverse embedding lookup, according to the model’s documentation.
Third, the underlying Qwen3.8 architecture already uses hybrid attention, mixing a cheaper linear attention method for most layers with full attention for a smaller subset. That keeps the 262K token context window computationally viable even on modest hardware.
Together, Prism ML’s documentation puts the idealized compression at about 9.3x smaller than FP16, with 26.2 million parameters (roughly 0.1% of the language model, covering things like normalization weights and the linear-attention recurrent state) left in higher precision to preserve stability.
What do you need to install it?
You need three things: a compatible runtime, the model weights, and enough VRAM or unified memory to hold them.
For CUDA or CPU, that means building Prism ML’s llama.cpp fork rather than using the stock version, since the ternary kernels and Hadamard rotation logic aren’t in mainline llama.cpp. The fork is linked from the model’s Hugging Face card. For Apple Silicon, there’s a corresponding MLX fork (Python and Swift) that handles the same rotation logic natively.
The weights themselves come in a few different packages:
- GGUF PTQ1_0 (5.95 GB): dense ternary packing, closest to the theoretical minimum size.
- GGUF PQ2_0 (7.21 GB): 2-bit slot packing, which trades some size for cheaper unpacking arithmetic.
- MLX 2-bit (7.67 GB language model, 8.60 GB total with vision tower): Apple’s grouped quantization format, which stores a scale and bias per group.
Once you’ve picked a format, you download it (via hf download or similar) and point your fork of llama.cpp or MLX at the file. A companion vision tower file (mmproj) is available separately if you want multimodal support alongside the text model.
How much VRAM does it actually use?
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
In hands-on testing using llama.cpp’s server with default settings, Bonsai 2 27B consumed just over 12 GB of VRAM. That’s higher than the raw model size (around 7 to 8 GB) because llama.cpp reserves KV cache space for four parallel request slots by default.
Dropping the parallel slot count to one (using a flag like --parallel 1 or -np 1) cuts memory use to under 8 GB. Choosing a smaller quantization format, like the 5.95 GB PTQ1_0 GGUF instead of the larger PQ2_0, reduces the footprint further.
On the vendor’s own benchmark table, throughput varies by hardware and packing format. An RTX 5090 hits close to 130 tokens/second on PQ2_0 for short-sequence decoding, while an RTX 4090 does better with PTQ1_0 than PQ2_0, since the two formats trade off differently depending on whether a GPU is bandwidth-bound or compute-bound. On Apple Silicon, an M5 Max laptop reportedly reaches about 47 tokens/second, while an M4 Pro is closer to 18 tokens/second, with prompt processing becoming the practical bottleneck on very long inputs.
Is Bonsai 2 27B actually reliable for real work?
This is where the picture gets more mixed than the compression numbers suggest.
In one hands-on test, Bonsai 2 was given a Docker-based full-stack application (FastAPI, Postgres, a live dashboard) with a deliberately planted timezone bug: the backend computed a 24-hour activity cutoff using local server time instead of UTC, silently dropping the most recent three hours of data. Asked to find and fix “the bug” with no other hints, the model’s reasoning trace actually walked right up to the correct issue, did the timezone math, and then talked itself out of it, concluding the code was fine. It then went and fixed a different, unplanted discrepancy in a 14-day trend chart instead. The real bug stayed broken.
On a code generation task (building a single-file HTML coffee-making simulation), the model produced a good-looking interface but shipped a broken “start sequence” button that didn’t trigger anything. Across several rounds of “this doesn’t work, please fix it,” the model reported success each time, but the interactivity remained broken in testing.
The model performed better on a vision and creative-writing test, correctly describing an image and producing a coherent, stylistically appropriate mock complaint letter. It struggled badly on a multilingual translation test, getting stuck in repetitive, broken output on certain languages including Tamil.
Prism ML’s own published benchmark table (run internally with EvalScope and vLLM) claims an average score of 84.78 across 14 categories in thinking mode, close to the FP16 baseline and ahead of a comparison “2-bit” build the company says is really closer to 2.8 bits per weight despite its label. Those are the vendor’s own numbers, not independently verified, and real-world testing suggests results vary noticeably by task type.
Frequently Asked Questions
Does Bonsai 2 27B work with regular llama.cpp?
No. It requires Prism ML’s llama.cpp fork, linked from the model’s Hugging Face page, because stock llama.cpp doesn’t implement the ternary kernels or the Hadamard rotation the model depends on for correct output.
How much disk space and memory do I need?
The GGUF files range from 5.95 GB (PTQ1_0) to 7.21 GB (PQ2_0) on disk, and the MLX build is 8.60 GB including the vision tower. Expect roughly 8 to 12 GB of VRAM depending on how many parallel request slots you configure.
What’s the difference between PTQ1_0 and PQ2_0?
PTQ1_0 packs ternary values densely and is smaller (5.95 GB), while PQ2_0 stores each value in a 2-bit slot (7.21 GB) for cheaper unpacking. Neither is universally faster: PTQ1_0 tends to win on memory-bandwidth-limited GPUs like the RTX 4090, while PQ2_0 wins on compute-bound cards like the H100.
Can it run on a laptop?
Yes. Prism ML reports around 47 tokens per second on an Apple M5 Max laptop using the MLX build, and the whole point of the compression is that a 27B-class model fits in memory where the 54 GB FP16 original could not.
Is the 98% performance claim trustworthy?
It comes from Prism ML’s own internal benchmark suite, not third-party verification. Independent hands-on testing found the model good at vision and creative tasks but inconsistent at bug-fixing and basic UI code generation, so results depend heavily on the specific task.
