Run Bonsai 2 27B Locally on a Mac: Full Setup Guide
How to run Ternary-Bonsai-2-27B, an 8.6GB ternary-quantized 27B model, locally on Apple Silicon with near-FP16 quality and speed.
What is Bonsai 2 27B and why does it matter for local inference?
Bonsai 2 27B is a ternary-quantized version of Qwen3.8-27B, packed down to 8.60 GB on disk while keeping 98.2% of the original FP16 model’s benchmark performance. That combination is unusual: most quantization this aggressive costs far more accuracy. A 27B-class model that normally needs about 54 GB in FP16 now fits comfortably in the unified memory of a MacBook, and Prism ML’s own numbers put decode speed around 47 tokens per second on an Apple M5 Max.
TL;DR
- Bonsai 2 27B compresses a 27B parameter Qwen3.8 model into 8.60 GB total (7.67 GB language model plus a 0.92 GB unquantized vision tower), versus roughly 54 GB for the FP16 original.
- The model uses ternary weights ({-1, 0, +1}) with group-wise FP16 scaling, landing at an effective 1.72 bits per weight, and scores 84.78 average across 14 thinking-mode benchmarks, which the model card reports as 98.2% of FP16 quality.
- On Apple Silicon it runs through MLX, requiring a custom loader bundled with the model because the pack uses a Hadamard-rotated weight basis that ordinary MLX loaders will silently misread.
- Reported throughput scales with GPU width: about 47 tok/s on an M5 Max, 28-29 tok/s on an M5 Pro, and 18 tok/s on an M4 Pro, all using roughly 7.2 GB of resident memory for the language model.
- A GGUF companion release exists for llama.cpp on CUDA, Metal, or CPU, in two packings (PTQ1_0 at 5.95 GB and PQ2_0 at 7.21 GB), but it needs a patched llama.cpp build, not the stock one.
- The generation defaults matter: Bonsai 2 ships with thinking-mode sampling parameters (temperature 1.0, top_p 0.95, top_k 20) baked into its config, and it runs at
xhighreasoning effort unless you explicitly dial it down tomedium.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How does ternary quantization get a 27B model under 9GB?
Standard weights in a neural network are stored as 16-bit or 32-bit floating point numbers. Ternary quantization throws that out and restricts every weight to one of three values: -1, 0, or +1. Each of those carries about 1.585 bits of information (log base 2 of 3), and Bonsai 2 groups weights into blocks of 128, sharing one FP16 scale factor per group. Do the math and the effective cost lands at roughly 1.72 bits per weight once you average in the small number of tensors kept at higher precision (about 0.1% of the language model, covering things like the recurrent state path in the linear-attention layers and normalization weights).
That’s the theoretical floor. What you actually download is a bit heavier because of how different runtimes pack the format. The model card lays out three versions of the same ternary weights:
- PTQ1_0 (GGUF): 1.75 bits/weight, 5.95 GB, packs the trits densely and lands closest to the ideal.
- PQ2_0 (GGUF): 2.13 bits/weight, 7.21 GB, stores each trit in its own 2-bit slot for cheaper unpacking.
- MLX 2-bit (this release): 2.25 bits/weight, 7.67 GB, because MLX’s low-bit container stores both a scale and a bias per group instead of just a scale.
All three decode to the exact same ternary values. The difference is container overhead, not representation. For comparison, the model card points out that a widely-used “2-bit” quantization of the same base model is actually 2.8 bits/weight at 9.4 GB once you account for how loosely some quantized builds are labeled.
There’s a second technique layered on top: a blockwise Hadamard rotation. Each weight matrix gets rotated by a fixed orthogonal transform before ternary values are assigned, and the runtime applies the matching transform to activations at inference time. This is folded into the stored weights offline, so it doesn’t cost extra bits or extra memory bandwidth, but it does mean a generic loader that skips this step will produce wrong output instead of an error. That’s the practical reason Bonsai 2 needs its own runtime code rather than working with any MLX checkpoint loader.
What do you need to run it on a Mac?
You need Apple Silicon with enough unified memory to hold roughly 8.6 GB comfortably alongside macOS and whatever else is running, so 16 GB is the practical floor and 32 GB or more gives you headroom for long contexts. The setup, per the model’s Hugging Face repository (prism-ml/Ternary-Bonsai-2-27B-mlx-2bit), looks like this:
- Download the pack with the Hugging Face CLI:
hf download prism-ml/Ternary-Bonsai-2-27B-mlx-2bit --local-dir bonsai2-27b-mlx - Install the runtime requirements bundled in the download’s
runtime/folder. - Load the model through the provided
vision_artifact.load_vl_modelhelper rather than a stockmlx-vlmloader, since the pack declares a custommodel_type(prism_hadamard_qwen35) that off-the-shelf loaders don’t recognize. - Generate with
mlx_vlm.generate, passing images if you want vision input or omitting them for text-only use.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
No source builds or forked MLX packages are required for the Python path. The model card is explicit that this pack runs on stock MLX packages as long as you use the bundled loader code, and points to the project’s own “Bonsai-demo” repository as the maintained, tested reference for setup across backends.
Is Bonsai 2 27B worth running over a smaller dense model?
That depends on what you’re optimizing for. If your bottleneck is disk space and memory footprint, Bonsai 2 gives you 27B-class reasoning, coding, and agentic tool-calling behavior in a package smaller than many 13B or 14B models ship at full precision. The model card’s benchmark table claims math performance within half a point of FP16 (96.57), coding roughly at parity with the baseline (89.42), and agentic tool-calling at 74.92, all while the model occupies under 9 GB.
If your bottleneck is raw speed, the picture is more mixed. On a laptop GPU, 27-47 tokens per second (depending on chip) is usable for interactive chat but well behind what a heavily optimized small dense model can do on the same hardware. The model card frames the win correctly: the meaningful comparison isn’t a speedup ratio against FP16, since FP16 simply doesn’t fit on a laptop at all. The win is that a model of this capability class runs interactively on consumer hardware in the first place.
Worth noting: throughput numbers reported for Apple platforms in the model card’s “Additional Apple Platforms” table are flagged as measured on an earlier pre-rotation build, pending re-measurement on the current stack. Treat those figures as directionally accurate rather than final.
How does it compare to running the GGUF version instead?
The GGUF release (prism-ml/Ternary-Bonsai-2-27B-gguf) targets llama.cpp across CUDA, Metal, and CPU, and comes in the PTQ1_0 and PQ2_0 packings described above. On Apple hardware specifically, the throughput numbers in the model card’s cross-platform table were captured using llama.cpp’s Metal backend against the GGUF packs, not the MLX safetensors pack. Both routes need a patched runtime, not stock software: the MLX path needs the bundled loader, and the GGUF path needs Prism ML’s llama.cpp fork, since stock llama.cpp doesn’t understand the ternary hybrid-attention kernels either.
Neither packing format dominates the other across hardware. The model card’s throughput table shows PTQ1_0 winning on memory-bandwidth-constrained cards (Ada-generation GPUs, the L4) because it moves less data per step, while PQ2_0 wins on cards where batch-1 decode is limited by instruction throughput rather than memory (H100, A100, Blackwell-class parts). For a Mac user, this mostly matters if you’re deciding between the MLX pack and the GGUF pack: the MLX version is the more direct route on Apple Silicon and is what the model card’s Mac-specific quickstart instructions assume.
Frequently Asked Questions
How much RAM do I need to run Bonsai 2 27B on a Mac?
The language model occupies about 7.67 GB on disk, with the full pack including the vision tower at 8.60 GB. Apple Silicon with 16 GB of unified memory can technically load it, but 32 GB gives more comfortable headroom for context and other running applications.
Does Bonsai 2 27B lose much quality from quantization?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
According to the model’s own published benchmarks, it retains 98.2% of FP16 intelligence, averaging 84.78 across 14 thinking-mode benchmarks, compared to 72.59 for a conventional sub-4-bit quantization of the same base model.
Can I use a regular MLX model loader with Bonsai 2 27B?
No. The weights are stored in a Hadamard-rotated basis, and the pack declares a custom model_type. A standard MLX or mlx-vlm loader will load the file without error but produce incorrect output because it skips the required activation transform.
What’s the difference between the MLX pack and the GGUF pack?
Both encode the same ternary weight values. The MLX pack (7.67 GB language model) targets Apple MLX on Python or Swift. The GGUF pack offers two packings, PTQ1_0 (5.95 GB) and PQ2_0 (7.21 GB), for llama.cpp on CUDA, Metal, or CPU, but requires a patched llama.cpp build rather than the stock release.
What generation settings should I use?
The model card recommends thinking-mode sampling of temperature 1.0, top_p 0.95, top_k 20 for full reasoning behavior, or temperature 0.7, top_p 0.80, top_k 20 with a 1.5 presence penalty for shorter, non-thinking responses. The model defaults to xhigh reasoning effort; switching to medium trades some accuracy for faster, shorter responses.
