Needle 3 Benchmarks: A Tiny Model Beating 10x Larger Rivals
Needle 3 packs tool calling and extraction into a sub-30MB file. Here's how its benchmarks stack up against models 10x its size.
What is Needle 3 and why does its benchmark matter?
Needle 3 is an open-weight foundation model built by Cactus Compute specifically for tool calling, structured extraction, and text embedding on constrained hardware: phones, wearables, robots, smart home devices, and microcontrollers. The whole model ships as a single file between 8 and 29 MB. What makes it notable isn’t raw scale but efficiency: Cactus reports that Needle beats models 10 times its size on mobile tool calls and matches models 2 to 3 times larger on extraction tasks, trading away general chatbot ability to specialize hard on structured, agentic tasks.
That tradeoff is the entire thesis behind the release. Most language models chase breadth: chat, code, reasoning, creative writing. Needle 3 chases one narrow job extremely well, and the benchmark numbers are the evidence Cactus points to for that bet paying off.
TL;DR
- Needle 3 is a sub-30MB on-device model built only for tool calling, structured extraction, and embeddings, not general chat.
- On exact-match tool calling accuracy, Cactus reports it beats models roughly 10 times its parameter count.
- On field micro-F1 extraction benchmarks, it matches models 2 to 3 times larger.
- The architecture uses a ladder design, meaning every depth from 2 to 20 layers is independently deployable as a smaller model.
- Fine-tuning on DroidCall lifts every subnetwork size by 18 to 36 points, and from 4 layers up (starting at 29M parameters), the tuned subnetwork surpasses DeepSeek V4 Flash.
- Most of the model’s 121M parameters live in an engram memory component, letting it do the computational work of a much smaller 50M-parameter model.
- Weights are compressed to CQ2-bit using Cactus’s own quantization scheme, and outputs are constrained by a byte-level grammar compiled from the app’s tool schemas.
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
How does Needle 3’s architecture achieve small-model accuracy?
Needle 3 is built on what Cactus calls a Laddered Simple Attention Network, a custom small-model recipe rather than a scaled-down version of a standard transformer. Several pieces work together:
- A Monarch Hadamard MLP replaces the standard feed-forward network, a structured matrix approach that cuts compute without gutting capacity.
- Grouped-query attention (GQA) with causal convolution taps handles the sequence modeling more cheaply than full attention.
- An engram acts as an n-gram memory bank, read via gather operations, and it’s where most of the model’s parameters actually live. Cactus states this lets the 121M model perform the arithmetic of a 50M model, since the engram offloads pattern memorization that would otherwise require larger dense layers.
- Multi-lane hyper-connections replace simple residual connections, giving the network more paths to route information across layers.
The “laddered” part is the key deployment trick: the model is trained so that every depth from 2 to 20 layers works as a standalone, deployable model. That means a developer isn’t stuck with one fixed size. They can slice out a 2-layer version for a microcontroller or keep the full 20-layer version for a phone, all from the same trained weights.
On top of the architecture, Needle 3 uses a decode grammar, a byte-level constraint compiled directly from a developer’s tool or schema definitions, so every generated token is forced to produce parseable output. There’s no risk of a malformed JSON object or a hallucinated function name outside the declared tool set. Each response also carries a calibrated confidence score from a dedicated learned head, which developers can use to decide whether to act on a result automatically, ask for confirmation, or refuse.
How was Needle 3 benchmarked against larger models?
Cactus evaluated Needle 3 on two categories of tasks, using standard, unambiguous metrics rather than open-ended judging:
- Tool calling: measured as exact-match accuracy across full test splits. This is a strict metric: the model either picks the correct function and fills every argument correctly, or it doesn’t. Ask Needle for two things in one request and it should return two calls in the correct order; ask it something no available tool covers and it should return an empty list rather than fabricate an answer.
- Structured extraction: measured as field micro-F1 across full test splits, meaning accuracy is scored per extracted field across the entire dataset rather than per full record.
Cactus published these results across six benchmarks comparing Needle 3 against baseline models of varying sizes, with the tool-calling comparisons including DroidCall, a benchmark focused on mobile and Android-style function invocation, and Mobile Actions. The company’s own claim is that on tool calling, Needle beats baselines roughly 10 times its parameter count, and on extraction it matches models 2 to 3 times its size.
What does fine-tuning do for Needle 3’s accuracy?
The base model is a general-purpose foundation for tool calling and extraction, but Cactus designed Needle 3 to be fine-tuned on a specific product’s actual tool set, and the reported gains from doing so are substantial. Fine-tuning on DroidCall lifts every subnetwork size (every layer depth from 2 to 20) by 18 to 36 accuracy points over the untuned base.
The more striking claim is about scale efficiency after tuning: starting from 4 layers, which corresponds to roughly 29M parameters, the fine-tuned subnetwork passes DeepSeek V4 Flash on the benchmarked tasks. That’s a large-scale model being outperformed by a subnetwork sliced from a sub-30MB base file, once that subnetwork has been narrowed to a specific app’s tools.
The practical workflow: Cactus’s Python package fine-tunes with LoRA on the frozen 20-layer base, then a build command (needle build --layers N) merges the LoRA adapter and slices out whichever depth a developer wants, from 2 to 20 layers, exporting a 4-bit .cact file that runs on the same lightweight engine used everywhere else. This means a developer building a smart-home app with a fixed set of 15 tools can fine-tune on their own tool schema, then ship a 2 or 4-layer subnetwork that’s smaller, faster, and more accurate at their specific job than the general-purpose base model would be.
Is Needle 3 worth it compared to using a larger cloud model?
The honest answer depends entirely on the use case. Needle 3 explicitly gives up general conversational ability to win at three narrow jobs: tool calling, structured extraction, and embeddings. If an application needs a chatbot that can discuss recipes, write essays, or reason about arbitrary topics, Needle 3 is the wrong tool. It’s not built for that and the model card says as much.
But for the specific case of an app that needs to map user speech or text to a fixed set of function calls, or pull structured fields out of messy text (invoices, bookings, notifications, forms), running that entirely on-device with no network round trip and no per-call inference cost changes the economics and the latency profile. A model that fits in under 30MB can run on a phone, a wearable, or even a microcontroller, which a multi-billion-parameter cloud model simply cannot do.
The benchmark claims (beating 10x larger models on tool calls, matching 2 to 3x larger models on extraction, and surpassing DeepSeek V4 Flash after fine-tuning) all come from Cactus’s own published results. They’re worth treating as a strong signal rather than independently audited fact, since third-party reproductions weren’t part of the source material here. For developers evaluating on-device agentic tooling, the practical test is running Needle 3 against the fine-tuning workflow on your own tool schema and comparing accuracy directly.
Frequently Asked Questions
What is Needle 3 used for?
Needle 3 is designed for three on-device tasks: tool calling (matching user requests to app functions and filling in arguments), structured extraction (pulling typed fields out of unstructured text), and text embedding (generating vectors for local search and matching). It is not designed for general-purpose chat.
How big is the Needle 3 model file?
The full model ships as a single file between 8 and 29 MB, depending on the layer depth and quantization used, small enough to run on phones, wearables, and microcontrollers.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
What does it mean that Needle 3 beats models 10x its size?
On exact-match tool-calling benchmarks, Cactus reports that Needle 3 outperforms baseline models with roughly 10 times its parameter count. This reflects its narrow specialization in tool calling rather than broader general-purpose capability.
What is DroidCall and why does it matter for this benchmark?
DroidCall is a benchmark focused on mobile and Android-style function-calling tasks. Cactus used it both to measure baseline tool-calling accuracy and to fine-tune subnetworks, reporting 18 to 36 point accuracy gains after tuning.
Can Needle 3 be made smaller or larger after training?
Yes. The architecture is “laddered,” meaning it’s trained so every depth from 2 to 20 layers works as an independently deployable model. Developers can fine-tune the full model, then slice out a smaller subnetwork sized for their target device.
