Needle 3: The 8-29MB Model Built for On-Device Tool Calling
Needle 3 is an 8-29MB foundation model for on-device tool calling and structured extraction. Here's how its architecture works and how to deploy it.
What is Needle 3?
Needle 3 is a foundation model from Cactus Compute built to run entirely on phones, wearables, robots, smart home hubs, automotive systems, and microcontrollers. The whole model ships as a single file between 8 and 29 MB, small enough to sit inside an app bundle rather than call out to a server. It gives up general chat ability on purpose, trading that capacity for three narrow jobs it does well on constrained hardware: picking and filling tool calls, extracting structured data from messy text, and producing text embeddings for local search and routing.
TL;DR
- Needle 3 is a specialist, not a chatbot: it trades general conversational ability for accuracy on tool calling, structured extraction, and embeddings, all running locally on the device.
- The file size ranges from 8 to 29 MB, small enough to bundle with a mobile app or flash to a microcontroller, and the runtime engine itself is under 1 MB per platform.
- The architecture uses a Laddered Simple Attention Network, a custom recipe combining a Monarch Hadamard MLP, GQA attention with causal convolution taps, and an n-gram memory called an engram that holds most of the model’s parameters.
- The model is designed to be sliced: every depth from 2 to 20 layers is a deployable model on its own, so a 2-layer subnetwork fine-tuned on one app’s tools can run on hardware far smaller than what the full 20-layer model needs.
- Weights are compressed to roughly 2 bits per parameter using a method called Cactus Quants, and every generated token is constrained by a grammar compiled from the app’s own function schemas, so outputs are guaranteed to parse.
- Fine-tuning on datasets like DroidCall lifts accuracy by 18 to 36 points across every subnetwork size, and from 4 layers up, a tuned Needle subnetwork reportedly outperforms DeepSeek V4 Flash on tool-calling accuracy, starting at just 29M parameters.
- The model is open-weight, distributed with a Python package, a C API, platform-specific engines, and guides for fine-tuning and deployment on GitHub and the Cactus Compute site.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
How does Needle 3 handle tool calling?
Needle 3 takes the functions an app exposes, matches them against what a user said, and fills in the arguments. If a request maps to two actions, it returns two calls in the right order. If nothing in the toolset matches, it returns an empty list instead of hallucinating a plausible-looking call. That refusal behavior matters more on-device than in a cloud chatbot, because a wrong tool call on a phone might actually dim the wrong light or send the wrong message, with no human review step in between.
Every response comes back as a single JSON object containing the function calls, a short reasoning trace, and a calibrated confidence score from a dedicated model head. Developers can use that confidence value to decide whether to execute an action automatically, ask the user to confirm, or refuse outright. This three-way routing (act, confirm, refuse) is one of the core patterns Cactus documents for building agentic features around the model.
How does structured extraction work?
The second job is turning unstructured text into typed data. A developer declares a shape (an invoice, a booking confirmation, a form, a notification) and hands the model raw text. Needle returns typed fields that match the declared schema. This works through the same decode-time grammar used for tool calling: a byte-level grammar compiled from the schema constrains every token the model generates, so the output is guaranteed to parse into valid JSON rather than occasionally breaking format like a general-purpose LLM might.
The same extraction mechanism generalizes to classification tasks, since a classification problem is really just extraction into an enum field. That means one small model covers form-filling, data extraction, and categorization without needing three separate specialist models.
What makes the architecture different from a standard small LLM?
Needle 3 doesn’t use a conventional transformer feed-forward block. Cactus calls the recipe a Laddered Simple Attention Network, and it swaps in several nonstandard pieces:
A Monarch Hadamard MLP replaces the usual feed-forward network, which is a structured, more parameter-efficient way to do the dense transformations most transformers rely on. Attention uses grouped-query attention (GQA) with causal convolution taps layered in, a combination meant to capture local sequence patterns cheaply. The most distinctive piece is what Cactus calls the “engram,” an n-gram memory component read by gather operations, and most of the model’s parameters actually live there rather than in traditional attention or MLP weights. According to Cactus, this lets the 121M-parameter version do the effective work of a 50M-parameter conventional model, because the engram stores pattern-level memory more efficiently than dense weight matrices do. Multi-lane hyper-connections round out the design, a routing mechanism between layers.
The “laddered” part refers to how the model is trained: every depth from 2 to 20 layers is trained to work as a standalone, deployable model. That’s what enables the model to be sliced down for tiny hardware without retraining from scratch.
How is the model compressed to fit on tiny hardware?
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
Weights are compressed using Cactus’s own quantization method, Cactus Quants, down to roughly 2.125 bits per weight in the shipped .cact file format. That’s well below the 4-bit or 8-bit quantization common in most local LLM deployments, and it’s a major reason the full 20-layer model fits in an 8 to 29 MB file. The .cact format is documented as a container the runtime engine maps and reads directly in place, avoiding a separate decompression step at load time. Each platform ships with an engine under 1 MB that loads the .cact weights at startup, so the total runtime footprint (engine plus weights) stays small enough for embedded and wearable targets, not just phones.
Is Needle 3 worth deploying instead of a cloud API or a bigger local model?
For narrow, well-defined tasks like tool routing and structured extraction, Needle 3’s pitch is that it beats models ten times its size on mobile tool-calling accuracy and matches models two to three times bigger on extraction, according to Cactus’s published benchmarks (exact-match accuracy for tool calling, field micro-F1 for extraction). The tradeoff is real: this is not a general chat model, and it will not hold an open-ended conversation or answer arbitrary knowledge questions well. It’s built for apps that need reliable function calling and data extraction without a network round trip, which matters for latency, offline use, privacy, and battery life.
Fine-tuning changes the calculus further. Cactus reports that fine-tuning on DroidCall, a tool-calling dataset, lifts accuracy by 18 to 36 points across every subnetwork size, and that from the 4-layer subnetwork upward (starting at 29M parameters) the tuned model passes DeepSeek V4 Flash on their tool-calling benchmark. If those numbers hold up under independent testing, it suggests a small, task-tuned Needle subnetwork can be a legitimate replacement for calling a much larger model over the network, at least for the specific tool schema it was tuned on.
How do you deploy and customize it?
The model installs via pip install cactus-needle, and a minimal example wraps a Python function as a tool with the @needle.tool decorator, then runs an agent against it locally. The GitHub repository holds the ready-to-run needle3.cact file (20 layers), the raw needle3.safetensors checkpoint for fine-tuning, and a separate engine build for each target platform.
Customization happens through LoRA fine-tuning on the frozen 20-layer base, followed by a needle build --layers N command that merges the adapter, slices out a subnetwork of the desired depth (from 2 to 20 layers), and exports a 4-bit .cact file sized for the target device. The same needle build --platform <folder> command fetches the right engine and header for a given platform and places the weights alongside them.
One practical constraint worth knowing: tool schemas share context space with the system prompt and conversation history. The needle_init call reports how many tokens the static prefix (tools plus system prompt) consumes and fails if it doesn’t fit. Cactus’s guidance for large tool catalogues is to keep descriptions concise, split big catalogues up, or declare more than five tools so the engine retrieves and keeps only the relevant ones in context per turn.
Frequently Asked Questions
How big is Needle 3 exactly?
The full model ships as a single file between 8 MB and 29 MB depending on configuration, with the runtime engine adding under 1 MB per platform.
What can Needle 3 not do well?
It’s not built for open-ended conversation or general knowledge questions. Its capacity is deliberately concentrated on tool calling, structured extraction, and embeddings rather than broad chat ability.
What is the “engram” in Needle’s architecture?
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
It’s an n-gram memory component read via gather operations, where most of the model’s parameters are stored. Cactus says this lets a 121M-parameter Needle model do work comparable to a 50M-parameter conventional model.
Can Needle 3 run on a microcontroller?
Yes. Cactus lists microcontrollers alongside phones, wearables, robots, and automotive systems as target hardware, and the model can be sliced down to as few as 2 layers for the smallest devices.
Is Needle 3 open source?
The weights and code are openly released on GitHub and Hugging Face, including the quantized .cact model, the safetensors checkpoint for fine-tuning, and platform engines, under Cactus Compute’s public repository.
