Needle 3: Running a Tiny On-Device Tool-Calling AI Model
Needle 3 packs tool calling, extraction and embeddings into an 8-29MB file. Here's how the model works and how to deploy it on-device.
What is Needle 3?
Needle 3 is a small foundation model built by Cactus Compute for running AI directly on phones, wearables, robots, smart home hubs, automotive systems and microcontrollers. The entire model ships as a single file between 8 and 29 MB, depending on how many layers you keep, and it handles three jobs: picking and filling tool calls, pulling structured fields out of messy text, and generating embeddings for local search and routing. It doesn’t try to be a general chatbot. It trades that away to beat models 10 times its size on mobile tool calling and match models 2 to 3 times larger on extraction tasks.
TL;DR
- Needle 3 is a single-file model (8-29 MB depending on layer depth) designed to run tool calling, structured extraction and text embeddings entirely on-device, with no server round trip.
- It sacrifices general chat ability on purpose, using that saved capacity to outperform models 10x its size on tool calls and match models 2-3x bigger on extraction, according to Cactus Compute’s published benchmarks.
- The architecture is a custom recipe called a Laddered Simple Attention Network, combining a Monarch Hadamard MLP, GQA attention with causal conv taps, an n-gram “engram” memory, and multi-lane hyper-connections, trained so every depth from 2 to 20 layers works as a standalone deployable model.
- Outputs are grammar-constrained, meaning a byte-level grammar compiled from your tool schemas forces every generated token to stay valid, so tool calls and extracted JSON are guaranteed to parse.
- Every response carries a calibrated confidence score, which developers can use to decide whether the app should act automatically, ask the user to confirm, or refuse the request outright.
- The model is fine-tunable and sliceable, so a team can train on their own tool set with LoRA, then export a subnetwork as small as 2 layers that still beats larger baseline models after tuning.
- Deployment is a one-line install (
pip install cactus-needle) plus per-platform engines under 1 MB each, with a CLI, a C API, browser and WASI support, and offline/air-gapped setup documented separately.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
How does Needle 3 actually work?
Needle 3 is not a scaled-down version of a large language model. It’s a purpose-built architecture called a Laddered Simple Attention Network. Instead of a standard feed-forward network, it uses a Monarch Hadamard MLP. Its attention mechanism is grouped-query attention (GQA) with causal convolution taps layered in. It also includes what Cactus calls an “engram,” an n-gram memory component that’s read by gather operations rather than computed fresh each time, and multi-lane hyper-connections that route information across layers.
The practical effect of the engram is important: most of the model’s parameters live there, which means the 121M-parameter version does the arithmetic work of a model closer to 50M parameters. That’s a meaningful compute saving on hardware where every megabyte and every cycle counts.
The “laddered” part of the name refers to training: Needle 3 is trained so that every depth from 2 layers up to 20 layers is a valid, deployable model on its own. A developer can slice out a 4-layer subnetwork for a smartwatch and a 20-layer version for a phone, from the same base model, without retraining from scratch.
On top of the architecture, the weights are compressed using Cactus’s own 2-bit quantization scheme (CQ2-bit, roughly 2.125 bits per weight). Output generation is constrained by a byte-level grammar compiled directly from the app’s tool schemas or data shapes, which is what guarantees a tool call or an extracted record always parses correctly instead of occasionally returning malformed JSON. A learned head also attaches a calibrated confidence score to every response.
What can you actually build with it?
Three use cases, all running locally with no network call:
Tool calling. Given a list of functions an app exposes, Needle 3 selects the right ones and fills in arguments based on what the user said. Ask it for two things in one sentence and it returns two calls in the correct order. Ask it something no available tool covers, and it returns an empty list rather than inventing a call or hallucinating an argument.
Structured extraction. Developers declare the shape of the data they want (an invoice, a booking, a form submission, a notification) and hand the model raw, messy text. It returns typed fields matching that shape. Because the grammar constrains the output, the result is guaranteed to be parseable JSON. The same mechanism extends naturally to classification tasks, where the “shape” is just an enum of category labels.
Embeddings. The same model can also return a vector for a sentence, which lets an app do local semantic search, matching, or routing without sending anything to a server or loading a separate embedding model.
How do you install and run it?
Getting started is a pip install:
pip install cactus-needleA minimal Python example from the model’s documentation shows the pattern: define a Python function, decorate it as a tool, and hand it to the Needle agent.
import needle
@needle.tool
def get_weather(city: str):
"Get the current weather for a city."
return {"city": city, "temp_c": 27, "sky": "clear"}
agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])Every turn returns one JSON object containing the function_calls, the model’s reasoning, and a calibrated confidence score. The engine and weights are downloaded once and cached locally after that.
For production deployment outside Python, each supported platform has its own engine binary under 1 MB that loads the .cact model file at startup. The CLI pattern looks like this:
./needle --model needle3.cact --tools tools.json --prompt "dim the living room to 30"
./needle --model needle3.cact --tools tools.json --serveOne practical detail worth knowing: tool schemas share context space with the system prompt and conversation history. If the tool descriptions are too verbose or the catalogue too large, the model’s static prefix won’t fit in context. Cactus’s guidance is to trim schema text, split large tool catalogues, or (if declaring more than five tools) let Needle retrieve only the relevant ones per turn instead of stuffing all of them into every prompt.
How does fine-tuning and model sizing work?
Needle 3 is explicitly designed to be customized rather than used only as shipped. Because of the ladder training approach, a subnetwork as small as 2 layers can be fine-tuned on a single product’s specific tool set and run efficiently on hardware far smaller than what the full 20-layer model needs.
Cactus’s published results on the DroidCall benchmark show fine-tuning lifts every subnetwork size by 18 to 36 points of accuracy. Starting at the 4-layer depth (around 29M parameters), the fine-tuned subnetwork surpasses DeepSeek V4 Flash on that benchmark, according to the model documentation.
The workflow uses LoRA fine-tuning on the frozen 20-layer base, then a build command merges the adapter and slices out whichever depth you need:
needle build --layers NThis exports a 4-bit .cact file that runs on the same lightweight engine as the base model, at any depth from 2 to 20 layers.
Is Needle 3 worth using instead of a bigger cloud model?
The trade-off is straightforward: Needle 3 gives up general conversational ability in exchange for speed, size, privacy and offline reliability on narrow, well-defined tasks. If an application’s AI needs are limited to calling a known set of functions, pulling structured data out of text, or doing local semantic search, running a 121M-parameter model at 2-bit quantization directly on-device removes network latency, server costs and the privacy exposure of sending user data to a cloud API. The benchmarks Cactus reports (exact-match accuracy for tool calling, field micro-F1 for extraction) claim it beats much larger general-purpose models specifically on these narrow tasks.
It is not a replacement for an open-ended chatbot or a reasoning-heavy assistant. Teams building general conversational products still need a larger model. But for the growing category of apps that just need an on-device agent to route a voice command to the right function call, or turn a scanned receipt into structured fields, a purpose-built tiny model avoids the overhead of running (or calling out to) something built for a much broader job.
Frequently Asked Questions
How big is the Needle 3 model file?
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Between 8 MB and 29 MB, depending on how many of the model’s 2 to 20 layers are included in the exported build. Smaller devices can use shallower slices; phones can use the full depth.
What is the “engram” in Needle 3’s architecture?
It’s an n-gram memory component read via gather operations rather than recomputed through standard attention. Most of the model’s parameters live in it, which lets the 121M-parameter model perform with roughly the compute cost of a 50M-parameter model.
Can Needle 3 run without an internet connection?
Yes. The engine and model weights are downloaded once and cached, and Cactus documents air-gapped and offline setup as a supported deployment path, including a WASI component and a C API for embedding it in other runtimes.
Does Needle 3 hallucinate tool calls it doesn’t have?
According to its documented behavior, no. If a user request doesn’t match any available tool, the model returns an empty list of function calls instead of guessing or inventing one.
How do I fine-tune Needle 3 on my own tools?
Cactus’s Python package supports LoRA fine-tuning on the frozen 20-layer base model, followed by a build step that merges the adapter and exports a specific layer depth (from 2 to 20) as a quantized .cact file ready to deploy.
