How to Run NeoHorse-1-4B Locally: Specs and Setup Basics
NeoHorse-1-4B ships as BF16 safetensors with a 262K native context, extensible to 1M. Here's what that means for local hardware.
What is NeoHorse-1-4B?
NeoHorse-1-4B is a 4-billion-parameter causal language model released by TokenRhythm, built on top of Qwen3.5-4B and post-trained for agentic tasks like tool use, coding, and instruction following. It’s a text-only model (no vision weights), distributed as BF16 safetensors, and licensed Apache 2.0. Running it locally mainly comes down to three things: matching your hardware to a 4B BF16 model, understanding its context length, and picking an inference stack that supports both.
TL;DR
- NeoHorse-1-4B is a 4B parameter model derived from Qwen3.5-4B, post-trained with a routing harness that TokenRhythm frames as an early step toward recursive self-improvement.
- The model ships in BF16 safetensors, which means roughly 8GB of raw weight data before accounting for KV cache and runtime overhead.
- Its native context window is 262,144 tokens, and the architecture supports extension up to 1,010,000 tokens with the right scaling configuration.
- On TokenRhythm’s own ten-benchmark comparison, NeoHorse-1-4B scores a 64.87 macro average versus 58.94 for base Qwen3.5-4B, a gain concentrated in agentic and tool-use benchmarks.
- It’s a text-only release: the vision components from any multimodal variant of the base model are stripped out, so this is strictly a language model.
- Because it’s a repackaged fine-tune, tensor values match the original training run even though config and tensor key names have changed, which matters if you’re debugging load errors against Qwen3.5-4B tooling.
What hardware do you need to run it locally?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
A 4B parameter model in BF16 (16-bit) precision needs about 2 bytes per parameter, which puts the raw weights at roughly 8GB. That’s before you add memory for the KV cache, which grows with context length and batch size, and before any framework overhead.
Practically:
- 8GB of VRAM is the rough floor for loading the weights alone, but you’ll want headroom for context. A card with 12 to 16GB gives you comfortable room for a few thousand tokens of context at BF16.
- CPU-only inference is possible with enough system RAM (16GB or more), but will be far slower than GPU inference, and long-context use cases (near the 262K window) become impractical without a GPU.
- Quantization is the standard way to shrink the footprint further. Converting to 8-bit or 4-bit formats (via tools like GGUF or AWQ) can cut memory needs roughly in half or more, at some cost to output quality. The model card documents the BF16 release; quantized community conversions are common for models at this size but weren’t part of the source specs here.
- Long-context runs are the real constraint. The KV cache scales with sequence length, so pushing toward the 262K native window, let alone the 1M extended window, requires significantly more memory than a short chat session. If you plan to use the full context, budget VRAM accordingly or rely on a serving stack with paged attention and cache offloading.
Why does the 262K to 1M context window matter?
Context length determines how much text (documents, code, conversation history, tool outputs) the model can consider at once without truncation. NeoHorse-1-4B’s native window is 262,144 tokens, roughly enough to hold a few hundred pages of text in a single pass. That’s already large enough for most document analysis, long codebases, or extended agent traces without chunking.
The model card also states the architecture is extensible up to 1,010,000 tokens. This kind of extension typically relies on techniques like RoPE scaling and isn’t automatic. It usually requires explicit configuration in the inference runtime and comes with a memory and speed cost proportional to the extra context. For agentic workloads where the model needs to track a long chain of tool calls and outputs, as TokenRhythm’s routing harness approach implies, a large native window is arguably more useful than raw parameter count.
How was NeoHorse-1-4B trained, and does that affect local use?
TokenRhythm describes the model as trained through “routing-guided agentic post-training,” which layers on top of the Qwen3.5-4B base. The approach, per the model card, works like this: a routing harness assigns tasks across a pool of models, logs tool interactions and outcomes, estimates where capability is lacking, and feeds that back into the next training mixture. The company frames this as a prototype loop toward recursive self-improvement (RSI), where updated models re-enter the harness for further evaluation and refinement.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
For someone running the model locally, this training pipeline doesn’t change how you load or serve the weights, it’s still a standard transformer checkpoint. But it does explain why the benchmark gains are concentrated in agentic and tool-calling evaluations rather than general knowledge tests. If your local use case is agent orchestration, function calling, or multi-step tool use, the post-training focus is directly relevant. If you need a general chat assistant, the gains may be less pronounced.
Data preparation is also documented: TokenRhythm applied exact and near-duplicate removal, decontamination against evaluation sets, structural validation, and semantic labeling of training scenes. None of this affects deployment, but it’s useful context for evaluating how much to trust the benchmark claims.
How does it perform compared to similarly sized models?
TokenRhythm’s published evaluation compares NeoHorse-1-4B against five other roughly 4B-class open models: Qwen3.5-4B (its own base), Gemma-4-E4B-it, Nanbeige-4.2-3B, Agents-A1-4B, and Spark-X2.5-4B, across ten benchmarks grouped by capability.
In the agentic category, results are notably strong: NeoHorse-1-4B posts the top score on QwenClawBench (44.68, versus 38.47 for base Qwen3.5-4B), WorkBuddy Bench (34.41 vs 24.62), and PinchBench (77.33 vs 71.19). It’s more middling on VitaBench (32.00, behind Agents-A1-4B’s 39.25) and roughly matches the base model on BFCL v4 (61.79 vs 61.02) and tau2-Bench. The overall macro average across all ten benchmarks comes to 64.87, a 5.93 point improvement over the 58.94 scored by the unmodified Qwen3.5-4B base.
Read this as: the fine-tuning meaningfully improves agentic and tool-use behavior over the base model, with mixed results on a couple of specific benchmarks where competing 4B models still lead. It’s not a uniform win across every category, but the direction is consistent with the model’s stated purpose.
Is running NeoHorse-1-4B locally worth it?
For anyone building agent workflows, tool-calling pipelines, or long-context document processing on consumer or prosumer hardware, a 4B model with a 262K native window is an attractive size class: small enough to fit on a single GPU, large enough to handle real agentic benchmarks competitively. The Apache 2.0 license also removes friction for commercial or research use.
The trade-offs are the ones inherent to any 4B model: it won’t match larger models on raw reasoning or knowledge-heavy tasks, and the benchmark table shows other 4B-class models still edge it out on a few specific evaluations. If your workload is specifically tool use, function calling, or agent orchestration within a long context, the specs and benchmark profile line up well with local deployment. If you need general-purpose chat quality at the frontier, look at larger models instead.
Frequently Asked Questions
What precision does NeoHorse-1-4B use?
It’s released in BF16 (bfloat16) safetensors format. This is a 16-bit floating point format commonly used for LLM weights, offering a balance between numerical range and memory footprint compared to full 32-bit precision.
Can I run NeoHorse-1-4B on a laptop GPU?
It depends on the GPU’s VRAM. An 8GB card can load the BF16 weights but will be tight on context length. Quantized versions (not part of the official BF16 release documented here) would lower the memory bar further, similar to how other 4B models are typically shrunk for consumer hardware.
Does NeoHorse-1-4B support vision input?
No. This release contains language-model weights only. Any vision components from the base model lineage were removed, and the model is repackaged strictly for text-only inference.
How is NeoHorse-1-4B different from its base model, Qwen3.5-4B?
It’s the same architecture and parameter count, but post-trained using TokenRhythm’s routing-guided agentic pipeline. Tensor values are fine-tuned versions of the original weights; only configuration and tensor key names were changed during repackaging.
What is the largest context length NeoHorse-1-4B can handle?
The native window is 262,144 tokens. The model card states the architecture is extensible up to 1,010,000 tokens, though reaching that scale typically requires specific runtime configuration and substantially more memory for the KV cache.
