K2-Horizon-MoVA-36B-A4B: A 36B MoE Model You Can Actually Run Locally
K2-Horizon-MoVA-36B-A4B packs 36B parameters with only 4B active per token and 512K context. Here's what it takes to run it.
What is K2-Horizon-MoVA-36B-A4B?
K2-Horizon-MoVA-36B-A4B is an open-weight language model from IFM that stores 36 billion parameters but activates only 4 billion of them per token. It uses a Mixture-of-Experts (MoE) design combined with something the model card calls Mixture-of-Values attention (MoVA), and it ships with a native context window of 524,288 tokens. Released under an Apache 2.0 license, it’s positioned as a sparse model that punches well above its active parameter count on agentic and reasoning benchmarks.
TL;DR
- Sparse activation is the headline feature: the model holds 36B total parameters but only 4B are active per forward pass, which is what keeps inference memory and compute manageable compared to a dense model of similar total size.
- 512K context comes standard: a native window of 524,288 tokens is baked in from the midtraining stage onward, not bolted on with a later context-extension patch.
- Benchmark claims target agentic and reasoning tasks: on tests like tau3-Banking (agentic tool use) and Terminal-Bench 2.1 (agentic terminal use), it reportedly outperforms several larger open models, including some with far more active parameters.
- The comparison set includes much bigger models: the model card benchmarks it against models with up to 550B total parameters and 55B active parameters, framing it as an efficiency play rather than a raw-scale play.
- Weights ship in 48 safetensors shards: the Hugging Face repo lists model-00001-of-00048.safetensors through model-00048-of-00048.safetensors, plus a custom modeling file (modeling_k2_horizon.py) and configuration file, meaning it needs
trust_remote_codesupport in your inference stack. - Full openness is promised, not fully delivered yet: IFM says intermediate training checkpoints, training data, and training code will be released, but at launch only the final checkpoint is public.
- It’s not dominant across the board: on GPQA Diamond and Humanity’s Last Exam, larger models like Nemotron 3 Ultra score higher, so the efficiency gains don’t mean it wins every category.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
How does the Mixture-of-Experts design work here?
A standard MoE model splits its feed-forward layers into multiple “expert” sub-networks. For any given token, a routing mechanism picks a small subset of experts to actually compute, rather than running every parameter in the network. This is why a model can list 36B total parameters while only touching 4B of them on a given token: most of the network sits idle for that particular computation, then a different subset activates for the next token.
The practical effect is that inference cost (in FLOPs, roughly) tracks closer to the 4B active figure than the 36B total figure, even though you still need to hold the full 36B in memory (or offload it) because you don’t know in advance which experts each token will need. This is the same tradeoff seen in other sparse MoE releases: you trade memory footprint for compute efficiency, not the other way around.
The model card also lists Mixture-of-Values attention (MoVA) as part of the architecture, positioned alongside the MoE routing in the feed-forward layers. The public documentation doesn’t spell out MoVA’s internal mechanics beyond the name, so treat it as a distinguishing attention-layer design choice rather than a fully documented technique, at least until IFM publishes the training code and more detailed methodology.
What do the benchmark numbers actually show?
The model card benchmarks K2-Horizon-MoVA-36B-A4B against a mix of open MoE models, open dense models, and (per the card’s framing) closed frontier models. The comparison set includes Nemotron 3 Ultra (550B total, 55B active), Nemotron 3 Super (120B total, 12B active), a model listed as G9v3-39A5B (39B total, 5B active), Qwen3.6-35B-A3B (35B total, 3B active), and two dense models, Muse Glimmer-30B and Gemma 4 31B-it.
On tau3-Banking, an agentic tool-use benchmark, K2-Horizon-MoVA-36B-A4B scores 26.8, ahead of every listed comparison model in the table (Nemotron 3 Ultra at 14.2, Qwen3.6-35B-A3B at 9.3, Gemma 4 31B-it at 14.8). On Terminal-Bench 2.1, a test of agentic terminal use, it scores 58.6, again the top score shown, ahead of Nemotron 3 Ultra’s 53.9 and Muse Glimmer-30B’s 51.7.
It doesn’t sweep every category. On SciCode, a scientific coding benchmark, Muse Glimmer-30B leads at 43.6 versus K2-Horizon’s 38.9. On GPQA Diamond, a graduate-level science QA test, Nemotron 3 Ultra leads at 86.7 versus K2-Horizon’s 80.8, and on Humanity’s Last Exam (without tools), Nemotron 3 Ultra again leads at 28.4 versus 25.2. The pattern that emerges: K2-Horizon-MoVA-36B-A4B is strongest on agentic, tool-use, and terminal tasks, and more middling on pure scientific knowledge recall and closed-book expert reasoning, where the much larger Nemotron 3 Ultra (13x more active parameters) still wins.
What hardware do you need to run it locally?
The repo ships weights as 48 safetensors shards, and with 36B total parameters, memory planning has to account for the full parameter count even though only 4B activate per token. As a rough rule of thumb for any transformer, a bf16/fp16 checkpoint needs about 2 bytes per parameter, so 36B parameters lands around 70-72GB just for weights, before accounting for KV cache, activations, or the overhead from a 512K context window. Running the model at or near its full context length will add substantially to memory needs because KV cache size scales with sequence length and batch size, and a window that large is one of the bigger practical costs of using this model, independent of the MoE routing.
For anyone without multi-GPU workstations or datacenter cards, quantized versions (4-bit or 8-bit, likely to appear from the community given the model’s Apache 2.0 license and Hugging Face presence) would substantially cut memory requirements, potentially into single high-VRAM consumer GPU territory (24GB-48GB class cards) for the 4-bit case, though the model card itself doesn’t publish official quantized checkpoints or exact VRAM figures. Because the model uses a custom architecture (k2_horizon model type with custom modeling_k2_horizon.py and configuration_k2_horizon.py files), it also requires trust_remote_code=True in Transformers, and broader ecosystem support (llama.cpp, vLLM, Ollama) will depend on the community or IFM porting the custom architecture, which isn’t guaranteed on day one for a brand-new architecture.
Is K2-Horizon-MoVA-36B-A4B worth running over a dense alternative?
If your workload leans toward agentic tasks (tool calling, terminal automation, multi-step task execution) the benchmark results suggest a real edge over both larger MoE models and similarly-sized dense models like Gemma 4 31B-it or Muse Glimmer-30B. The active-parameter efficiency (4B active vs. 30-31B active for the dense comparisons) means inference should be meaigfully faster and cheaper per token, assuming your serving stack supports the MoE routing efficiently.
If your workload is dominated by closed-book scientific reasoning or knowledge-heavy QA, the numbers here suggest a much larger dense or MoE model, such as Nemotron 3 Ultra, will outperform it, at the cost of needing far more compute to run. The decision comes down to matching the benchmark profile to your actual use case rather than chasing a single aggregate score.
Frequently Asked Questions
What does “36B-A4B” mean in the model name?
It denotes the total and active parameter counts: 36 billion total parameters stored in the model, with 4 billion active (used in computation) for any given token, thanks to its Mixture-of-Experts routing.
What is MoVA in this model’s architecture?
MoVA stands for Mixture-of-Values attention, an attention-layer design paired with the model’s MoE feed-forward routing. The public model card names it but doesn’t detail its internal mechanics; more information is expected once IFM releases training code.
Does K2-Horizon-MoVA-36B-A4B support a 512K context window out of the box?
Yes. The model card states it has a native 524,288-token context window that was established from the midtraining stage onward, rather than added later through extension techniques.
Can I run this model on a single consumer GPU?
Not in full precision. At roughly 36B parameters, a bf16 checkpoint needs on the order of 70GB+ for weights alone, before context and KV cache overhead. A quantized version (4-bit or 8-bit) would lower that substantially, but the model card does not list official quantized releases or exact VRAM benchmarks.
Is the training data and code available yet?
Not at initial release. IFM has published the final model checkpoint and says intermediate checkpoints, training data, and training code will follow, but as of the current Hugging Face listing, only the final weights and documentation are public.
