K2-Horizon-MoVA-36B-A4B Benchmarks: A 4B-Active Model That Punches Up
K2-Horizon-MoVA-36B-A4B uses just 4B active parameters yet beats larger MoE and dense models on agentic tool use and Terminal-Bench.
What is K2-Horizon-MoVA-36B-A4B?
K2-Horizon-MoVA-36B-A4B is an open-weight Mixture-of-Experts (MoE) language model that stores 36 billion total parameters but activates only 4 billion per token. It uses an attention variant called Mixture-of-Values attention (MoVA), and its model card shows it outperforming several larger open models, including a 550B-parameter Nemotron variant, on agentic tool-use and coding benchmarks, despite running with a fraction of the compute per token.
TL;DR
- Sparse activation is the headline feature: the model holds 36B parameters but only routes 4B of them per forward pass, which keeps inference cost close to a small dense model while retaining the capacity of a much bigger network.
- It beats a 550B model on agentic tool use: on the tau3-Banking benchmark, K2-Horizon-MoVA-36B-A4B scores 26.8 versus 14.2 for Nemotron 3 Ultra, a MoE model with 550B total and 55B active parameters.
- Terminal-Bench 2.1 is its strongest coding result: it posts 58.6, ahead of every comparison model listed on the card, including Nemotron 3 Ultra (53.9) and Muse Glimmer-30B (51.7).
- It doesn’t win everywhere: on GPQA Diamond it scores 80.8, behind Nemotron 3 Ultra’s 86.7, and on SciCode it trails both Muse Glimmer-30B and Gemma 4 31B-it.
- 512K native context is built in from the midtraining stage onward, useful for long agentic sessions or large codebases.
- The release is meant to be studied, not just used: the maintainers plan to publish intermediate checkpoints, training data, and training code alongside the final weights.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
How does K2-Horizon-MoVA-36B-A4B compare to Nemotron and Qwen3.6?
The benchmark table on the model card lines up K2-Horizon-MoVA-36B-A4B against six open-weight models: Nemotron 3 Ultra, Nemotron 3 Super, G9v3-39A5B, Qwen3.6-35B-A3B, Muse Glimmer-30B, and Gemma 4 31B-it. The comparison set spans both MoE and dense architectures, which makes the active-parameter gap the real story.
Nemotron 3 Ultra is the largest model in the set by far: 550B total parameters with 55B activated per token, roughly 15 times K2-Horizon-MoVA-36B-A4B’s active footprint. On tau3-Banking, an agentic tool-use benchmark, K2-Horizon-MoVA-36B-A4B nearly doubles Nemotron 3 Ultra’s score (26.8 vs 14.2). On Terminal-Bench 2.1, it also comes out ahead (58.6 vs 53.9). But Nemotron 3 Ultra reclaims the lead on the two reasoning-heavy evaluations: Humanity’s Last Exam without tools (28.4 vs 25.2) and GPQA Diamond (86.7 vs 80.8).
Qwen3.6-35B-A3B is the closer size match: 35B total parameters with 3B active, nearly identical to K2-Horizon-MoVA-36B-A4B’s footprint. Here the gap favors K2-Horizon-MoVA-36B-A4B more consistently: 26.8 vs 9.3 on tau3-Banking, 58.6 vs 44.9 on Terminal-Bench 2.1, and 80.8 vs 84.1 on GPQA Diamond (a rare case where Qwen3.6 edges ahead). Against a model this close in active-parameter count, the agentic and terminal-use gaps are the most notable data points on the card.
Why does the agentic gap matter more than raw parameter count?
Benchmark tables like this usually get read as a ranking, but the more useful lens is efficiency per activated parameter. Total parameter count affects storage and loading, but activated parameter count is what governs inference latency and compute cost for every token generated. A model that activates 4B parameters and beats one activating 55B on the same task is doing more useful work per FLOP.
That’s the framing IFM leans into with K2-Horizon-MoVA-36B-A4B: on agentic and reasoning benchmarks, it claims to outscore open dense models around 30B in size and MoE models up to 15 times its size, while staying competitive with closed frontier models. The tau3-Banking and Terminal-Bench 2.1 results back that framing up for agentic and coding-agent workloads specifically. Those two benchmarks test tool use and terminal interaction, tasks that resemble what a coding agent or automation pipeline actually does in production, rather than static single-turn Q&A.
Where does the model fall short?
The card doesn’t paint an unqualified win. Three results stand out as weaknesses:
On SciCode, a scientific coding benchmark, K2-Horizon-MoVA-36B-A4B scores 38.9, behind Nemotron 3 Ultra (39.9), Muse Glimmer-30B (43.6), and Gemma 4 31B-it (43.4). Scientific coding tasks tend to require precise domain knowledge and multi-step derivations, an area where the two dense models in the comparison (Muse Glimmer-30B and Gemma 4 31B-it) hold their own despite having no MoE routing advantage.
On Humanity’s Last Exam without tool use, a benchmark designed to probe expert-level reasoning across disciplines, Nemotron 3 Ultra’s 550B-parameter scale pulls it to 28.4 against K2-Horizon-MoVA-36B-A4B’s 25.2. Raw scale still seems to help on the hardest, most knowledge-dense reasoning questions when tools aren’t available to compensate.
On GPQA Diamond, a graduate-level science QA set, K2-Horizon-MoVA-36B-A4B’s 80.8 sits below both Nemotron 3 Ultra (86.7) and Qwen3.6-35B-A3B (84.1). The pattern across all three weaker results points to the same conclusion: sparse activation and MoVA attention seem tuned for agentic and tool-driven tasks more than for pure knowledge recall or dense scientific reasoning.
What is Mixture-of-Values attention, and why does it matter here?
The model card names MoVA as a defining architectural feature but doesn’t fully spell out its mechanics in public documentation yet. What’s confirmed is that it’s paired with the MoE routing to let the model store 36B parameters of capacity while activating only 4B per token, and that it’s compatible with the model’s native 512K token context window from the midtraining stage onward. Long-context support paired with sparse activation is relevant for agentic workloads specifically: agents that maintain long tool-call histories or work across large codebases need both the context length and the low per-token cost to stay usable in production.
Because IFM has committed to releasing training data, training code, and intermediate checkpoints (not just the final weights), researchers will be able to inspect how MoVA’s behavior evolves over the training run rather than only seeing the finished capability profile. That’s a meaningfully more open release stance than most frontier-adjacent model drops, which typically ship only final weights and a technical report.
Is K2-Horizon-MoVA-36B-A4B worth using for agentic or coding work?
Based on the published benchmark numbers, it looks like a strong pick specifically for tool-using agents and terminal-based coding tasks, where it leads every open model in the comparison set. If the workload centers on GPQA-style scientific QA or SciCode-style scientific programming, the numbers suggest a dense model like Gemma 4 31B-it or a much larger MoE like Nemotron 3 Ultra may still have an edge. The practical takeaway is that “biggest model” and “best model for the job” aren’t the same axis here: a 4B-active model is demonstrably competitive with a 55B-active one on the tasks that matter most for coding agents, even if it isn’t uniformly superior across every benchmark category.
Frequently Asked Questions
What does “A4B” mean in K2-Horizon-MoVA-36B-A4B?
It refers to activated parameters: the model has 36 billion total parameters distributed across its Mixture-of-Experts layers, but only about 4 billion are activated for any given token during inference.
How does K2-Horizon-MoVA-36B-A4B perform against Nemotron 3 Ultra?
It beats Nemotron 3 Ultra on tau3-Banking (26.8 vs 14.2) and Terminal-Bench 2.1 (58.6 vs 53.9), but loses on GPQA Diamond (80.8 vs 86.7) and Humanity’s Last Exam without tools (25.2 vs 28.4), despite Nemotron 3 Ultra activating roughly 14 times more parameters per token.
Does K2-Horizon-MoVA-36B-A4B support long context?
Yes. It has a native 524,288-token (512K) context window, in place from the midtraining stage of training onward.
Will the full training data and code be released?
According to the model card, IFM plans to release intermediate checkpoints along with training data and training code, in addition to the final model weights already published.
Is K2-Horizon-MoVA-36B-A4B better than Qwen3.6-35B-A3B?
On the published benchmarks it leads Qwen3.6-35B-A3B on tau3-Banking and Terminal-Bench 2.1 by wide margins, while Qwen3.6-35B-A3B scores slightly higher on GPQA Diamond (84.1 vs 80.8). The two models have almost identical total and active parameter counts, making this the closest architecture-matched comparison on the card.
