K2-Horizon-MoVA-36B-A4B Benchmarks vs Nemotron, Qwen, Gemma
K2-Horizon-MoVA-36B-A4B runs 4B active params yet beats models up to 15x larger on agent tasks. Here's how it stacks up on benchmarks.
What is K2-Horizon-MoVA-36B-A4B?
K2-Horizon-MoVA-36B-A4B is an open-weight Mixture-of-Experts (MoE) language model that stores 36 billion parameters but activates only 4 billion per token. It uses an attention variant called Mixture-of-Values attention (MoVA), and its release includes the final training checkpoint, with intermediate checkpoints, training data, and training code promised for later publication. The pitch is simple: get results close to models many times its size while running a fraction of the compute per forward pass.
TL;DR
- K2-Horizon-MoVA-36B-A4B activates just 4B parameters out of a 36B total, making it cheap to run per token compared to dense models of similar or larger size.
- On tau3-Banking, an agentic tool-use benchmark, it scores 26.8, ahead of every other model in its published comparison table, including Nemotron 3 Ultra (a 550B-parameter model with 55B active).
- On Terminal-Bench 2.1, a benchmark for agentic terminal use, it leads the pack at 58.6, beating Nemotron 3 Ultra’s 53.9 and Muse Glimmer-30B’s 51.7.
- It does not win everywhere: Nemotron 3 Ultra still leads on Humanity’s Last Exam (28.4 vs 25.2) and GPQA Diamond (86.7 vs 80.8), and Gemma 4 31B-it edges it out on SciCode.
- The model ships with native 512K token context (524,288 tokens) starting from its midtraining stage, useful for long-document or long-agent-trajectory workloads.
- The comparison set spans MoE models (Nemotron 3 Ultra/Super, G9v3-39A5B, Qwen3.6-35B-A3B) and dense models (Muse Glimmer-30B, Gemma 4 31B-it), giving a reasonably broad picture of where sparse architectures stand against dense ones right now.
- The full training pipeline, including intermediate checkpoints and training data, is planned for release, which would let researchers study how these capabilities emerge rather than judging a single frozen snapshot.
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
How does K2-Horizon-MoVA-36B-A4B compare on agent benchmarks?
Agent benchmarks measure whether a model can use tools, follow multi-step instructions, and operate in something resembling a real workflow rather than just answering a static question. This is where K2-Horizon-MoVA-36B-A4B posts its strongest numbers.
On tau3-Banking, a benchmark testing agentic tool use in banking-style scenarios, K2-Horizon-MoVA-36B-A4B scores 26.8. The next closest is G9v3-39A5B at 22.1, then Nemotron 3 Ultra at 14.2, Gemma 4 31B-it at 14.8, Nemotron 3 Super at 10.3, and Muse Glimmer-30B at 9.3. That’s a meaningful gap over models with far more total and active parameters, including Nemotron 3 Ultra’s 550B total and 55B active parameters, roughly 15 times K2-Horizon’s active parameter count.
On Terminal-Bench 2.1, which tests whether a model can operate a terminal environment agentically (running commands, interpreting output, recovering from errors), K2-Horizon-MoVA-36B-A4B again leads at 58.6, ahead of Nemotron 3 Ultra (53.9), Muse Glimmer-30B (51.7), Qwen3.6-35B-A3B (44.9), Gemma 4 31B-it (43.4), and Nemotron 3 Super (38.6).
Together these two results are the core of the “frontier-class results at 4B active parameters” claim on the model card. Agent tasks reward a model’s ability to plan and chain actions correctly, and that seems to be where this architecture’s design pays off most visibly in the published numbers.
How does it perform on coding and scientific reasoning?
Coding results are more mixed. On SciCode, a benchmark for scientific coding tasks, K2-Horizon-MoVA-36B-A4B scores 38.9, behind Muse Glimmer-30B (43.6) and Gemma 4 31B-it (43.4), and also slightly behind Nemotron 3 Ultra (39.9). It’s ahead of Nemotron 3 Super (36.0), Qwen3.6-35B-A3B (35.8), and G9v3-39A5B (34.0), but this isn’t a category where it dominates.
Scientific reasoning tells a similar story. On Humanity’s Last Exam (run without tools), a benchmark designed to probe expert-level reasoning across disciplines, Nemotron 3 Ultra leads at 28.4, with K2-Horizon-MoVA-36B-A4B at 25.2, still ahead of Nemotron 3 Super (20.8), Qwen3.6-35B-A3B (17.5), Muse Glimmer-30B (22.2), and Gemma 4 31B-it (23.6).
On GPQA Diamond, a graduate-level science question-answering benchmark, the pattern holds: Nemotron 3 Ultra tops the table at 86.7, and K2-Horizon-MoVA-36B-A4B sits at 80.8, roughly matching G9v3-39A5B (80.5) and Nemotron 3 Super (80.0).
The takeaway is that raw scientific knowledge recall and deep expert reasoning still favor the largest model in the comparison, Nemotron 3 Ultra, which activates far more parameters per token. K2-Horizon-MoVA-36B-A4B trades some of that ceiling for efficiency, and wins decisively where the task is about acting rather than just knowing.
Is a 4B active parameter model actually competitive with much larger ones?
Based on the published benchmark table, yes, but selectively. K2-Horizon-MoVA-36B-A4B is not uniformly better than every model it’s compared against. It loses to Nemotron 3 Ultra, a 550B-parameter model with 55B active, on both scientific reasoning benchmarks (Humanity’s Last Exam and GPQA Diamond). It also loses narrowly to dense models like Gemma 4 31B-it and Muse Glimmer-30B on SciCode.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Where it wins, it wins clearly: both agent benchmarks (tau3-Banking and Terminal-Bench 2.1) put it in first place across the entire comparison set, including against models with 15 times its active parameter count. That’s a specific and verifiable claim rather than a blanket “beats everything” statement, and it lines up with a broader trend in MoE research: sparse architectures with strong training recipes can match or exceed dense or larger MoE models on tasks that reward procedural, tool-using behavior, even if they don’t automatically win on pure knowledge-recall benchmarks.
For teams building agent-driven applications (tool use, terminal automation, multi-step task execution), these numbers suggest K2-Horizon-MoVA-36B-A4B is worth evaluating directly against workloads that resemble tau3-Banking or Terminal-Bench style tasks. For teams that need maximum performance on deep scientific reasoning or graduate-level QA, the benchmark table suggests a larger dense-per-token model like Nemotron 3 Ultra still has an edge.
What does the MoVA architecture and 512K context add?
Mixture-of-Values attention (MoVA) is the attention mechanism underpinning this model, though the model card doesn’t detail its internal mechanics beyond naming it. What is specified is the practical effect: alongside the MoE routing that keeps per-token compute at 4B parameters, the model supports a native context window of 524,288 tokens (512K), active from the midtraining stage onward.
A context window that large matters for workloads like long agent trajectories (many tool calls and observations accumulating in context), long codebases, or long documents that need to be processed without chunking. Combined with the low active-parameter count, this is aimed at making long-context, tool-heavy workloads cheaper to run than they’d be on a dense model or a larger MoE with a higher per-token compute cost.
How does this fit into the broader open MoE model race?
The comparison table itself is a useful snapshot of where open MoE and dense models stand as of this release. It includes MoE models spanning three orders of magnitude in total size, from K2-Horizon’s 36B up to Nemotron 3 Ultra’s 550B, alongside dense models like Muse Glimmer-30B and Gemma 4 31B-it. The fact that a 4B-active model can lead on two agent benchmarks against a 55B-active model reinforces a pattern seen across the MoE field: total parameter count is a weak predictor of task performance once you’re comparing well-trained sparse models against each other, and active parameter count, training data quality, and architecture choices matter more.
Whether K2-Horizon-MoVA-36B-A4B holds up outside these specific benchmarks depends on independent testing once the full checkpoint, training code, and data become available, which the model card commits to releasing.
Frequently Asked Questions
What does “36B-A4B” mean in the model name?
It means the model has 36 billion total parameters stored across its Mixture-of-Experts layers, but only 4 billion of those parameters are activated (used in computation) for any given token. This is standard MoE naming shorthand: total size followed by active size.
Which models does K2-Horizon-MoVA-36B-A4B beat on the published benchmarks?
It leads the comparison table on tau3-Banking (agentic tool use) and Terminal-Bench 2.1 (agentic terminal use), ahead of Nemotron 3 Ultra, Nemotron 3 Super, G9v3-39A5B, Qwen3.6-35B-A3B, Muse Glimmer-30B, and Gemma 4 31B-it. It trails Nemotron 3 Ultra on Humanity’s Last Exam and GPQA Diamond, and trails Muse Glimmer-30B and Gemma 4 31B-it on SciCode.
Is K2-Horizon-MoVA-36B-A4B fully open source?
The final trained checkpoint is released as open weights. The model card states that intermediate checkpoints, training data, and training code are planned for future release, which would make the full training process reproducible and auditable, not just the final model.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
What is Mixture-of-Values attention (MoVA)?
MoVA is the attention mechanism used in this model family. The model card names it as a distinguishing architectural feature but doesn’t publish implementation details in the released documentation, so its exact mechanics aren’t independently confirmed beyond the model card’s description.
How long a context window does the model support?
It supports a native 524,288-token (512K) context window, active from the midtraining stage of its training process onward, according to the model card.
