How to Run K2-Horizon-MoVA-36B-A4B Locally
A practical guide to running K2-Horizon-MoVA-36B-A4B locally: hardware needs, quantization options, and how its 512K context and MoE design work.
What is K2-Horizon-MoVA-36B-A4B?
K2-Horizon-MoVA-36B-A4B is an open-weight Mixture-of-Experts (MoE) language model that stores 36 billion parameters total but activates only about 4 billion per token. It uses a technique the model card calls Mixture-of-Values attention (MoVA), and it ships with a native context window of 524,288 tokens (512K), meaning long-document and long-conversation support is built into the architecture rather than bolted on later. The model’s small active-parameter footprint is what makes local deployment realistic on prosumer and workstation-class hardware instead of requiring a multi-GPU server rack.
TL;DR
- Sparse activation is the whole story here: only 4B of the 36B parameters fire per token, which is why this model can run on hardware that would choke on a dense 30B+ model.
- 512K native context was trained in from the midtraining stage onward, so long-context performance isn’t a fine-tuned add-on, it’s part of the base checkpoint.
- VRAM needs scale with total parameters, not active ones, since MoE weights all have to sit in memory even though only a fraction compute per forward pass.
- Quantized GGUF or AWQ builds are the realistic path for most local setups, since full-precision 36B weights demand data-center-grade GPUs.
- Benchmark results are strong for the parameter class: it beats open dense and MoE models many times its size on agentic and coding benchmarks like tau3-Banking and Terminal-Bench 2.1.
- Only the final checkpoint is public right now, with intermediate checkpoints, training data, and training code promised for later release.
- Inference engines with MoE support (vLLM, llama.cpp, SGLang) are required, since the architecture needs expert-routing logic that generic transformer loaders may not handle correctly.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
How does the MoE architecture affect hardware requirements?
The defining fact about K2-Horizon-MoVA-36B-A4B is the split between total and active parameters. The model has 36B parameters in total, spread across multiple experts, but a router selects a subset for each token, activating only around 4B parameters per forward pass. That’s the source of its speed and its ability to compete with much larger models on inference latency and compute cost per token.
The catch: memory requirements are driven by total parameters, not active ones. Every expert has to be loaded into VRAM or system RAM even if only a few are used per token, because the router can pick different experts for different tokens and even different layers. So while the model computes like a 4B model, it occupies memory more like a 36B model. This is the central tradeoff to understand before picking hardware: MoE models are cheap to run per-token but not cheap to load.
For comparison, the model card lines it up against other open MoE and dense models in its class, including Qwen3.6-35B-A3B (35B total, 3B active) and G9v3-39A5B (39B total, 5B active). K2-Horizon-MoVA-36B-A4B sits in the same bracket: a 36B-parameter footprint you need to fit in memory, with a 4B-parameter compute cost per token.
What hardware do you need to run it locally?
At full precision (FP16/BF16), a 36B-parameter model needs roughly 70-75GB of memory just for weights, before accounting for the KV cache that a 512K context window demands. That rules out most single-consumer GPUs for full-precision inference.
Realistic local setups depend on quantization:
- 4-bit quantization (GGUF Q4, AWQ, GPTQ) brings weight memory down to roughly 18-20GB, putting it within reach of a single 24GB consumer GPU (RTX 4090, RTX 3090, or similar) for the weights alone.
- 8-bit quantization roughly doubles that to the high 30GB range, suited to a 40GB or 48GB professional card (A6000, L40) or dual consumer GPUs split across memory.
- KV cache overhead grows with context length. Running near the full 512K context window adds substantial memory on top of the base weight footprint, even with the model’s presumably efficient attention design. Shorter working contexts (8K-32K) keep KV cache overhead modest and are a more practical starting point for most local setups.
- CPU or hybrid CPU/GPU inference is viable through llama.cpp-style engines, offloading some experts to system RAM. This is slower but avoids the need for a high-VRAM GPU entirely, and it’s a common strategy for MoE models since inactive experts don’t need to be near the compute unit at all times.
A single 24GB GPU is a reasonable floor for a heavily quantized version with a moderate context window. Comfortable full-context use at 512K, with headroom for concurrent requests, points toward 48GB+ of VRAM or a multi-GPU setup.
How do you actually load and run it?
Because this is an MoE model with a non-standard attention mechanism, you need an inference stack that explicitly supports MoE routing and whatever the MoVA attention implementation requires. Generic single-file loaders built only for dense transformer architectures may not route experts correctly or may simply fail to load the checkpoint.
Practical options, in order of typical local-use popularity:
- llama.cpp / GGUF builds: the most common path for consumer hardware once a quantized GGUF conversion exists. Supports CPU offload, partial GPU offload, and various quant levels (Q4_K_M, Q5_K_M, Q8_0) that trade memory for accuracy.
- vLLM: better suited to GPU-only setups where you want higher throughput and batched requests, commonly used when serving the model behind an API for multiple users.
- SGLang or similar MoE-aware serving frameworks: useful if you want production-style serving with expert-parallelism across multiple GPUs.
Whichever engine you use, check that it explicitly lists support for the model’s architecture family before assuming it will “just work.” MoE routing logic and long-context attention variants are exactly the kind of thing that breaks silently in unsupported runtimes, producing garbled output rather than a clear error.
Is running it locally worth it compared to a hosted API?
For teams already paying per-token for closed frontier models, the appeal of K2-Horizon-MoVA-36B-A4B is cost and control at a favorable performance tier. The benchmark table shows it outperforming open dense and MoE models several times its size on agentic tool-use tasks (tau3-Banking) and terminal-based coding tasks (Terminal-Bench 2.1), while running at a compute cost closer to a 4B model. That combination, strong task performance with cheap inference, is the main argument for self-hosting: you get frontier-adjacent capability without frontier-scale inference bills, and you keep data on your own infrastructure.
The tradeoffs are real, though. You need to manage quantization quality yourself, tune context length against available memory, and keep an inference stack updated as MoE tooling evolves quickly. Scientific reasoning benchmarks like GPQA Diamond and Humanity’s Last Exam show it trailing larger models like Nemotron 3 Ultra, so it’s not a universal replacement for the largest closed or open models on every task. It’s best suited to workloads where agentic tool use, coding, and long-context handling matter more than the very top end of expert-level scientific reasoning.
Frequently Asked Questions
How much VRAM does K2-Horizon-MoVA-36B-A4B need?
It depends on quantization. Full precision needs roughly 70-75GB for weights alone. A 4-bit quantized version fits in around 18-20GB, workable on a single 24GB consumer GPU, though the KV cache for long contexts adds more on top.
Does the 512K context window require special hardware?
Not special hardware, but more memory. The KV cache grows with sequence length, so running near the full 512K context consumes significantly more memory than a short 8K-32K working context. Most local users will keep context lengths well below the maximum unless they have a high-VRAM setup.
Can you run this model on a CPU only?
Yes, through engines like llama.cpp that support CPU inference and partial GPU offload, but expect much slower generation speeds than a GPU-based setup. This works because MoE models only need the active experts near the compute unit at any moment, which suits hybrid CPU/RAM and GPU/VRAM configurations.
Is the full training data and code available yet?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
No. The model card states that only the final checkpoint has been released so far. Intermediate checkpoints, training data, and training code are planned for later release, which will let researchers study how capabilities changed across training rather than only at the final checkpoint.
How does it compare to similarly sized open models?
On agentic and coding benchmarks like tau3-Banking and Terminal-Bench 2.1, it scores higher than several open dense and MoE models of comparable or larger size, including Qwen3.6-35B-A3B and G9v3-39A5B. On scientific reasoning benchmarks like GPQA Diamond, it trails larger models such as Nemotron 3 Ultra, reflecting the usual tradeoff between active parameter count and raw reasoning depth.