NVIDIA Nemotron Bolt 74B 2-Bit: How to Run It Locally on 24GB
NVIDIA's Nemotron Bolt packs a 74B MoE model into 24GB using 2-bit experts. Here's how it works, how it performs, and how to run it.
What is Nemotron Bolt 74B-A9B and why does it matter?
NVIDIA Nemotron 3.5 Bolt 74B-A9B is a mixture-of-experts model with 74 billion total parameters and roughly 9 billion active per token, built on NVIDIA’s hybrid Mamba-transformer architecture. It handles text, images, and video, supports context windows up to 256K tokens, and ships with its experts quantized to 2 bits, letting the entire model fit inside 24GB of VRAM. That makes it one of the first large, capable MoE models designed from the ground up to run on a single consumer-grade GPU like the RTX Spark, alongside other apps, rather than needing a multi-GPU server.
The model matters less because of raw benchmark scores and more because of how NVIDIA got there. It’s not a model someone rounded down after training. It’s the end product of a pruning, distillation, and quantization-aware training pipeline that started with a 120-billion-parameter parent model. That pipeline is arguably more useful to understand than the model itself, because it previews where small-but-capable local models are headed.
TL;DR
- Nemotron Bolt 74B-A9B is a 74B-parameter MoE model with about 9B active parameters per token, quantized so the whole thing fits in 24GB of VRAM.
- The 2-bit claim is specific to the expert layers; the Mamba/attention layers run at NVFP4 and embeddings sit at 8-bit, making this a mixed-precision model rather than uniformly 2-bit.
- NVIDIA used quantization-aware training and distillation (QAD), not post-training rounding, which is why it avoids the steep quality collapse normally seen below 4-bit.
- The model is derived from Nemotron 3.5 Super, a 120B-parameter, 12B-active parent, shrunk via an architecture search method called Star Elastic that cut active experts per token from 22 down to 9.
- On NVIDIA’s own benchmarks, compressing from BF16 (148GB) to the 2-bit release (22GB) dropped scores only slightly, for example around 90 to 88 on one open benchmark and around 89 to 86 on GPQA Diamond.
- Real-world throughput on an RTX Spark comes in around 84 tokens per second, which trails Qwen 3 and Gemma 4 models of similar size, meaning Bolt is pitched as a smarter-per-gigabyte model, not a faster one.
- The same recipe (prune, distill, quantization-aware train) is becoming a pattern across the industry, with Google’s Gemma 4, OpenAI’s GPT-OSS, and Moonshot’s Kimi models all applying some version of it.
How does 2-bit quantization usually break models, and how did NVIDIA avoid it?
The standard local-AI rule of thumb has been to pick the largest model you can fit at 4-bit and avoid going much lower. Post-training quantization, the kind used for most GGUF Q4 or Q8 files, takes a model trained in 16-bit and rounds every weight down afterward, usually guided by a small calibration dataset to minimize error. At 8-bit this is nearly free in terms of quality loss. At 4-bit it’s usually still close to free, which is why 4-bit became the default.
Below 4-bit, though, post-training quantization tends to fall apart. The transition between 2-bit and 3-bit is where researchers typically see the sharpest quality drop. At 3-bit and above, a quantized model usually stays reasonably close to the original. At 2-bit and below, the internal representations change so much that the model effectively needs to relearn how to function, not just get rounded.
NVIDIA’s approach sidesteps this by not treating quantization as a final, separate step. It uses quantization-aware training (QAT), where low-bit rounding is simulated during training itself while a full-precision copy of the weights handles the actual updates. The model learns to operate well on the lower-bit “grid” instead of being forced onto it after the fact. NVIDIA adds a twist called quantization-aware distillation (QAD): the quantized model acts as the student, distilling from the full-precision version, which lets the team reuse the benefits of supervised fine-tuning and reinforcement learning without needing to rerun the full training dataset.
This pattern isn’t unique to NVIDIA. Google used QAT for Gemma 3 and extended it across the whole Gemma 4 family, pushing the smallest models’ token-generation layers down to 2-bit while keeping core reasoning layers at higher precision. OpenAI used a related approach for its GPT-OSS 120B model, which is part of why it fits on a single 80GB GPU. Moonshot has applied similar ideas to Kimi. The common thread: major labs are increasingly planning for low-bit deployment during training rather than leaving it to the open-source community to figure out afterward.
How was Nemotron Bolt actually built?
The pipeline runs in three stages.
Stage one: prune with a memory budget in mind. NVIDIA starts from Nemotron 3.5 Super, a 120-billion-parameter model with about 12 billion active parameters. A method called Star Elastic performs an architecture search under a fixed memory budget: a smaller router decides, layer by layer, how many experts to keep and how wide each MoE block should be. The number of active experts per token drops from 22 to 9. Critically, the budget isn’t just “hit 74B parameters.” It’s defined as fitting under 24GB at batch size 1 with a 128K context window, and that budget already accounts for the experts eventually landing at 2-bit. The Mamba state layers stay at 16-bit throughout this stage. The pruned model is then distilled from its parent using around 120 billion tokens, mostly at 8K context for speed, with later training stretching to 128K to restore long-context performance.
Stage two: standard post-training. The vision encoder, separated earlier, gets reattached. NVIDIA runs further supervised fine-tuning, merges checkpoints, and does multiple rounds of reinforcement learning aimed at agentic tasks. At this point the model is still in BF16.
Stage three: quantization-aware distillation. This is where the 2-bit experts, NVFP4 Mamba/attention layers, and 8-bit embeddings get baked in through QAD rather than a simple post-hoc rounding pass.
How fast is it, and is it actually worth running locally?
Speed is where the tradeoffs show up. On an RTX Spark, NVIDIA’s own numbers put Bolt at around 84 tokens per second, which is behind comparably sized Qwen 3 and Gemma 4 releases. Testing on an NVIDIA GB10-class system (DGX Spark hardware) showed throughput around 220 tokens per second, noticeably slower than running the full Nemotron 3.5 Super model on the same hardware, since Super has more headroom for parallel compute even though it’s a larger file. The takeaway: this is a smarter-per-gigabyte model, not a faster-per-gigabyte one.
Three reasons still make the tradeoff reasonable for local setups. First, at batch size 1 (the typical local use case), you’re bottlenecked by memory bandwidth, not compute, so fewer bits per weight directly means fewer bytes to move and more tokens per second, all else equal. Second, headroom matters: an agent setup running a KV cache, embedding model, speech model, browser, and IDE all on one machine needs the LLM itself to leave room. Third, the choice between a 35B model at 4-bit and a 70B model at 2-bit in a similar footprint is a real design decision, and NVIDIA’s benchmark drops (roughly 90 to 88 on one benchmark, 89 to 86 on GPQA Diamond when moving from the 148GB BF16 version to the 22GB quantized version) suggest the bigger, more aggressively quantized model holds up better than expected.
What are the alternatives to this approach?
Two other techniques are worth knowing if you’re tracking this space. Expert pruning, used in methods like Cerebras’s REAP, drops entire experts from a MoE model rather than shrinking every layer uniformly. Research here shows pruning whole experts tends to outperform merging them, since merging damages the router’s ability to pick the right expert, though which experts get cut depends heavily on the calibration data used (a code-focused calibration set will preserve coding ability at the expense of other skills).
The more radical alternative is Microsoft’s BitNet, which trains models from scratch with weights restricted to -1, 0, or 1, the so-called 1.58-bit approach. It shows promise but currently lacks proper hardware acceleration on mainstream GPUs, which limits its practicality compared to NVIDIA’s train-big-then-compress strategy.
Frequently Asked Questions
Is Nemotron Bolt 74B really running entirely at 2-bit?
No. The 2-bit precision applies specifically to the mixture-of-experts layers. The Mamba/attention (member) layers run at NVFP4 (4-bit), and the embeddings and output layer run at 8-bit. It’s a mixed-precision model, and NVIDIA is explicit about this in the model card.
How much VRAM do I need to run Nemotron Bolt locally?
The quantized release is designed to fit in 24GB of VRAM, which is why NVIDIA paired it with the RTX Spark class of hardware. The full BF16 parent model, by contrast, runs around 148GB.
Why does a 2-bit model actually perform well here, when most 2-bit quantization doesn’t?
Because it wasn’t quantized after training. NVIDIA used quantization-aware training and quantization-aware distillation, simulating low-bit rounding during training itself so the model adapts to the lower precision instead of being forced onto it afterward.
Is Nemotron Bolt faster than similarly sized local models?
Not necessarily. NVIDIA’s own figures show around 84 tokens per second on the RTX Spark, which is slower than comparable Qwen 3 and Gemma 4 models. The appeal is capability per gigabyte of VRAM, not raw speed.
Where does Nemotron Bolt come from architecturally?
It’s derived from Nemotron 3.5 Super, a 120-billion-parameter model with 12 billion active parameters. NVIDIA pruned it using a method called Star Elastic under a strict memory budget, cut active experts per token from 22 to 9, then distilled and fine-tuned the result before applying quantization-aware distillation.
