How NVIDIA Compressed a 74B Model Into 24GB at 2-Bit
NVIDIA's Nemotron Bolt uses pruning, distillation, and quantization-aware training to fit a 74B model into 24GB at 2-bit. Here's the recipe.
What is NVIDIA’s Nemotron Bolt model and why does it matter?
Nemotron 3.5 Bolt 74B A9B is a mixture-of-experts model from NVIDIA with 74 billion total parameters and roughly 9 billion active per token, compressed so that its expert weights run at 2-bit precision. The whole model fits inside 24GB of memory, the footprint of a single consumer-grade GPU like those in NVIDIA’s RTX Spark laptops. What makes it notable isn’t just the file size. It’s that NVIDIA got there through a multi-stage pipeline of pruning, distillation, and quantization-aware training, rather than simply rounding an existing model down after the fact.
TL;DR
- NVIDIA built Bolt by shrinking Nemotron 3.5 Super, a 120 billion parameter model with 12 billion active parameters, down through pruning and distillation before quantizing anything.
- The 2-bit figure only applies to the expert weights, since the attention/MoE routing layers run at NVFP4 and the embeddings and output layer sit at 8-bit, making Bolt a mixed-precision model rather than uniformly 2-bit.
- Quantization-aware training (QAT) lets the model adapt to low-bit rounding during training instead of having precision stripped away afterward, which is why Bolt holds up better than a typical post-training 2-bit quant.
- NVIDIA used a variant called quantization-aware distillation (QAD), training the quantized model as a student against the unquantized, fully post-trained version as the teacher.
- Benchmark drops from the full 16-bit model to the 2-bit version were modest, roughly 90 to 88 on one benchmark and 89 to 86 on GPQA Diamond, a small cost for cutting file size from around 148GB to about 22GB.
- Running locally is bottlenecked by memory bandwidth, not compute, so fewer bits per weight directly translates into faster token generation at batch size one.
- Bolt isn’t the fastest model in its class, clocking around 84 tokens per second on the RTX Spark, behind some competing models, which makes it a smarter-per-gigabyte model rather than a faster-per-gigabyte one.
Why does post-training quantization break down below 4-bit?
The standard way to shrink a model is post-training quantization: train in 16-bit, then round every weight down afterward using some calibration data to minimize the damage. This works cleanly at 8-bit and is usually close to free at 4-bit, which is why 4-bit became the unofficial floor for running models locally.
Below that floor, things change fast. The transition between 3-bit and 2-bit is where quality typically falls off a cliff. Above 3-bit, a quantized model generally stays close to its original behavior. At 2-bit and below, the internal representations shift so much that the model is effectively being asked to operate as a different network, one it never got a chance to adapt to. Rounding after the fact simply doesn’t give it that chance, which is why community 2-bit and 1-bit quantizations tend to be rough.
Research from Meta (the ParetoQ paper) found a workaround: if a model is trained properly at low bit-widths rather than rounded afterward, a 2-bit or 3-bit model can actually beat a post-training-quantized 4-bit model at the same memory budget. In other words, a bigger model with fewer bits per weight can outperform a smaller model with more bits per weight, provided the low-bit training is done right.
How did NVIDIA actually build Bolt?
The pipeline has three stages, and quantization is only the last one.
Stage one: pruning. NVIDIA started with Nemotron 3.5 Super VL, a 120 billion parameter model with 12 billion active parameters. They ran an architecture search process (referred to as “star elastic” in the source material) that works like an automated search under a memory budget: a smaller router learns, layer by layer, how many experts to keep and how wide each mixture-of-experts block should be. The number of active experts per token was cut from 22 down to 9. Critically, the pruning target wasn’t an arbitrary parameter count, it was a specific memory budget: fit under 24GB at batch size one with a 128K context window, with that budget already assuming the experts would eventually land at 2-bit. The attention layers stayed at 16-bit throughout.
Stage two: distillation and post-training. The pruned model was distilled from its larger parent using around 120 billion tokens, mostly at shorter context lengths (roughly 8K) for speed, with later stages extending to the full 128K context to recover long-context ability. After distillation, NVIDIA reattached the vision encoder, ran additional supervised fine-tuning, merged checkpoints, and ran several rounds of reinforcement learning for agentic tasks. At this point the model was still sitting in bfloat16.
Stage three: quantization-aware training. Rather than rounding the finished bfloat16 model down to 2-bit, NVIDIA trained the model to live on the low-bit grid directly. During QAT, low-bit rounding is simulated on every forward pass while a full-precision copy of the weights is kept around for gradient updates. The model learns to perform well under the constraint instead of being shocked by it after the fact.
NVIDIA’s specific approach is called quantization-aware distillation (QAD): the quantized model acts as the student, and the final, fully post-trained bfloat16 model acts as the teacher. This means all the benefits of supervised fine-tuning, RL, and checkpoint merging carry over, and the quantization step can be rerun without needing the full original training dataset.
Is quantization-aware training actually common, or is this unique to NVIDIA?
QAT is becoming a standard move among major labs, not a one-off trick. Google has shipped QAT checkpoints for Gemma models, including Gemma 3 and the full Gemma 4 family down to the smallest sizes, with the lightest versions pushing token-generation layers to 2-bit while keeping core reasoning layers at higher precision. Google’s smallest Gemma 4 model dropped below a gigabyte using a similar mixed-precision idea as Bolt, just applied to phone-scale models instead of GPU-scale ones.
OpenAI used similar techniques for the GPT-OSS models, part of why the 120B version fits on a single 80GB GPU. Moonshot has applied comparable methods to its Kimi models. The pattern across labs is the same: instead of leaving quantization to the community after release, labs are building it into the training pipeline so that the official low-bit versions are the good ones.
What other compression techniques are worth knowing about?
Two adjacent techniques show up frequently in this space. The first is expert pruning for mixture-of-experts models, where entire experts are dropped rather than shrinking every weight. Cerebras’s REAP method is one example, and it found that outright removing experts works better than merging them, since merging damages the router’s ability to select the right expert. Which experts get cut depends heavily on the calibration data used, so a model pruned on code data will stay strong at coding but may lose ground elsewhere.
The second is extreme low-bit training from scratch, such as Microsoft’s BitNet approach, where weights are trained from the first step to take only the values -1, 0, or 1 (a true 1.58-bit representation). It shows promise, but current GPUs lack proper hardware support for it, which limits practical deployment. That gap is part of why NVIDIA’s approach, training large, then pruning, distilling, and quantizing down, looks like the more realistic near-term path for running big models on consumer hardware.
What’s the actual cost of compressing Nemotron down to 2-bit?
NVIDIA’s own benchmark numbers show a real but modest quality cost. Going from the bfloat16 version to the 2-bit file, scores dropped from around 90 to 88 on one benchmark and from about 89 to 86 on GPQA Diamond. Those are measurable hits, but they come alongside shrinking the model from roughly 148GB down to about 22GB, a file size reduction of nearly 7x. On that trade-off, a few points of benchmark score looks like a reasonable price.
It’s also worth being clear about what “2-bit” means here. Only the expert weights in the mixture-of-experts layers run at 2-bit. The attention/routing layers use NVFP4 (4-bit), and the embeddings and output layer run at 8-bit. Bolt is a mixed-precision model, with each component quantized at the level that makes sense for it, not a uniformly 2-bit model.
Frequently Asked Questions
What is Nemotron 3.5 Bolt 74B A9B?
Without shadow IT.
It’s an NVIDIA mixture-of-experts model with 74 billion total parameters and about 9 billion active per token, built by pruning and distilling the larger Nemotron 3.5 Super model and then quantizing it so the expert weights run at 2-bit, fitting the whole model in 24GB.
Why does 2-bit quantization usually hurt model quality so much?
Below roughly 3-bit precision, a model’s internal weight representations have to change dramatically to fit the available range. Simply rounding a trained model’s weights down afterward doesn’t give it a chance to adapt to that shift, which is why standard post-training 2-bit quantization tends to degrade quality sharply.
What is quantization-aware training (QAT)?
QAT simulates low-bit rounding during the training process itself, on every forward pass, while keeping a full-precision copy of the weights for gradient updates. This lets the model learn to perform well under the low-bit constraint rather than having that constraint imposed after training is finished.
Is Nemotron Bolt actually the whole model at 2-bit?
No. Only the mixture-of-experts weights are quantized to 2-bit. The attention and routing layers use NVFP4 (4-bit), and the embeddings and output layer are kept at 8-bit, making it a mixed-precision model rather than a uniform 2-bit one.
Is Bolt faster than other models of similar size?
Not necessarily. NVIDIA’s own figures put it at around 84 tokens per second on the RTX Spark, behind some comparable models. The benefit of Bolt is memory footprint and quality retention at that footprint, not raw generation speed.
