How to Run NVIDIA Nemotron-3.5 Bolt 74B-A9B Locally
A practical guide to running NVIDIA's Nemotron-3.5 Bolt 74B-A9B locally with llama.cpp, covering VRAM needs, 2-bit quantization, and real test results.
What is NVIDIA Nemotron-3.5 Bolt 74B-A9B?
Nemotron-3.5 Bolt is NVIDIA’s compressed vision-language model with 74 billion total parameters but only 9 billion active per token, built by distilling down from a larger 120 billion parameter sibling. NVIDIA then quantized the mixture-of-experts layers to 2-bit using quantization-aware training (QAT), producing a model file of roughly 22 GB that can run on a single consumer or workstation GPU with 24 GB to 48 GB of VRAM. It’s a hybrid Mamba-2 and Transformer architecture, handles text, images, and video, and supports context windows up to 256K tokens.
TL;DR
- Nemotron-3.5 Bolt packs 74 billion total parameters into a mixture-of-experts design with only 9 billion active per token, keeping inference fast despite the large parameter count.
- The model uses 2-bit quantization-aware training on its expert layers, shrinking the full model to about 22 GB so it fits on a single 24 GB to 48 GB GPU.
- On a real hardware test, the model consumed close to 30 GB of VRAM when served locally through llama.cpp with speculative decoding enabled.
- It ships with a small drafter model for speculative decoding, which llama.cpp uses to predict and verify tokens faster than standard autoregressive generation.
- Hands-on testing showed strong vision reasoning and scientific reasoning (correctly tracing object locations across a multi-character story, correctly solving a hand-drawn circuit problem) but weak performance on agentic coding and bug-fixing tasks.
- Multilingual translation results were inconsistent, with major languages translating reasonably well and some less common languages producing hallucinated or looping output.
- The model card lists 90 safetensors shards under the
nemotron_h_omniarchitecture, built on NVIDIA’snemotron-post-training-v3andnemotron-pre-training-datasets, and licensed under NVIDIA’s own terms rather than a standard open license.
How do you set up Nemotron-3.5 Bolt locally?
The practical path to running this model is llama.cpp, which has support for NVIDIA’s Nemotron-H architecture and its hybrid Mamba-2/Transformer design. The general workflow looks like this:
- Get llama.cpp built with recent support for Nemotron-H / MoE models. You’ll want a build new enough to handle the hybrid state-space and attention layers plus the omni (vision) components.
- Pull the quantized GGUF-compatible weights. NVIDIA publishes the model (
nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GBon Hugging Face) pre-quantized to 2-bit on the expert layers, which is what makes the ~22 GB footprint possible for a 74B-parameter model. - Enable the drafter model for speculative decoding. Nemotron-3.5 Bolt ships with a small companion drafter model specifically for this purpose. llama.cpp’s speculative decoding implementation uses the drafter to propose multiple tokens ahead, then the main model verifies or discards them in batches, which speeds up generation without changing output quality.
- Launch with GPU offload flags matching your VRAM. In a real test run, the model consumed close to 30 GB of VRAM on a single GPU while serving alongside other local processes, lining up with NVIDIA’s stated target range of 24 GB to 48 GB.
- If you don’t own a GPU with that much VRAM, rent one. Cloud GPU rental services make it straightforward to spin up a 24 GB or higher card for a session without buying hardware.
This setup is accessible to anyone comfortable with command-line tools. There’s no need for multi-GPU clusters or datacenter-grade hardware, which is the entire point of the compression and quantization work NVIDIA did here.
Why does the 2-bit quantization matter?
Quantization-aware training (QAT) is the detail that makes Nemotron-3.5 Bolt interesting rather than just another large model release. Standard post-training quantization takes a model trained in full precision and rounds its weights down afterward, which often degrades accuracy sharply at low bit depths like 2-bit. QAT instead trains the model with the quantization error baked into the training process, so the weights adapt to low precision rather than getting blindsided by it.
The practical result: NVIDIA compressed a 120 billion parameter model down through an intermediate step to 74 billion total parameters, then quantized the mixture-of-experts layers to 2-bit, and still produced a model that benchmarks competitively against its own 35 billion parameter Nemotron sibling on several tasks, according to NVIDIA’s own published comparisons. For anyone trying to run a large MoE model on a single GPU instead of a server rack, this is what makes the difference between “needs 200+ GB of VRAM” and “needs 24 to 48 GB.”
Is Nemotron-3.5 Bolt actually good at coding?
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
Based on hands-on local testing, the honest answer is: not reliably, at least not yet for agentic or bug-fixing work. In one test, the model was pointed at a real full-stack application (a Flask API, Postgres database, Redis, and an Nginx frontend running via Docker Compose) containing a single-line bug: a horse gestation period hardcoded as 330 days instead of the correct ~340. The model was asked to find and fix the bug with no hints.
It failed. The model’s reasoning went in circles, it verified the wrong number as correct, and it ended up modifying unrelated parts of the codebase (a group-by clause and a dashboard component) without fixing the actual issue. A second coding test, asking the model to generate a realistic animated HTML5 canvas scene (a rotating kebab skewer over a gas heater) from a text brief, also came up short on the first pass. Even after being shown a screenshot of its own flawed output and asked to self-correct, the model claimed to have fixed the bug but produced no visible change.
This matters for anyone considering Nemotron-3.5 Bolt for coding assistant or agentic workflows: the aggressive 2-bit compression appears to cost the model some precision in exactly the kind of multi-step, state-tracking reasoning that bug fixing and iterative code generation require.
Where does Nemotron-3.5 Bolt actually perform well?
The same testing found genuine strength in two areas: vision reasoning and scientific/quantitative reasoning.
In a vision reasoning test, the model was given a photo of a cafe table and a complex narrative prompt involving a missing laptop that several characters each believed was in a different location. The model correctly tracked who moved the laptop, who believed what, and where it actually ended up, answering all nine questions correctly, including several designed to be tricky by testing beliefs about other people’s beliefs.
In a separate test, the model was given a hand-drawn circuit diagram containing an intentionally mislabeled resistor, a dangling wire, and a voltmeter reading of 0V with no leads drawn in. The task was to see whether the model would trust the (wrong) labels or reason from the actual wiring. It got all seven numeric answers correct and correctly identified the mislabeled resistors and the dangling wire as carrying no current, rather than taking the diagram’s labels at face value.
Multilingual translation was more of a mixed bag: asked to translate a Turkish newspaper headline into over 80 languages, the model handled major world languages reasonably well but began hallucinating and looping on some less common ones.
Frequently Asked Questions
How much VRAM does Nemotron-3.5 Bolt need?
NVIDIA designed the model to run in 24 GB to 48 GB of VRAM thanks to 2-bit quantization of its expert layers. In practice, a local test run showed VRAM usage close to 30 GB on a single GPU while running through llama.cpp, leaving room for other applications to run alongside it.
What is speculative decoding and why does Nemotron-3.5 Bolt use it?
Speculative decoding uses a small, fast “drafter” model to guess several tokens ahead, which the larger main model then verifies in a batch instead of generating one token at a time. Nemotron-3.5 Bolt ships with its own drafter model specifically for this, and llama.cpp supports running the pair together to speed up inference without sacrificing output quality.
Can I run Nemotron-3.5 Bolt without an NVIDIA GPU with 24GB+ of VRAM?
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
Not practically on consumer hardware below that threshold, since the model’s 22 GB file size needs to fit in VRAM alongside the KV cache and other overhead. Renting a cloud GPU with sufficient VRAM is a common workaround if you don’t own compatible local hardware.
Is Nemotron-3.5 Bolt good for coding tasks?
Based on real-world bug-fixing and code-generation tests, not consistently. The model struggled to locate a known bug in a full-stack application and to correct flawed HTML5 canvas output even after being shown a screenshot of the error, suggesting its 2-bit compression trades off some capability in multi-step coding reasoning.
What makes Nemotron-3.5 Bolt different from a standard large language model?
It combines a Mamba-2 and Transformer hybrid architecture with a mixture-of-experts design (74 billion total parameters, 9 billion active per token), 2-bit quantization-aware training on the expert layers, native vision and video input, and context windows up to 256K tokens, all aimed at making a large model deployable on a single GPU.
