How to Run Nemotron 3.5 Lightning Locally on Your Own GPU
A practical guide to running and fine-tuning NVIDIA's Nemotron 3.5 Lightning MoE model locally, covering hardware needs, NVFP4, and Unsloth.
Drafted with Claude from source material, checked by automated verification, and reviewed before release. How we make these.
What is Nemotron 3.5 Lightning and why run it locally?
Nemotron 3.5 Lightning is NVIDIA’s open-weights mixture-of-experts model, built with 30 billion total parameters but only 3 billion active at inference time. It’s designed for the “grunt work” layer of agentic systems: tool calls, validation, retrieval, formatting, summarization, and classification, the tasks that eat most of an agent’s token budget but don’t need frontier-level reasoning. Because it’s small, fast, and shipped as open weights with training recipes, it’s one of the more practical models to run and fine-tune on consumer GPUs or NVIDIA’s DGX Spark, rather than renting cloud API access.
TL;DR
- Nemotron 3.5 Lightning is a 30B MoE model with only 3B active parameters, distilled from Nemotron 3 Ultra and built specifically for agent execution work, not chat or deep reasoning.
- It uses a hybrid Mamba-transformer architecture plus multi-token prediction and speculative decoding (via D-Flash and a DGX Spark-tuned variant called D-Spark) to hit high throughput.
- NVIDIA ships it in both BF16 and NVFP4 checkpoints, with NVFP4 being the practical choice for most people running it on RTX cards or DGX Spark.
- On a single DGX Spark, one creator measured roughly 71 tokens per second on a single stream, with multi-token prediction accepted about 70% of the time.
- The model is meant to pair with a routing layer like NVIDIA’s open-source SwitchYard, which sends easy tasks to Lightning and escalates harder ones to a bigger model like Kimi K3.
- Fine-tuning is cheap: NVIDIA’s case studies describe partners training it for about one epoch in under three hours for roughly $100, and Unsloth provides scripts to fine-tune it on consumer hardware.
- The Open Model license allows commercial use and output distillation with no attribution required, and NVIDIA released post-training data and recipes alongside the weights.
What hardware do you need to run it?
Because only 3B parameters are active per token despite the 30B total size, Lightning behaves more like a small dense model at inference time in terms of speed, while still needing enough memory to hold all the experts. NVIDIA built and tuned it around its own hardware stack: RTX consumer GPUs and the DGX Spark. If you have an RTX card with enough VRAM to hold the model in your chosen quantization format, or a DGX Spark unit, you’re in the target audience for this release. One creator ran it directly on a single DGX Spark and got about 71 tokens per second in single-stream inference, which is fast enough that a 1,400-token reasoning task finished in around 20 seconds locally, versus roughly 50 seconds for the same task sent to Kimi K3 over an API.
That speed gap is the point. Lightning isn’t trying to out-think larger models. It’s trying to finish routine work before a bigger model would even start.
BF16 vs NVFP4: which quantization should you use?
NVIDIA ships Nemotron 3.5 Lightning in two formats: a full BF16 checkpoint and an NVFP4 quantized version. BF16 preserves more precision and is the safer choice if you’re doing serious fine-tuning work or need maximum output fidelity, but it demands more VRAM and bandwidth. NVFP4 is a 4-bit floating point format tuned specifically for NVIDIA hardware, and it’s the version most people running this on RTX GPUs or DGX Spark will actually want. It cuts memory footprint substantially with a manageable accuracy tradeoff, which matters a lot for a model whose entire value proposition is speed and low cost. If your goal is local inference for agentic grunt work rather than research-grade output quality, NVFP4 is the practical default.
How does the architecture achieve high throughput?
Lightning is built on the same hybrid Mamba-transformer backbone as the rest of the Nemotron 3 family, which helps it handle NVIDIA’s 1 million token context window more efficiently than a pure transformer would. On top of that, NVIDIA baked in a multi-token predictor during continued pre-training of the underlying Nano 3 base model, meaning the model can generate more than one token per forward pass.
It also incorporates speculative decoding through a drafting model called D-Flash, and a variant tuned specifically for DGX Spark called D-Spark. Both trace back to methodology DeepSeek published on using speculative decoding to speed up generation. The combined effect: NVIDIA claims roughly 4x the throughput of similarly sized models from the Qwen family, and on a 10,000-task benchmark called Pinch Bench, Lightning reportedly ran 30 to 35% faster than the similarly sized Qwen 3.6 MoE. On the Artificial Analysis intelligence index, Lightning lands on par with GPT-OSS, but at meaningfully higher throughput, which is the tradeoff the model is explicitly designed around.
Is Nemotron 3.5 Lightning actually good at agentic tasks?
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
It’s good at the specific, narrow things it was built for, and it’s not trying to be anything else. Testing across agentic benchmarks showed it consistently passing tasks like retrying after injected errors, correctly deciding not to call a tool when one isn’t needed, chaining multiple tool calls together, and handling long-running task sequences. It doesn’t produce long chains of thought the way frontier reasoning models do, and it doesn’t need to for this kind of work.
Where it struggles is prompt injection resistance, which is a meaningful gap if you’re exposing it directly to untrusted input. The practical implication: don’t put Lightning at the front of your pipeline as a standalone agent. Put an orchestrator or a more capable model above it to handle edge cases, security-sensitive decisions, and anything requiring real reasoning, and let Lightning handle the repetitive execution layer underneath.
How does fine-tuning Nemotron 3.5 Lightning work?
NVIDIA released full post-training data sets and recipes alongside the weights, which means the fine-tuning path is unusually transparent for a model of this class. You can inspect exactly how NVIDIA post-trained it and adapt the same techniques, whether that’s curriculum learning, on-policy distillation, or reinforcement learning approaches, for your own data.
In practice, NVIDIA’s early case studies show how cheap this can be. CrowdStrike reportedly fine-tuned Lightning to reach accuracy comparable to the larger Nemotron 3 Super model at around one-fifth the cost. Code Rabbit and Base10 trained a version for about one epoch in under three hours, costing roughly $100, and used the result as a routing classifier in production. Unsloth provides scripts specifically built to fine-tune Lightning on consumer-grade GPUs, which is what makes this realistic for individual developers rather than just enterprise teams with cluster access. The general workflow: take a narrow task (validation, retrieval formatting, a specific classification job), gather a modest amount of representative data, and run a short fine-tuning pass rather than trying to retrain the model broadly.
Why pair Nemotron 3.5 Lightning with a router like SwitchYard?
Lightning is designed to work alongside NVIDIA’s SwitchYard, an open-source routing layer built on top of RouteLLM. SwitchYard decides, per task or per session, whether to send a request to a fast, cheap local model like Lightning or escalate to a more capable model like Kimi K3 or Claude Opus over an API. It supports four routing strategies: random splits for A/B testing, an LLM classifier that reads a request and picks a tier, a “stage router” that reads existing signals like failing tool calls without an extra model call, and an escalation mode that starts every task cheap and bumps it up only when a judge model detects real trouble.
The economic case is direct: one comparison showed Opus 4.5 completing about 80% of a task set at roughly $180 in API costs, while routing between Opus and Lightning with SwitchYard achieved similar completion rates at substantially lower spend. The routing decision should generally happen once per task or conversation, not per turn, since re-routing on every message adds cost and breaks prompt caching.
Frequently Asked Questions
What is Nemotron 3.5 Lightning built for?
It’s built for the execution layer of long-running AI agents: tool calls, validation, retrieval, summarization, and classification, not for chat or complex reasoning.
Can I run Nemotron 3.5 Lightning without a DGX Spark?
Yes. NVIDIA designed it to run on RTX consumer GPUs as well, particularly using the NVFP4 quantized checkpoint to reduce memory requirements.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Do I need SwitchYard to use Nemotron 3.5 Lightning?
No, you can run the model standalone, but it’s designed to pair well with a routing layer so that easy tasks go to Lightning and harder ones escalate to a more capable model.
Is Nemotron 3.5 Lightning good for coding agents?
Not on its own. It lacks the reasoning depth for complex coding tasks and struggles with prompt injection resistance, so it works best as a supporting execution model under a more capable orchestrator.
How expensive is it to fine-tune?
Reported case studies describe fine-tuning runs costing around $100 and taking under three hours for a single epoch, using tools like Unsloth on accessible hardware.
