NVIDIA Dory: Running the Looped Transformer Model Locally
NVIDIA's Dory reuses the same 4B weights multiple times per token. Here's how looped inference works and how to run it yourself.
What is NVIDIA Dory?
Dory is a 3.97 billion parameter open-weight language model from NVIDIA’s Nemotron Labs that reruns a block of its own layers multiple times per token instead of adding more parameters. NVIDIA calls it a looped transformer, or latent recurrent transformer. The same set of weights gets reused one, two, four, or more times depending on how much compute you want to spend on a given answer, and the model’s memory footprint never changes regardless of how many loops you choose.
TL;DR
- Looped inference means Dory splits its layers into an input block, a recurrent block, and an output block, and only the recurrent block (40 of its layers) repeats, effectively deepening the network without adding weights.
- Elastic inference compute lets you pick the loop count at inference time: one loop behaves like a fast, shallow model, while four or more loops behaves like a much deeper model, all using the same 3.97B parameters on the GPU.
- Quality gains are task-dependent, with NVIDIA’s composite benchmark showing Dory improving from winning 1 of 10 comparisons against Gemma 3 27B at one loop to 5 of 10 at four loops, especially on agentic and instruction-following tasks.
- Knowledge-heavy benchmarks don’t improve with looping, because looping adds compute depth, not stored knowledge, so Dory still trails larger mixture-of-experts models on tasks like GPQA.
- The model shipped in multiple formats, including BF16, MXFP8, and NVFP4, plus community GGUF conversions, and was pretrained on 6 trillion tokens, with the first 4.5 trillion run natively in NVFP4.
- KV cache size scales with loop count unless you use NVIDIA’s “last loop” caching mode, which keeps memory nearly flat across loop settings at a small accuracy cost.
- Serving it today requires workarounds, since inference engines like vLLM currently need the loop count fixed at startup, so running multiple loop depths side by side means running multiple instances.
How does a looped transformer actually work?
A standard transformer runs every layer exactly once, start to finish. Dory instead organizes its layers into three sections: a 16-layer input block, a 40-layer recurrent block, and a 16-layer output block. The input and output blocks each run once per token, same as always. The recurrent block is where things change: instead of running once, it runs in a loop, and you choose how many times.
Each pass through the recurrent block doesn’t just feed its own output back into itself blindly. The original input representation gets re-added at each iteration, similar to how residual connections work in ResNet-style architectures. The practical effect is that the model gets “reminded” of the original token context on every lap, refining its internal representation rather than drifting off on tangents. One useful way to think about it: each loop is a bit like rereading the prompt before taking another pass at the answer.
If you unroll the loops, a four-loop pass is doing the computational work of a much deeper network, roughly equivalent to a 9B dense model’s compute, while still only holding 3.97B parameters in GPU memory. One loop gives you about 72 effective layers of depth, two loops push that to 112, and four loops reach around 192, all from the same weights.
Critically, the output block was trained to interpret the recurrent block’s state after any number of loops, not a fixed number. That means one loop, two loops, or eight loops are all valid configurations of the same checkpoint. You don’t need separate fine-tunes for each depth. NVIDIA refers to this flexibility as elastic inference compute: you decide the compute-per-token tradeoff at inference time, not at training time.
What are the memory and hardware tradeoffs?
The headline benefit is that looping adds compute depth without adding parameters. A GPU that can hold 3.97 billion parameters can run Dory at one loop or at eight loops with the exact same weight memory footprint. That’s a meaningful departure from how most people scale quality today, which is either bigger dense models or mixture-of-experts models with more total parameters.
The catch is the KV cache. In the default “per-loop” mode, every loop through the recurrent block keeps its own cache, so more loops means a larger KV cache alongside the (unchanged) weight memory. NVIDIA also built a “last loop” mode, where each recurring attention layer keeps a single cache and simply overwrites the previous loop’s values rather than stacking them. In testing referenced by the model’s documentation, at a 32K context window, two-loop generation needs roughly 2GB per sequence for the KV cache in per-loop mode, versus about 1.3GB using the last-loop approach, with accuracy and output length reported to stay close to the per-loop version. For anyone serving long-context workloads with many concurrent users, last-loop mode is the setting to test first.
What quantization and deployment formats are available?
- Patient records✓ Granted
- Surgery schedule✓ Granted
- Billing export✕ Denied
Dory was released under NVIDIA’s open model license (allowing commercial use) in several formats: a BF16 version, an MXFP8 version, and an NVFP4 version, alongside community GGUF conversions for local inference tools. Notably, pretraining itself used NVFP4 natively for the first 4.5 trillion of its 6 trillion training tokens, before switching to 16-bit precision for the remaining pretraining, a 300 billion token context-extension phase, and finally supervised fine-tuning and reinforcement learning (GRPO).
The model supports context windows up to 256K tokens. At 3.97B parameters, it’s small enough to run on consumer and prosumer GPUs in quantized form, which is part of why it’s practical to test loop behavior locally rather than only reading about it in a paper.
One current limitation: popular serving engines like vLLM need the loop count fixed when the server starts. There’s no built-in way yet to dynamically route a given request to a different loop depth within a single running instance. Running one-loop, two-loop, and four-loop variants side by side today means launching separate instances of the same model, each pinned to a different loop setting.
Is Dory worth running, and how does it benchmark?
On NVIDIA’s own composite benchmark comparing Dory to Gemma 3 27B, loop count directly tracks with benchmark wins: Dory at one loop wins 1 of 10 benchmark categories, climbs to 4 of 10 at two loops, and reaches 5 of 10 at four loops, putting the four-loop configuration ahead of Gemma 3 27B on NVIDIA’s composite score.
The gains aren’t uniform across task types. Dory does well on agentic work: instruction following, long-context handling, and benchmarks like SWE-bench Verified improve noticeably with more loops. It does not do well on knowledge-heavy benchmarks like GPQA or Humanity’s Last Exam, where it performs considerably worse than Gemma 3 27B regardless of loop count. That’s consistent with what looping actually buys you: more compute depth per token, not more stored knowledge. Gemma 3 27B is a mixture-of-experts model with about 4 billion active parameters per token, similar compute to Dory, but it has many more total parameters available as experts to store factual knowledge. Looping can make a model reason harder over what it knows; it can’t give the model more to know.
For anyone considering deployment, a sensible default echoed by NVIDIA’s own guidance is to run two loops as a baseline, then route harder agentic or multi-step tasks to a four-loop (or higher) configuration when the extra compute is worth the latency cost.
Why does this matter beyond one model?
Looped or recurrent transformers represent a different axis for scaling model quality: instead of generating longer chains of thought (more output tokens) or adding more parameters (bigger models), you spend more compute per token on the same input, with the loop count chosen at inference time. That’s a meaningful lever because it’s adjustable per request rather than baked in at training time. There have been persistent claims that upcoming frontier models from other labs use some form of looped or recurrent computation internally, though those claims aren’t independently confirmed. Dory matters regardless, because it’s one of the first looped architectures shipped as downloadable open weights rather than described only in a research paper.
Frequently Asked Questions
What does “looped transformer” mean in Dory?
It means a block of the model’s layers (40 layers in Dory’s case) runs multiple times per token instead of once, with the output block trained to interpret the result after any number of loops, so one checkpoint supports multiple compute depths.
Does more looping always produce better answers?
No. More loops generally improve agentic, instruction-following, and long-context tasks, but knowledge-heavy tasks like GPQA don’t improve with looping, since looping adds reasoning depth, not stored facts.
How much GPU memory does Dory need?
The weights stay fixed at roughly 3.97 billion parameters regardless of loop count, available in BF16, MXFP8, NVFP4, and GGUF formats. The variable cost is the KV cache, which grows with loop count in per-loop mode but stays close to flat in NVIDIA’s last-loop caching mode.
Can I run different loop counts at the same time?
Not within a single running instance of common engines like vLLM today, since the loop count is set at startup. Running multiple loop depths simultaneously currently means launching separate model instances, each configured with its own loop setting.
Is Dory meant for production use?
NVIDIA frames it primarily as a research release demonstrating elastic inference compute. It’s usable and commercially licensed, but its main value right now is showing how looped inference trades compute for quality on a fixed parameter budget rather than being a polished production deployment.
