NVIDIA Dory1.0-4B: How Looped LLMs Trade Compute for Accuracy
NVIDIA's Dory1.0-4B uses a looped latent-recurrent architecture that trades inference compute for accuracy without retraining.
What is NVIDIA Dory1.0-4B?
NVIDIA Dory1.0-4B is a 4-billion-parameter open-weight language model built on a looped, or latent-recurrent, architecture instead of a standard fixed stack of transformer layers. Rather than running each input through one pass of unique layers, Dory re-applies a shared block of layers multiple times, letting a single checkpoint trade inference compute for accuracy on demand. You pick the loop count (1, 2, or 4 are benchmarked) at serving time, with no retraining or model swap required.
TL;DR
- Dory1.0-4B splits its layer stack into three groups: an Input block, a shared Recurrent block, and an Output block, and runs the Recurrent block multiple times per forward pass.
- Loop count is a runtime knob, not a training-time decision: the same BF16 checkpoint can run at 1, 2, or 4 loops, and NVIDIA recommends 2 loops as the default balance of cost and accuracy.
- Most of the accuracy gain shows up by loop 2, with diminishing but real returns at loop 4, especially on agentic benchmarks like Terminal-Bench and SWE-bench Verified.
- Looping also cuts output tokens, since extra reasoning happens in latent space across loops instead of in longer generated chains of thought.
- Last-loop KV caching keeps memory flat regardless of loop count, solving the usual problem where recurrent models need a separate KV cache per loop.
- The model was pretrained natively in NVFP4, NVIDIA’s 4-bit floating point format, on an nGPT (normalized GPT) backbone, before continuing training in BF16.
- It ships in four formats (BF16, MXFP8, NVFP4, Q5_K_M) with support landing in vLLM and llama.cpp, and benchmarks against Qwen3.5, Qwen3.6, and Gemma competitors show it punching above its 4B weight class on coding and agentic tasks.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
How does the looped architecture actually work?
Dory’s layer stack divides into three named groups. The Input (I) block processes the raw tokens first. The Recurrent (R) block, which holds the bulk of the network (20 of the model’s layers, versus 8 each for Input and Output), then runs multiple times in a loop. Critically, the original input is re-injected into the recurrent block on every pass, not just the first one, so each loop has fresh access to the source context rather than working purely off a decayed internal state. After the final loop, the Output (O) block converts the recurrent representation into token predictions.
This is architecturally different from simply stacking more unique transformer layers. A conventional 4B dense model has a fixed number of distinct layers and a fixed compute cost per token. Dory instead has a smaller set of parameters that get reused across loops, so the parameter count stays constant (4B) while the compute spent per token scales with however many loops you choose to run. The attention layers themselves use a hybrid pattern of interleaved sliding-window attention and full attention, a design increasingly common in efficient long-context models.
Why does loop count matter for accuracy?
Because the Recurrent block processes the same input representation multiple times, each loop gives the model another chance to refine its internal “thinking” before committing to an output. NVIDIA’s benchmark table shows this effect clearly across task categories. On SWE-bench Verified, Dory1.0-4B scores 50.3 at one loop, climbing to 59.5 at two loops and 61.8 at four loops. Terminal-Bench 2.1 shows a similar curve: 12.5, 18.1, 22.2. GDPval-AA-V2 Elo ratings move from 815 to 946 to 967.
The gains aren’t uniform, though. Some benchmarks, like GPQA Diamond and AA-Omniscience, show flat or even slightly negative movement between 2 and 4 loops (72.41 then 72.16 on GPQA Diamond, for instance). NVIDIA’s own guidance reflects this: most of the benefit comes from the jump to two loops, and going to four loops is worth it mainly for compute-heavy agentic work like terminal use and software engineering tasks, where the extra latent computation pays off more directly. This matters for anyone deploying the model: doubling or quadrupling inference compute doesn’t uniformly buy you accuracy, and testing on your actual workload before fixing a loop count is the only way to know if it’s worth the cost.
Is looping just slower inference for the same output?
Not quite, and this is one of the more interesting claims around Dory. Looping adds compute per token, but it can also reduce the total number of tokens the model needs to generate to solve a task. The idea is that extra computation gets spent “thinking” in latent space between loops rather than being written out as longer chains of reasoning text. NVIDIA reports that as loop count increases, the model solves the same benchmark tasks with fewer generated output tokens.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
That creates a real cost tradeoff rather than a strict latency penalty: more compute per forward pass, but potentially fewer passes needed overall. Whether that nets out faster or slower, and cheaper or more expensive, depends on the task and the serving stack, which is why NVIDIA explicitly recommends evaluating end-to-end latency and cost per workload rather than assuming more loops is always better.
How does Dory handle KV cache memory as loops increase?
This is the architecture’s other major engineering problem, and NVIDIA built an explicit fix for it. In a naive implementation of looped attention, each loop would need its own KV cache, since the recurrent attention layers process the input anew each pass. That means memory scales linearly with loop count: run four loops and you need four times the cache of running one, which is a serious limitation for serving long-context requests at scale.
Dory supports two caching modes to address this. Per-loop caching stores a separate cache for every loop (the straightforward approach, with memory cost proportional to loop count). Last-loop KV caching instead keeps a single cache per recurrent attention layer, with each loop simply overwriting the previous loop’s entries. The result is that the cache only ever holds the final loop’s keys and values, so memory footprint stays at the one-loop size no matter how many loops you run. NVIDIA reports that accuracy with last-loop caching stays close to per-loop caching, with output lengths nearly unchanged, meaning you can get the loop-count accuracy benefit without paying a memory tax for it. That headroom can go toward longer context windows or more concurrent requests instead.
What makes the training approach notable?
Dory is built on an nGPT (normalized GPT) backbone and was pretrained natively in NVFP4, NVIDIA’s 4-bit floating-point format, before continuing training in BF16. Native low-precision pretraining is itself a nontrivial technical bet: training directly in 4-bit formats is generally harder to stabilize than quantizing a model after the fact, but it can cut training compute and memory substantially if it works. The release of post-training formats in BF16, MXFP8, NVFP4, and a Q5_K_M quantization for llama.cpp suggests NVIDIA is positioning Dory for a range of deployment environments, from full-precision GPU serving down to quantized local inference.
Is Dory1.0-4B worth using over a similarly sized dense model?
The benchmark table NVIDIA published compares Dory1.0-4B against Qwen3.5-4B, Qwen3.5-9B, Gemma 4 26B A4B, and Qwen3.6-35B-A3B. The picture is mixed by category, which is useful information for anyone deciding whether to adopt it. Dory looks strongest in agentic and instruction-following territory: at four loops it beats Qwen3.5-4B and Qwen3.5-9B on SWE-bench Verified (61.8 vs. 54.7 and 61), posts a notably higher τ³-bench Banking score than every listed comparison model including the much larger Qwen3.6-35B-A3B, and leads IFBench outright (79 vs. 53.83 to 77.89 across the field).
It looks weaker on broad knowledge and pure reasoning benchmarks: GPQA Diamond and AA-Omniscience scores trail the larger Qwen and Gemma models by a wide margin, and HLE scores are also behind. That’s a reasonably coherent pattern for a 4B model with elastic compute: looping seems to help most on tasks that reward extended, structured problem-solving (agentic workflows, instruction adherence, long-context retrieval) rather than tasks that mostly require broad factual recall baked into parameters.
Frequently Asked Questions
What does “looped” or “latent-recurrent” mean in Dory1.0-4B?
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
It means a shared block of the model’s layers is applied repeatedly to the same input during a single forward pass, rather than each token passing through a fixed set of unique layers once. The loop count (how many times the block repeats) can be changed at inference time to trade compute for accuracy.
How many loops should I use with Dory1.0-4B?
NVIDIA recommends two loops as a default balance of accuracy and compute cost. One loop minimizes compute per token. Four loops is suggested when accuracy matters more, particularly for agentic tasks like terminal use and software engineering benchmarks, though gains beyond two loops are smaller on many tasks.
Does increasing loop count always make Dory1.0-4B slower?
Not necessarily in total. More loops mean more compute per token, but NVIDIA’s data shows the model can need fewer generated output tokens to solve a task as loop count rises. Net latency and cost depend on the specific workload and should be measured directly.
How does Dory1.0-4B avoid excessive KV cache memory use with more loops?
It supports a “last-loop” KV caching mode where each loop overwrites the same cache slot instead of allocating a new one per loop. This keeps memory usage flat at the one-loop level regardless of loop count, with accuracy reported to stay close to the memory-heavier per-loop caching mode.
What formats and tools support Dory1.0-4B?
The model is released in BF16, MXFP8, NVFP4, and Q5_K_M (GGUF) formats. Support is available through vLLM (pending inclusion in an official release, installable from an open pull request) and llama.cpp, including a build path for Apple Silicon Macs.