How to Run NVIDIA Dory1.0-4B Locally with vLLM or llama.cpp
Step-by-step guide to running NVIDIA's looped Dory1.0-4B model locally, covering GGUF downloads, loop counts, and KV cache tuning.
What is NVIDIA Dory1.0-4B and why does it run differently from other local LLMs?
NVIDIA Dory1.0-4B is a 4-billion-parameter open-weight model built on a looped, latent-recurrent architecture rather than a standard transformer stack. Instead of running each token through a fixed set of layers once, Dory repeats a shared “recurrent” block of layers multiple times per forward pass, controlled by a setting called the loop count. Running it locally means you’re not just loading weights, you’re also choosing how much extra compute the model spends per token and how its KV cache behaves, both of which are configurable at inference time without retraining.
TL;DR
- Dory1.0-4B uses a looped architecture with an 8-layer input block, a 20-layer recurrent block applied multiple times, and an 8-layer output block, instead of a flat transformer stack.
- Loop count trades compute for accuracy on the same checkpoint, so you can run 1, 2, 3, or 4 loops without downloading a different model or retraining anything.
- Two loops is the recommended default, since most of the accuracy improvement from looping shows up by the second pass, with smaller gains at three or four loops.
- Last-loop KV caching keeps memory flat as loop count increases, storing only the final loop’s keys and values instead of a separate cache per loop, which matters a lot for long-context or high-concurrency setups.
- vLLM support requires an unmerged pull request (vllm-project/vllm#60149) as of this writing, so you install from that branch rather than a stable release.
- llama.cpp support also ships via a pull request (ggml-org/llama.cpp#30306), and the reference setup pins a specific commit rather than tracking the moving PR head.
- The model is available in BF16, MXFP8, NVFP4, and Q5_K_M (GGUF) formats, with the GGUF export suited to local CPU/GPU inference via llama.cpp and the other formats aimed at server deployment.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
How do you set up Dory1.0-4B with vLLM?
vLLM support for Dory is not yet in an official release. The model card instructs installing vLLM directly from the pull request branch:
VLLM_USE_PRECOMPILED=1 pip install "git+https://github.com/vllm-project/vllm.git@refs/pull/60149/head"The VLLM_USE_PRECOMPILED=1 flag pulls prebuilt kernels instead of compiling CUDA locally, which saves significant setup time. Once installed, you launch the server pointing at the BF16 checkpoint on Hugging Face:
vllm serve --model nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 \
--hf-overrides '{"n_recurrent_loops": 2, "recurrent_kv_cache_mode": "per_loop"}' \
--max-model-len 262144 \
--max-num-seqs 128 \
--enable-prefix-caching \
--async-scheduling \
--reasoning-parser nemotron_v3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choiceThe two settings that matter most here live inside --hf-overrides: n_recurrent_loops controls how many times the recurrent block runs, and recurrent_kv_cache_mode controls how the KV cache is stored across loops. The model has been validated up to 256K tokens of context, so --max-model-len can be reduced if you’re short on VRAM or want more cache headroom at higher concurrency.
How do you set up Dory1.0-4B with llama.cpp?
llama.cpp support also comes from a pull request (ggml-org/llama.cpp#30306) rather than a tagged release. Because the PR branch moves, the documented setup pins a specific commit (98a74c97ea09ac72f6ceda1559808fc0a2f447ce) instead of fetching the latest head, which is worth doing yourself if you want reproducible behavior.
On Apple Silicon, after installing Homebrew and Xcode command line tools, the build looks like this:
brew install cmake openssl@3
git init llama.cpp-dory
cd llama.cpp-dory
git remote add origin https://github.com/brianchmielNV/llama.cpp.git
git fetch --depth 1 origin 98a74c97ea09ac72f6ceda1559808fc0a2f447ce
git checkout --detach FETCH_HEAD
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_OSX_ARCHITECTURES=arm64 \
-DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON \
-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Apple \
-DLLAMA_BUILD_SERVER=ON \
-DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_APP=OFF \
-DLLAMA_USE_PREBUILT_UI=ON \
-DLLAMA_OPENSSL=ON -DOPENSSL_ROOT_DIR="$(brew --prefix openssl@3)"
cmake --build build --parallel 4 --target llama-serverThis compiles with Metal acceleration and a prebuilt browser UI, so there’s no separate JavaScript build step or Python environment needed just to run the server.
Where do you get the GGUF file and how do you verify it?
llama.cpp doesn’t load the BF16 Hugging Face checkpoint directly, it needs a GGUF export. NVIDIA publishes a quantized dory-q5_k_m.gguf file (Q5_K_M quantization) on Hugging Face under the GGUF repo for the model. After downloading it into your llama.cpp-dory/ directory, it’s worth checking the file size (3,673,288,864 bytes expected) and SHA-256 checksum before launching:
echo "96080be49ffd67b916018b7bdef8c484762576fb52780e10ab98ac4978a660e0 dory-q5_k_m.gguf" | shasum -a 256 -c -Keep in mind this is a quantized export. The benchmark numbers NVIDIA publishes are measured on the BF16 version, so expect some accuracy drift with Q5_K_M, as with any quantized model.
Once the file is in place, start the server:
./build/bin/llama-server \
-m ./dory-q5_k_m.gguf \
--alias dory \
--host 127.0.0.1 --port 8080 \
-ngl 999 \
--ctx-size 4096 --parallel 1 \
--batch-size 512 --ubatch-size 512 \
--cache-type-k f16 --cache-type-v f16 \
--flash-attn off \
--no-context-shift \
--jinja --reasoning on \
--temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 --repeat-penalty 1 \
--override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop"Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
The chat UI is then available at http://127.0.0.1:8080, and the OpenAI-compatible API sits at http://127.0.0.1:8080/v1 with the model name dory (set via --alias). This server is local-only and unauthenticated by default, so don’t expose the port to the open internet without adding your own auth layer.
How should you choose the loop count?
Loop count is the main lever for trading inference compute against accuracy on this single checkpoint. NVIDIA’s own benchmark table shows the pattern clearly: on SWE-bench Verified, Dory scores 50.3 at one loop, jumps to 59.5 at two loops, and reaches 61.8 at four loops. On Terminal-Bench 2.1, it goes from 12.5 to 18.1 to 22.2 across one, two, and four loops. The jump from one to two loops is consistently the largest, with diminishing returns after that.
NVIDIA recommends two loops as the default starting point. Use one loop when minimizing per-token compute is the priority, for example in latency-sensitive or high-throughput settings. Consider four loops for agentic or coding-heavy workloads like Terminal-Bench or SWE-bench style tasks, where the accuracy gain is more pronounced. Because higher loop counts can also reduce the number of generated tokens needed to solve a task (the model “thinks” more per token instead of writing longer outputs), it’s worth measuring end-to-end latency and cost on your actual workload rather than assuming more loops always costs more overall.
In vLLM, loop count is set via n_recurrent_loops in --hf-overrides. In llama.cpp, it’s set via dory.num_loops in --override-kv, accepting values of 1, 2, 3, or 4.
What does KV cache mode change, and which one should you use?
Looped models have a memory problem that standard transformers don’t: if you naively cache keys and values separately for every loop, memory grows linearly with loop count even though the layer weights are shared. Dory addresses this with two cache modes.
Per-loop caching (per_loop) stores a distinct KV cache for each recurrent pass. This is the default in both the vLLM and llama.cpp examples above, and memory overhead scales with however many loops you configure.
Last-loop caching (last_loop) keeps only one cache per recurrent attention layer, with each loop overwriting the previous loop’s entries. The result is that Dory’s KV cache footprint stays at the one-loop size regardless of how many loops you run, which frees up memory for longer context windows or more concurrent requests. According to NVIDIA, accuracy with last-loop caching stays close to per-loop mode, with output lengths largely unchanged.
If you’re running on limited VRAM or want to push context length and concurrency, last-loop caching is the mode to enable. Switch it by setting recurrent_kv_cache_mode to "last_loop" in vLLM’s --hf-overrides, or dory.recurrent_kv_cache_mode=str:last_loop in llama.cpp’s --override-kv.
Is Dory1.0-4B worth running locally compared to similarly sized models?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
On NVIDIA’s reported benchmarks, Dory1.0-4B at two or four loops outperforms Qwen3.5-4B on several tasks, including SWE-bench Verified (61.8 vs 54.7 at four loops), Terminal-Bench 2.1 (22.2 vs 15), IFBench (79 vs 53.83), and AA-LCR (73 vs 64.7). It’s less competitive on broad knowledge and some reasoning benchmarks, trailing Qwen3.5-4B and larger models like Qwen3.5-9B and Qwen3.6-35B-A3B on GPQA Diamond, HLE, and AA-Omniscience. The appeal for local deployment isn’t that it’s the strongest 4B model across the board, it’s that a single checkpoint lets you dial compute up or down per task, and the last-loop caching option means that extra reasoning compute doesn’t have to cost you memory headroom.
Frequently Asked Questions
Is official vLLM or llama.cpp support available for Dory1.0-4B yet?
Not in a tagged release as of this writing. Both integrations exist as open pull requests (vllm-project/vllm#60149 and ggml-org/llama.cpp#30306), so you need to install vLLM from the PR branch or build llama.cpp from the Dory-enabled fork.
What’s the difference between per-loop and last-loop KV caching?
Per-loop caching stores a separate KV cache for every recurrent pass, so memory grows with loop count. Last-loop caching keeps only the most recent loop’s cache per layer, holding memory flat regardless of how many loops you run, with accuracy close to per-loop mode.
How many loops should I use for coding or agentic tasks?
NVIDIA’s benchmarks show the biggest gains on agentic tasks like SWE-bench and Terminal-Bench come from going beyond two loops, so four loops is worth testing if accuracy matters more than raw speed on those workloads.
Can I run the quantized GGUF version and expect the same accuracy as BF16?
Not exactly. The published benchmark scores are measured on the BF16 checkpoint. The Q5_K_M GGUF export used by llama.cpp is quantized, so some accuracy degradation versus BF16 should be expected, consistent with other quantized LLMs.
What context length does Dory1.0-4B support?
The model card lists a validated context length of up to 256K tokens. Local setups can lower --max-model-len (vLLM) or --ctx-size (llama.cpp) if VRAM is limited or more KV-cache headroom is needed for concurrent requests.