How to Run Raon-1 Mini Locally on a Single RTX 5090
A guide to running KRAFTON's Raon-1 mini preview model locally using NVFP4 quantization on a single 32GB RTX 5090 GPU setup.
What is Raon-1 mini and why does it fit on one GPU?
Raon-1 mini is a 30B-A4B Mixture-of-Experts language model built from scratch by KRAFTON AI, released under Apache 2.0. It has 28.8B total parameters but only activates 4.3B per token (6 of 128 routed experts plus 1 shared expert), which is why it can run fast despite its size. The model ships in an NVFP4 quantized format specifically so its full weight set fits inside the 32GB of VRAM on a single NVIDIA RTX 5090, something that would normally require a multi-GPU setup or a data center card.
TL;DR
- Raon-1 mini is a 28.8B-parameter MoE model (4.3B active per token) trained from scratch by KRAFTON AI and released under Apache 2.0, with strong Korean-language and agentic task performance.
- NVFP4 quantization is the key to single-GPU deployment, compressing the model enough to run entirely on a 32GB RTX 5090 without offloading to system RAM.
- The architecture uses a hybrid attention design, 18 Raon Delta Attention layers plus 7 full attention (Gated MLA) layers, which keeps the KV cache small (about 8 KiB per token in BF16) and helps long-context inference stay memory-efficient.
- It natively supports 262,144 tokens of context, which is unusually long for a model this size and a big part of why memory efficiency matters so much here.
- A DFlash2 draft model is provided alongside the main weights for speculative decoding, which can meaningfully speed up generation on local hardware.
- The model supports tool calling for agentic workflows, including compatibility with Hermes Agent, Pi, and OpenCode, making it usable for more than chat.
- Benchmarks show it competitive with or ahead of similarly-sized MoE models like Qwen3.6-35B-A3B and Gemma 4 26B-A4B on reasoning tasks such as GPQA Diamond and AIME 2026.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
What hardware do you actually need?
The headline requirement is a single RTX 5090 with its full 32GB of VRAM, paired with the NVFP4-quantized build of Raon-1 mini. NVFP4 is a 4-bit floating point format designed for NVIDIA’s newer tensor cores, and it’s what makes the difference between “needs a multi-GPU server” and “runs on a desktop card.” The model card also lists an MXFP8 option, which uses more memory and is better suited to higher-VRAM accelerators rather than consumer cards.
Beyond the GPU, you’ll want enough system RAM to load and convert checkpoints comfortably (32GB or more is a safe baseline for a model this size), fast NVMe storage for the weight files, and a reasonably current CUDA driver stack since NVFP4 support depends on newer NVIDIA tensor core generations. The RTX 5090’s 32GB VRAM ceiling is the hard constraint here: drop to a 24GB card and you’re likely looking at offloading some experts to CPU RAM, which will cost you throughput.
How does NVFP4 quantization make this possible?
NVFP4 is a 4-bit quantization scheme that shrinks model weights to a quarter of their typical 16-bit footprint, with careful scaling to limit accuracy loss. For a MoE model like Raon-1 mini, this matters more than for a dense model because all 28.8B total parameters need to be resident in memory even though only 4.3B are active for any given token. You don’t get to skip loading the inactive experts, since routing decisions change per token and per layer.
By quantizing all weights to NVFP4, KRAFTON brings the effective memory footprint down into the range a 32GB consumer card can hold, including room for activations and the KV cache during inference. The hybrid attention design helps here too. Because 18 of the 25 layers use Raon Delta Attention with a fixed-size state rather than a growing KV cache, memory doesn’t balloon as context length increases the way it would in a conventional transformer. Only the 7 Gated MLA layers maintain a traditional KV cache, and KRAFTON cites a compact ~8 KiB per token in BF16 for that cache, which keeps long-context inference (up to 262,144 tokens) from blowing past the 5090’s VRAM budget.
What’s the architecture doing under the hood?
Raon-1 mini’s design mixes a few techniques aimed at efficiency rather than raw scale. It uses what KRAFTON calls Raon Delta Attention (RDA), a variant that decouples the erase and write gates found in Kimi Delta Attention, intended to improve long-context handling. It also uses Raon-mHC, a cheaper version of DeepSeek’s manifold-constrained hyper-connections, aimed at reducing training cost rather than inference cost directly.
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
The layer layout is 18 RDA layers to 7 full-attention Gated MLA layers, a roughly 3:1 ratio. Routing sends each token through 6 of 128 available experts plus a shared expert, with an expert intermediate size of 1,088. The model has no explicit positional encoding (NoPE), a large 259,840-token vocabulary, and an untied LM head. None of this is unusual in isolation, but combined it’s clearly engineered to keep the active compute and memory path small while preserving a large total parameter pool for capability.
Is Raon-1 mini worth running locally instead of using an API?
For developers already set up to run local inference, Raon-1 mini offers a few concrete advantages. It’s Apache 2.0 licensed, so there are no usage restrictions blocking commercial deployment. Benchmark numbers put it in the same tier as other MoE models in its size class: on GPQA Diamond it scores 84.3, ahead of Qwen3.6-35B-A3B’s 83.2 and Gemma 4 26B-A4B’s 80.7, and on AIME 2026 it scores 93.3, again ahead of both comparison MoE models. On MMLU Pro it trails those two models (76.5 versus 85.9 and 82.9), so it’s not uniformly stronger, but reasoning and math benchmarks are where it’s being positioned to compete.
The multilingual training set (Korean, English, Japanese, Chinese, Spanish, French, Thai) with particular strength in Korean makes it a notable option for anyone building Korean-language applications, a use case not well served by most Western open-weight models. Combined with tool-calling support for agentic frameworks like Hermes Agent, Pi, and OpenCode, it’s positioned as a model you can actually wire into an agent pipeline rather than just chat with.
The tradeoff is that this is explicitly a preview release. KRAFTON’s own documentation states specs and benchmark results may change before a final version ships, so treat current numbers as provisional rather than final.
How do you get it running?
The practical path is to pull the NVFP4-quantized weights from KRAFTON’s Hugging Face repository and load them with an inference stack that supports NVFP4 on NVIDIA’s current tensor core architecture. Because the model includes a DFlash2 draft model for speculative decoding, pairing the two during inference setup can improve generation speed, which matters more on a single consumer GPU than on a multi-GPU server with headroom to spare.
Given the long context window, pay attention to how your inference framework handles the hybrid attention layout. The fixed-state RDA layers and the small KV cache on the Gated MLA layers behave differently from a standard transformer’s KV cache, so not every serving framework will handle the architecture optimally out of the box. Check for explicit support for the hybrid attention design before assuming a generic transformer loader will give you full performance.
If you don’t have a 5090 available, KRAFTON also runs a hosted playground for trying the model without local setup, useful for evaluating whether it fits your use case before committing to a local deployment.
Frequently Asked Questions
What GPU do I need to run Raon-1 mini locally?
A single NVIDIA RTX 5090 with its full 32GB of VRAM, using the NVFP4-quantized version of the model. The standard or MXFP8 versions require more memory and aren’t intended for single consumer-GPU deployment.
How big is Raon-1 mini?
It has 28.8B total parameters as a Mixture-of-Experts model, but only 4.3B are active per token, since just 6 of 128 routed experts plus one shared expert fire on any given forward pass.
What is NVFP4 and why does it matter here?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
NVFP4 is a 4-bit floating point quantization format built for recent NVIDIA tensor cores. It reduces Raon-1 mini’s memory footprint enough to fit entirely within a 32GB RTX 5090, which wouldn’t be possible at higher precision formats like BF16 or even MXFP8.
Can Raon-1 mini handle long documents or conversations?
Yes, it natively supports up to 262,144 tokens of context. Its hybrid attention design, mostly fixed-state Raon Delta Attention layers with a small number of traditional KV-cache attention layers, keeps memory use manageable even at long context lengths.
Is Raon-1 mini free to use commercially?
Yes, it’s released under the Apache 2.0 license, which permits commercial use without the restrictions found in some other open-weight model licenses. Keep in mind it’s labeled a preview release, so the final version may differ.