Edge0-145B NVFP4: A 145B Agent Model on One RTX 5090
Edge0-145B is a 145B-parameter sparse MoE model quantized to NVFP4, built to run locally on a single 24GB consumer GPU for agent tasks.
What is Edge0-145B?
Edge0-145B is a 145-billion-parameter sparse mixture-of-experts (MoE) model released in NVFP4 format, designed to run on a single consumer GPU with 24GB of VRAM, such as an RTX 5090. It’s published by Edge0 on Hugging Face, built on a “latent MoE” architecture derived from Qwen3.8-Flash-Next, and positioned for agent and coding workloads rather than general chat. The headline claim is straightforward: a model with parameter counts normally reserved for multi-GPU server racks, squeezed down to fit on a desktop card.
TL;DR
- Edge0-145B packs 145 billion parameters into a footprint that targets a single 24GB consumer GPU, using 4-bit quantization rather than the 16-bit or 8-bit precision typical of large open models.
- The model uses NVFP4, an FP4 format geared toward newer NVIDIA hardware, as opposed to older integer-based 4-bit quantization schemes like GPTQ or AWQ.
- It’s built on a sparse MoE design with a “latent MoE” twist, meaning only a fraction of its total parameters activate for any given token, which is what makes serving a 145B model on one GPU plausible at all.
- The base model is Qwen3.8-Flash-Next, and Edge0 applies its own quantization and architectural packaging on top of that base.
- The release ships as a full inference stack, not just weights, including a GGUF skeleton, expert and dense binaries, a tokenizer, a start script, and CUDA-oriented tooling, under its own
edge0library rather than standardtransformersloading. - The repo also lists smaller Edge0-120B variants in NVFP4 and Q3_K quantization, suggesting a family of models trading size and precision against hardware requirements.
- Tags point to coding and agent use cases, not conversational chat as the primary target, which lines up with the “think big, run here” framing of a locally hosted agent model.
Big news
firstNov 17
How does a 145B model fit on a 24GB GPU?
The short answer: it doesn’t fit as a dense model, and it isn’t meant to. Two things make it possible.
First is sparsity. Edge0-145B is a mixture-of-experts model, which means the 145 billion parameters are split across many expert sub-networks, and only a subset of experts activate per token. This is the same basic idea behind MoE models like Mixtral or DeepSeek’s MoE line: the total parameter count is large, but the active compute and memory footprint per forward pass is much smaller. The model card frames this explicitly as “sparse by design,” with a “latent MoE” architecture that routes computation at a latent level rather than naively dispatching across all experts.
Second is quantization. NVFP4 compresses weights down to roughly 4 bits per parameter, compared to the 16 bits used in most native model weights or the 8 bits used in earlier quantization efforts. A model natively trained or adapted at 4-bit precision needs a fraction of the storage and memory bandwidth of its full-precision counterpart. Combine aggressive quantization with sparse activation, and a parameter count that would otherwise require multiple data-center GPUs becomes something a single 24GB card can plausibly hold and run.
The repository’s file list backs this architecture up: separate dense.bin and experts.bin files, a model-skeleton.gguf, latent dense weights at both 4-bit and 8-bit precision (latent_dense_q4.bin, latent_dense_q8.bin), and a profile.bin. This isn’t a single monolithic weights file; it’s a modular package built around the specific demands of running a sparse, quantized MoE efficiently on constrained hardware.
What is NVFP4, and why does it matter here?
NVFP4 is a 4-bit floating-point format intended for NVIDIA’s newer GPU generations, which include hardware-level support for low-precision math. It differs from older post-training quantization methods like GPTQ, AWQ, or GGUF’s integer-based k-quants, which approximate full-precision weights after the fact. NVFP4-native handling, where supported, can keep more of a model’s accuracy intact at 4 bits because the format and the training or calibration process are matched to how the hardware actually computes.
The practical relevance for anyone trying to run large models locally: precision format isn’t just a storage detail, it affects both quality and speed. A model quantized carelessly to 4 bits can lose noticeable capability, especially on reasoning or coding tasks. A model designed around 4-bit precision from the start, or carefully adapted to it, tends to hold up better. Edge0-145B’s tags list both 4-bit and nvfp4 alongside cuda, signaling that the intended runtime path assumes CUDA-capable NVIDIA hardware with FP4 support, not a generic CPU or cross-vendor setup.
The repo also ships Q3_K variants (for the smaller Edge0-120B model), which use llama.cpp-style k-quantization instead. That gives users a choice between the NVIDIA-specific NVFP4 path and a more hardware-agnostic GGUF quantization path, depending on what GPU they actually have.
Is Edge0-145B worth running locally?
Remy.
That depends on what “worth it” means for a given use case. The appeal is clear: a model at this parameter scale would ordinarily require enterprise-grade multi-GPU infrastructure, and Edge0-145B is explicitly pitched at a single 24GB consumer card. For developers who want to experiment with a large sparse model’s behavior on agent or coding tasks without renting cloud GPU time, that’s a meaningful opening.
The caveats are also clear. Aggressive quantization and sparse routing both introduce tradeoffs. A 4-bit MoE model will generally not match the raw quality of a dense, full-precision model of similar total size, and real-world agent and coding performance depends heavily on how well the MoE router and the quantization were tuned. The model card’s own file structure (separate quality-overview assets, a dedicated quality folder) suggests Edge0 publishes benchmark comparisons against other quantization tiers and model sizes, which is the right place to check actual task performance before relying on it for serious work. Treat vendor-published benchmark charts as a starting point, not a verdict, and validate on your own prompts.
There’s also a practical setup cost. The model doesn’t load through standard transformers tooling; it uses Edge0’s own edge0 library, a GGUF skeleton file, a start.sh script, and a custom tokenizer setup. That’s a heavier lift than pulling a model and calling .from_pretrained(), and it means compatibility with existing local-inference pipelines (Ollama, LM Studio, vLLM, etc.) isn’t guaranteed out of the box unless those tools add explicit support.
What hardware do you actually need?
The model card’s framing centers on “one consumer GPU” with the RTX 5090’s 24GB of VRAM as the reference target. That number is consistent with the math: a 145B-parameter sparse MoE model, quantized to roughly 4 bits per parameter, with only a subset of experts resident or active per token, can land in the double-digit-gigabyte range rather than the 200GB-plus a dense FP16 model of similar scale would demand.
Beyond VRAM, NVFP4 support is tied to newer NVIDIA GPU architectures, so the “consumer GPU” claim is narrower than “any GPU.” Older cards without native FP4 support, or non-NVIDIA GPUs, aren’t part of the target hardware. The cuda tag in the model’s metadata reinforces that this is built and tested specifically against NVIDIA’s software stack, not a generic cross-platform runtime.
How does it compare to the smaller Edge0-120B?
The same Hugging Face repository space includes badges and assets referencing Edge0-120B in both NVFP4 and Q3_K formats. That points to a deliberate product line: a 120B variant for users with less VRAM headroom or who want the more portable GGUF Q3_K format, and the larger 145B NVFP4 flagship for users with a 24GB card who want the most capable version Edge0 offers. The exact performance gap between the two isn’t detailed in the available metadata, but the existence of multiple size and quantization tiers suggests Edge0 is explicitly optimizing for “best model that fits your GPU” rather than a single one-size release.
Frequently Asked Questions
What GPU do I need to run Edge0-145B?
The model is built around a single 24GB consumer GPU, with the RTX 5090 cited as the reference card. NVFP4 support depends on newer NVIDIA GPU architectures, so older or non-NVIDIA cards aren’t the intended target.
Is Edge0-145B a dense or MoE model?
Without shadow IT.
It’s a sparse mixture-of-experts model using a “latent MoE” architecture, meaning only a portion of its 145 billion total parameters activate per token. That sparsity, combined with 4-bit NVFP4 quantization, is what allows it to fit on a single GPU.
What base model is Edge0-145B built on?
It’s based on Qwen3.8-Flash-Next, with Edge0 applying its own architectural and quantization packaging on top.
Can I load Edge0-145B with standard Hugging Face transformers code?
No. The repository uses Edge0’s own edge0 library along with a GGUF skeleton file, custom binaries for experts and dense layers, and a start.sh launch script, rather than the standard transformers loading pattern.
What is NVFP4 and how is it different from GGUF quantization?
NVFP4 is a 4-bit floating-point format designed for newer NVIDIA GPU hardware. It differs from GGUF-style integer k-quants (like Q3_K, which Edge0 also offers for its 120B model) in that it’s tied to specific NVIDIA hardware support rather than being a hardware-agnostic CPU/GPU format.
