Edge0-145B: A 145B Agent Model That Runs on One 24GB GPU
Edge0-145B is a sparse MoE agent model built to run locally on a single consumer GPU. Here's what the specs and quantization formats mean.

What is Edge0-145B?
Edge0-145B is an open-weight AI model with 145 billion total parameters, released by Edge0 and built on a sparse mixture-of-experts (MoE) architecture derived from Qwen’s Qwen3.8-Flash-Next base. The pitch is specific: a model this large is normally something you’d rent on a multi-GPU cloud server, but Edge0 packaged it to run on a single consumer GPU with 24GB of VRAM. It ships in two quantized formats, NVFP4 and Q3_K, both distributed as GGUF files designed for CUDA-based local inference, and it’s positioned as an “edge-inference” agent and coding model rather than a general chatbot.
TL;DR
- Edge0-145B packs 145 billion total parameters into a sparse MoE design, but only a fraction of those parameters activate per token, which is what makes single-GPU inference realistic.
- The model is distributed in two quantization formats: NVFP4, a 4-bit format trained natively rather than bolted on afterward, and Q3_K, a more aggressive 3-bit GGUF quantization for tighter memory budgets.
- Edge0 targets 24GB consumer GPUs, the VRAM tier of cards like the RTX 4090 and RTX 5090, rather than requiring datacenter-class accelerators.
- The NVFP4 release has pulled in over 55,000 downloads and 468 likes on Hugging Face, while the Q3_K version has logged over 92,000 downloads, suggesting the lower-memory variant is the more popular pick for real-world rigs.
- The model is tagged for coding and agent workloads specifically, not just chat, and includes architecture components like a multi-token prediction (MTP) module and a vision projector (mmproj) for multimodal input.
- It’s licensed under a custom license built on top of Qwen’s own licensing terms, since it’s derived from Qwen3.8-Flash-Next.
How does a 145B model fit on one GPU?
The short answer is sparsity. Edge0-145B is a mixture-of-experts model, which means the full parameter count (145 billion) is split across many specialized “expert” subnetworks, but only a small subset of those experts actually fires for any given token. Dense models of this size would need to hold every parameter active in VRAM simultaneously, which is why a 70B or 140B dense model typically demands multiple high-end GPUs. MoE architectures break that constraint by routing each token through a handful of experts instead of the whole network, which cuts the active compute and memory footprint dramatically even though the full parameter set still has to live somewhere.
The model card describes this as a “latent MoE” design, paired with aggressive quantization, to get the whole stack under the 24GB ceiling that defines mainstream consumer GPUs like the RTX 4090 and RTX 5090. The file list backs this up: the repo separates “dense” weights from “experts” weights (dense.bin, experts.bin), along with a labels file that presumably handles expert routing, and a separate “latent_dense” set in both 4-bit and 8-bit variants. That split architecture is exactly the kind of engineering that lets a model with a huge nominal parameter count behave, at inference time, more like a much smaller dense model in terms of active compute.
What do NVFP4 and Q3_K actually mean?
Both are quantization formats, methods for shrinking a model’s weights down from the 16-bit or 32-bit precision used in training to something that takes far less memory to store and run. The difference between them matters for anyone deciding which file to download.
NVFP4 is a 4-bit floating-point format associated with NVIDIA’s newer GPU generations. What’s notable here, per the model card, is that Edge0-145B was “trained natively at 4 bits” rather than trained at higher precision and quantized down afterward. Native low-bit training generally preserves more model quality than post-hoc quantization, because the model learns its weight distributions with the final precision constraint already baked in, instead of being forced into a narrower representation after the fact.
Q3_K is a 3-bit quantization scheme from the GGUF/llama.cpp ecosystem, the format widely used for running large language models on consumer hardware through tools like llama.cpp, Ollama, and LM Studio. Dropping from 4 bits to 3 bits per weight shrinks the file size and VRAM requirement further, at some cost to output quality or precision, particularly on harder reasoning or long-context tasks. The tradeoff is standard in local-LLM deployment: Q3_K suits GPUs with less headroom or setups where you want to leave VRAM free for a large context window, while NVFP4 (where supported) tends to retain more of the original model’s behavior.
Both ship as GGUF files, the standard container format for quantized weights designed for CPU/GPU hybrid or pure-GPU inference outside of the original training frameworks.
Is Edge0-145B worth running locally?
For developers who specifically need an agent or coding model and already own a 24GB-class GPU, Edge0-145B is a reasonable thing to try, and the download numbers suggest plenty of people already have. The Q3_K variant’s 92,000+ downloads versus NVFP4’s 55,000+ on Hugging Face indicates the broader compatibility of the GGUF Q3_K path (it runs on more hardware, including older cards and partial CPU offload setups) is winning out over the newer, more NVIDIA-specific NVFP4 format, at least so far.
The caveats are the same ones that apply to any local MoE deployment. Total disk footprint still reflects the full 145B parameter count, even though only part of that loads into active VRAM during inference, so storage and initial load times are nontrivial. Quality at 3-bit and 4-bit precision will not match a full-precision or cloud-hosted version of the same model family on every task, particularly tasks requiring long chains of precise reasoning. And because this is tagged specifically for coding and agent use cases, with an MTP (multi-token prediction) component included for faster generation, it’s not built or benchmarked as a general creative-writing or chat model.
Where it’s compelling is the scenario it’s targeting directly: running a genuinely large, agent-capable model entirely offline, with no API costs and no data leaving the machine, on hardware that a serious hobbyist or small team can actually own. That’s a different value proposition than chasing state-of-the-art benchmark scores against frontier cloud models.
How does it compare to other local models?
Edge0-145B sits in a growing category of models specifically engineered for the “big parameter count, small active footprint” tradeoff, rather than being a scaled-down model to begin with. Its base, Qwen3.8-Flash-Next, is part of Alibaba’s Qwen model family, which has produced several MoE variants aimed at efficient inference. The distinguishing feature Edge0 adds is the packaging: native 4-bit training, a latent MoE routing scheme, and a release structure (separate dense/expert/latent weight files, a vision projector, an MTP module) clearly built around squeezing a 145B model onto a single 24GB card rather than a server rack.
Compared to more conventional quantized releases of large open models, where a team takes an existing checkpoint and quantizes it after the fact, Edge0’s claim of native 4-bit training is the more unusual engineering choice, and it’s the detail most worth scrutinizing if you’re evaluating output quality against benchmarks before committing disk space and download time.
Frequently Asked Questions
How much VRAM does Edge0-145B need?
The model is built to run on a single consumer GPU with 24GB of VRAM, the tier occupied by cards like the RTX 4090 and RTX 5090. This is achievable because of its sparse MoE architecture and 4-bit (NVFP4) or 3-bit (Q3_K) quantization, which keep active memory use far below what the full 145B parameter count would suggest.
What’s the difference between the NVFP4 and Q3_K versions?
NVFP4 is a 4-bit floating-point format that Edge0 says the model was trained in natively, which generally preserves more quality than after-the-fact quantization. Q3_K is a 3-bit GGUF quantization, more aggressive and broadly compatible with existing local-inference tools like llama.cpp, trading some precision for a smaller footprint.
Is Edge0-145B a chat model or something else?
It’s tagged specifically for coding and agent workloads rather than general conversation. The architecture includes a multi-token prediction (MTP) module for faster generation and a vision projector (mmproj) file, suggesting multimodal input support alongside its text-generation and agent-oriented tagging.
What base model is Edge0-145B built on?
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
It’s derived from Qwen3.8-Flash-Next, part of Alibaba’s Qwen model family, and is licensed under a custom license that incorporates Qwen’s own licensing terms as a dependency.
Which quantized version is more popular?
Based on Hugging Face download counts, the Q3_K GGUF version has more downloads (over 92,000) than the NVFP4 version (over 55,000), likely because Q3_K’s broader hardware and tooling compatibility appeals to a wider range of local setups, even though NVFP4 is the format the model was natively trained in.





