Edge0-35B-A3B: A 35B MoE Model That Runs in 3GB of RAM
Edge0-35B-A3B streams MoE experts from SSD to disk to run a 35B parameter model in under 3 GiB of active memory at 15 tokens per second.
What is Edge0-35B-A3B?
Edge0-35B-A3B is a 35 billion parameter mixture-of-experts (MoE) language model that runs with under 3 GiB of active memory by keeping most of its weights on disk and streaming only the experts it needs, when it needs them. It’s built on Qwen3.5-MoE 35B-A3B, quantized to 4-bit, and paired with two custom components (a prerouter and a Recover-LoRA adapter) that make storage-backed inference fast enough for interactive chat. On a Mac mini M4 Pro, it decodes at roughly 15 to 18 tokens per second.
TL;DR
- Edge0-35B-A3B is a 35B-parameter MoE model that runs with a peak active memory footprint of about 2.9 GiB, low enough for phone-class or entry-level hardware rather than a workstation GPU.
- The model uses SSD expert offload, keeping the full weight set on storage and pulling only the routed experts into RAM for each token, so memory scales with the active parameter count (3B active per token) instead of the full 35B.
- A trained prerouter predicts which experts will be needed one step ahead, overlapping the SSD fetch with computation and adding up to 59% more decode throughput than naive on-demand loading.
- A distillation technique called Recover-LoRA trains adapters on top of the frozen int4 base to close most of the accuracy gap introduced by quantization, landing within 3.9 points of the fp16 original on average across five benchmarks.
- Benchmarked speeds are 14.9 to 17.7 tokens per second decode and 113 to 140 tokens per second prefill, measured on a Mac mini M4 Pro with 24 GB of unified memory.
- The release is Apache 2.0 licensed, ships as a complete model directory (base weights plus adapters) for the open-source edge0 inference framework, and currently targets Apple Silicon via an MLX backend.
- It’s explicitly a preview: agentic behavior (tool use, multi-step planning) is weak in this release, and the model is tuned mainly for the base model’s original languages.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How does a 35B model fit in 3GB of RAM?
The short answer: it doesn’t, in the sense that “fit” usually implies. The full 4-bit checkpoint stays on storage the entire time. What lives in RAM is only the subset of the network actually needed to compute the next token.
This works because Edge0-35B-A3B is a sparse mixture-of-experts model. Instead of one dense stack of layers that all activate for every token, the model has 256 experts per layer and routes each token through only 4 of them (K=4). That’s the “A3B” in the name: roughly 3 billion parameters are active per forward pass, even though the total parameter count across all experts is around 35 billion.
Standard MoE serving still loads every expert into memory up front, because you don’t know in advance which ones will be routed to. Edge0 skips that. It treats the SSD (or fast internal flash) as the primary store for expert weights and streams the ones actually selected by the router, on demand, per token. Peak memory is bounded by the active set at any given moment, not by the total parameter count. According to the model card, this is measured at about 2.9 GiB of peak active memory during benchmarking on a Mac mini M4 Pro, with short contexts (longer contexts add their own KV cache overhead on top).
What is a “prerouter” and why does it matter?
Streaming experts from storage on every token sounds like it should be slow, since disk I/O is orders of magnitude slower than RAM access. Edge0’s answer is a small trained model called a prerouter, which predicts the routing decision one step ahead of when it’s actually needed.
Normally, the router decides which experts to activate only after seeing the current hidden state, which means the model has to compute first, then fetch weights, then compute again, a strictly sequential chain that stalls on every storage read. The prerouter breaks that dependency by forecasting likely expert choices in advance, so the weight fetch from SSD can happen in parallel with the ongoing forward pass instead of blocking it.
The model card reports this overlap yields up to 59% higher decode throughput compared to loading experts reactively. It also notes that the benefit scales up with three factors: how slow the storage is, how large the model is, and how many experts are routed per token (K). That makes the prerouter especially useful for exactly the situation Edge0 targets, running big MoE models on modest hardware where storage latency would otherwise dominate.
Does quantization ruin the model’s quality?
Not by much, according to the benchmarks in the model card. The base checkpoint is quantized to 4-bit, which normally causes a noticeable accuracy hit on top of speed and memory savings. Edge0 addresses this with what it calls Recover-LoRA: the int4 base is frozen, and LoRA adapters are trained via distillation from the original FP16 model to recover accuracy lost during quantization.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
The published numbers, run with OpenCompass under identical settings for both models, show the int4-plus-adapters version landing an average of 3.9 points below the fp16 Qwen3.5-MoE 35B-A3B baseline across five benchmarks (AIME 2026, HumanEval, GPQA-Diamond, MMLU-Pro, and IFBench), on a 100-point scale:
| Benchmark | Edge0-35B (int4) | Qwen3.5-MoE 35B-A3B (fp16) |
|---|---|---|
| AIME 2026 | 86.6 | 92.7 |
| HumanEval | 90.9 | 95.1 |
| GPQA-Diamond | 79.8 | 81.8 |
| MMLU-Pro | 81.0 | 84.6 |
| IFBench | 57.9 | 61.7 |
| Average | 79.2 | 83.2 |
A useful architectural detail: the LoRA adapters stay unmerged with the base weights. That means a single read-only int4 base checkpoint can serve multiple different adapter sets without re-quantizing anything, which matters for anyone trying to serve several fine-tuned variants off one storage-bound base model.
Is Edge0-35B-A3B worth using right now?
It depends on what you need. As a preview release, it’s clearly aimed at people experimenting with edge inference rather than production agent workloads. The model card is explicit that agentic capability (tool use, multi-step planning, long-horizon autonomy) is currently weak, with the promise that a full release will strengthen this. It’s also primarily tuned for the base model’s original languages, and the MLX backend currently only targets Apple Silicon, so this isn’t yet a cross-platform solution.
Where it looks genuinely useful is anywhere GPU memory is the bottleneck but fast storage is available. That includes on-device inference on machines without a dedicated GPU, and batch serving scenarios where one base model needs to support many different LoRA adapter sets without duplicating or re-quantizing weights for each one. Chat and reasoning tasks with the bundled “thinking mode” template are the stated use case, not coding agents or long autonomous runs.
The license is Apache 2.0, and the full stack (base checkpoint, LoRA adapter, and prerouter adapter) ships together as a ready-to-run directory for the open-source edge0 framework, so there’s no separate assembly step required to try it.
What hardware do you need to run it?
The published benchmarks come from a Mac mini M4 Pro with 24 GB of unified memory, but the whole point of the SSD-offload design is that you don’t need 24 GB of fast memory dedicated to the model, since only about 2.9 GiB is active at once. The real requirement is fast storage: the architecture assumes NVMe or comparable internal flash, since the prerouter’s throughput gains scale with storage latency, meaning slower drives will show a bigger relative benefit from prerouting but still cap out lower in absolute speed than a fast SSD would.
Long context windows are the main caveat. The 3 GiB figure applies to short contexts; extending context length grows the KV cache independently of the streaming-experts mechanism, so users aiming to stay at the lowest memory footprint need to keep prompts and generations relatively short.
Frequently Asked Questions
What does “A3B” mean in Edge0-35B-A3B?
It denotes an MoE model with roughly 35 billion total parameters across all experts, of which about 3 billion are active for any given token. Only the active experts are computed and, in Edge0’s case, only those are loaded into memory.
How is this different from just quantizing a big model to fit in less RAM?
Quantization alone reduces the size of every weight but still requires loading the entire model into memory. Edge0 additionally never loads the full model at once: it streams only the routed experts from storage per token, which is why it can go from a nominal 35B-parameter footprint to under 3 GiB of active RAM.
What speed can I expect?
The model card reports 14.9 to 17.7 tokens per second during decode and 113 to 140 tokens per second during prefill, benchmarked on a Mac mini M4 Pro with 24 GB of memory using the project’s own bench.py script.
Can I run this on Windows or Linux with an NVIDIA GPU?
Not yet in this preview. The current backend is MLX, which targets Apple Silicon specifically. The model card lists other backends as being on the project’s roadmap but not yet available.
Is Edge0-35B-A3B good at agentic tasks or tool use?
No, not in this preview. The model card states plainly that tool use, multi-step planning, and long-horizon autonomy are currently weak, and that a future full release is intended to address this gap.