Edge0: Run a 35B Parameter Model in Under 3GB of Memory
Edge0 streams MoE experts from disk to run Qwen 3.5 35B-A3B in under 3GB RAM. Here's how the architecture works and how to install it.
What is Edge0?
Edge0 is an open-source streaming inference framework that lets a 35 billion parameter mixture-of-experts model run in under 3GB of active memory. It does this by keeping the full model checkpoint on disk and pulling only the experts needed for the current token into RAM, rather than loading the entire model up front the way standard inference engines do. The project currently runs on Apple Silicon Macs through MLX, with CUDA support planned but not yet available.
TL;DR
- Edge0 decouples model size from memory footprint by streaming only the active experts for each token off disk instead of loading all 256 experts into RAM at once.
- The flagship model, Edge0-35B, runs in about 2.9GB of peak memory at 15 to 18 tokens per second on a Mac Mini M4 Pro, despite being built on a 35 billion parameter Qwen 3.5 mixture-of-experts base with 40 layers and 4 active experts per token out of 256.
- A smaller Edge0-8B variant, based on a different architecture with 128 experts and 8 active per token, uses roughly 1GB of memory and runs faster, at 24 to 25 tokens per second.
- Both models ship as 4-bit quantized checkpoints with trained adapters bundled in, so a single download gives you a ready-to-run model without extra conversion steps.
- Quantization has a measurable but small quality cost: the int4 35B model loses about 3.9 points on average versus the FP16 version across benchmarks like MMLU-Pro and HumanEval, clawed back partly through a technique the project calls Recovered LoRA.
- The framework is explicitly a preview, not production-ready, and the model card states it is weak at agentic use cases like tool calling, multi-step planning, and long-horizon tasks.
- Right now it only works on Apple Silicon via MLX; Nvidia/CUDA users cannot run it hands-on until that backend ships.
How does Edge0 fit a 35B model into 3GB of RAM?
Every other technique for shrinking large language models works by making the model itself smaller: fewer parameters, distillation into a compact architecture, pruning weights, or dropping to lower precision. Edge0 takes a different route. It keeps the full 35 billion parameter checkpoint intact on disk and never loads all of it into memory at once.
The checkpoint is stored as memory-mapped files. When the model needs a particular expert for the current computation, Edge0 streams that expert off storage on demand instead of holding all 256 experts resident in RAM. Since a mixture-of-experts model only activates a small subset of its total experts per token anyway (4 out of 256 for the 35B model), most of the checkpoint sits idle on disk at any given moment. Edge0 exploits that idle time by not loading what isn’t being used.
This is the core idea: parameter count and RAM footprint are no longer the same number. A model can be enormous on disk while staying small in active memory, as long as the inference engine can predict and fetch the right pieces fast enough to keep up with generation speed.
What keeps it fast enough to be usable?
Streaming experts off disk sounds like it should be slow, and normally it would be. The trick Edge0 uses to avoid stalling is a small trained component called a pre-router. Its job is to predict which experts the next token will need one step ahead of time, so the loading for that expert happens in parallel while the current token is still being computed.
That overlap between prediction and computation is where most of the speed comes from. According to the project’s own claims, this pipelining delivers up to 59% faster decoding compared to a naive stream-on-demand approach without prediction. It’s the difference between a model that technically works but feels sluggish and one that hits usable token-per-second numbers on consumer hardware.
What are the actual specs for Edge0-35B and Edge0-8B?
Edge0 currently ships two model tiers:
Edge0-35B is built on a Qwen 3.5 mixture-of-experts base with 40 layers and 256 total experts, of which 4 are active per token. It needs about 2.9GB of peak memory and runs at 15 to 18 tokens per second on a Mac Mini M4 Pro.
Edge0-8B is built on a smaller Llama 3 based architecture with 128 experts and 8 active per token. It runs faster, at 24 to 25 tokens per second, using around 1GB of memory.
Both models are distributed as 4-bit quantized checkpoints with trained adapters bundled in, so downloading either one gives you a working model without needing to source a base checkpoint separately or run your own quantization.
Does quantizing to 4-bit hurt quality?
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
Yes, but the tradeoff appears to be modest according to figures shared by the project. The int4 version of Edge0-35B loses about 3.9 points on average compared to the FP16 version of the same model, measured across benchmarks including MMLU-Pro and HumanEval. For a roughly 90% cut in memory requirements, that’s a relatively small quality hit, though it’s worth treating as a starting point rather than a full picture since detailed benchmark breakdowns aren’t extensively published yet.
The technique used to recover most of that lost quality is called Recovered LoRA. Adapters are trained through distillation from the full-precision model and layered on top of the frozen int4 base to claw back accuracy that quantization would otherwise cost. Critically, this adapter is unmerged, meaning the base checkpoint stays untouched and read-only while the adapter sits alongside it. That separation keeps the architecture flexible for future backend ports.
What can’t Edge0 do yet?
The project’s own model card is upfront about limitations. This preview release is not optimized for agentic use. Tool use, multi-step planning, and long-horizon tasks are all described as weak points right now. In practice, that means popular agent harnesses and tool-calling frameworks won’t work well with these models yet. Edge0 is positioned as a strong chat and reasoning model in a small footprint, not a backbone for autonomous agents.
There’s also the hardware limitation. Edge0 runs through MLX, which means it’s restricted to Apple Silicon Macs for now. CUDA support is on the roadmap but hasn’t shipped. Anyone on Nvidia hardware will need to wait, though the project’s architecture is reportedly built so a CUDA backend can plug into the same codebase later without requiring a rewrite.
Is Edge0 worth trying right now?
If you have an Apple Silicon Mac and want to experiment with running a genuinely large model on constrained memory, Edge0 is worth a look, with the caveat that this is early-stage, preview-quality software. Nothing about it is described as production ready. The lack of agentic capability rules it out for anyone building tool-using agents or automation pipelines today.
For everyone else, especially Nvidia GPU users, the value right now is understanding the architecture rather than running it. The expert-streaming approach, paired with a predictive pre-router and an unmerged recovery adapter, represents a genuinely different strategy for fitting large models onto small hardware. Whether or not Edge0 itself becomes the tool people standardize on, the technique of decoupling total model size from active memory footprint seems likely to show up in other frameworks once a CUDA backend exists.
How do you install and run Edge0 on a Mac?
The setup follows a fairly standard Python workflow:
- Set up a virtual environment and install Edge0 with pip in editable mode.
- Download a model tier, either the 35B or 8B version, using the project’s fetch script, which pulls the base checkpoint and both trained adapters into a single folder.
- Point Edge0 at that folder, either through an environment variable or by passing the path directly. It auto-detects the model configuration from there.
- Run the included demo command to confirm the installation works.
- Serve the model as an OpenAI-compatible API on localhost, so it can be queried with curl or plugged into any tool that speaks the standard OpenAI chat format.
- For a quick one-off test, use the direct chat command, which takes a prompt and prints a response without starting a server.
Frequently Asked Questions
What is Edge0?
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Edge0 is an open-source inference framework that streams mixture-of-experts model weights from disk into memory on demand, allowing a 35 billion parameter model to run in under 3GB of active RAM instead of requiring the full checkpoint loaded at once.
Can I run Edge0 on Windows or Linux with an Nvidia GPU?
Not yet. Edge0 currently only supports Apple Silicon Macs through MLX. CUDA support for Nvidia GPUs is planned but hadn’t shipped as of the framework’s initial preview release.
How much memory does Edge0-35B actually need?
About 2.9GB of peak memory on a Mac Mini M4 Pro, running at 15 to 18 tokens per second, according to the project’s published figures.
Does Edge0 support AI agents and tool calling?
No, not reliably. The project’s own model card states this preview isn’t optimized for agentic use, and tool use, multi-step planning, and long-horizon tasks are described as weak points.
How does quantization affect model quality in Edge0?
The 4-bit quantized 35B model loses about 3.9 points on average compared to its FP16 counterpart across benchmarks like MMLU-Pro and HumanEval. A technique called Recovered LoRA, using trained adapters layered on the frozen quantized base, helps recover part of that quality loss.
