Mixture of Experts Architecture Explained: How GLM 5.2 Runs 40B Active Parameters
GLM 5.2 has 744B total parameters but only 40B active per token thanks to MoE routing. Learn how this architecture enables local inference on consumer hardware.
The Paradox of a 744B-Parameter Model You Can Run Locally
A model with 744 billion parameters sounds like it belongs in a data center with millions of dollars of GPU hardware. Yet GLM 5.2 — which has exactly that many parameters — can run inference on consumer-grade hardware by activating only about 40 billion of them per token.
That’s not a trick. It’s the Mixture of Experts (MoE) architecture working exactly as designed. Understanding how it works explains a lot about where large language model development is heading and why the gap between “total parameters” and “active parameters” is one of the most important distinctions in AI right now.
This article breaks down how MoE routing works mechanically, what GLM 5.2’s specific architecture looks like, and why this matters for anyone who wants to run capable models locally or choose the right model for production workloads.
What Dense Models Actually Do (and Why It Gets Expensive)
To understand Mixture of Experts, you first need to understand what the alternative looks like.
A traditional “dense” transformer model — like early versions of GPT-3 or Llama 2 — activates every single parameter for every single token it processes. Every layer, every attention head, every neuron in every feed-forward block participates in every forward pass.
This is computationally clean but wasteful. Consider what it means in practice:
- A 70B-parameter dense model requires roughly 140GB of GPU memory at FP16 precision
- Every token processed draws on all 70B parameters, regardless of whether those parameters are relevant to the task
- Scaling up means scaling compute proportionally — double the parameters, roughly double the cost per token
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
The dense approach made sense when models were smaller. As parameter counts crossed the hundreds of billions, it became the dominant bottleneck for both training and inference costs.
The Fundamental Insight Behind MoE
The core idea is simple: not every neuron needs to process every input. A model handling a math problem doesn’t need the same neural pathways as one handling poetry. If you could route each token to only the relevant parts of the model, you’d get the knowledge of a large model with the compute cost of a smaller one.
That’s what Mixture of Experts does. Instead of one monolithic feed-forward network that always runs, MoE replaces it with multiple “expert” subnetworks and a lightweight routing mechanism that picks which experts to use for each token.
How MoE Routing Works
The MoE mechanism sits inside the transformer’s feed-forward layers. Here’s the sequence for each token:
Step 1: The Router Evaluates
Before any expert processes a token, a small learned network called the router (or gating network) looks at the token’s current hidden state and produces a probability distribution over all available experts.
The router is lightweight — typically a single linear layer. It doesn’t add meaningful compute overhead.
Step 2: Top-K Expert Selection
Rather than using all experts, the model selects the top-K experts by probability score. In most modern MoE models, K equals 2, meaning each token is processed by exactly two experts regardless of how many total experts exist.
GLM 5.2 follows this pattern. With a large number of expert groups distributed across the model, each token activates only a small fraction of the total parameter space.
Step 3: Weighted Combination
The outputs from the selected experts are combined using the router’s probability scores as weights. The result gets passed to the next layer, just like the output of a standard feed-forward block.
The model during training learns to:
- Assign experts to specific types of knowledge or reasoning patterns
- Route similar inputs to similar experts consistently
- Balance load across experts so no single expert handles everything
Step 4: Expert Load Balancing
A known failure mode in MoE training is “expert collapse” — where the router learns to always pick the same one or two experts and ignores the rest. To prevent this, MoE training typically includes an auxiliary load balancing loss that penalizes uneven expert utilization.
GLM 5.2 and other modern MoE models use variations of this technique to ensure the full expert capacity is used during both training and inference.
GLM 5.2’s Architecture: The Numbers Explained
GLM 5.2 (developed by THUDM at Tsinghua University) is a MoE model with:
- 744B total parameters — the complete set of weights across all experts, attention layers, and other components
- ~40B active parameters per token — what actually gets computed during a single forward pass
The ratio here is roughly 18.6:1. For every parameter activated during inference, about 17.6 more exist in the model but sit idle for that particular token.
How the Parameter Budget Breaks Down
In a MoE transformer, not all parameters are in expert layers. The architecture typically divides into:
Shared parameters (always active):
- Embedding layers
- Attention mechanisms (Q, K, V, output projections)
- Layer normalization
- The router itself
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
Expert parameters (selectively active):
- Multiple parallel feed-forward networks, each constituting one “expert”
- Only K of these run per token
In GLM 5.2’s case, the attention and embedding layers contribute a significant baseline of always-active parameters. The expert layers multiply this by providing many specialized feed-forward networks, but only 2 (or a small fixed number) activate per token.
The 40B active figure represents the sum of: all shared parameters + the parameters in the selected experts for each token.
Why 744B Matters Even If You Only Use 40B
The total parameter count isn’t irrelevant — it represents the model’s knowledge capacity. A MoE model with 744B total parameters can store far more information across its expert networks than a dense 40B model can.
Think of it this way: a 40B dense model has 40B parameters that learn to handle everything. A 744B MoE model distributes specialized knowledge across many experts — some might specialize in code, others in mathematical reasoning, others in multilingual tasks. During inference, the right specialists show up.
The result is a model with the knowledge of something far larger than 40B parameters, running at roughly the compute cost of a 40B dense model.
Why MoE Enables Local Inference on Consumer Hardware
GPU memory is the main bottleneck for running large language models locally. A model must fit in VRAM (or be offloaded to RAM, with significant speed penalties).
Here’s how the math works out for GLM 5.2:
| Precision | Memory for full 744B model | Memory for 40B active |
|---|---|---|
| FP16 | ~1,488 GB | ~80 GB |
| INT8 | ~744 GB | ~40 GB |
| INT4 | ~372 GB | ~20 GB |
In practice, local inference tools like Ollama, llama.cpp, and LMStudio don’t load only 40B — they still need to load the full model weights into memory so the router can select experts. But at INT4 or aggressive quantization, 744B parameters becomes manageable across consumer hardware setups with large amounts of RAM (even system RAM with CPU offloading).
The Memory vs. Compute Trade-off
This is the key distinction for MoE models when running locally:
- Memory requirement: Scales with total parameters — you need storage for all experts
- Compute requirement: Scales with active parameters — you only run 40B per token
A machine with 192GB of RAM and a mid-range GPU can run GLM 5.2 at reasonable speeds, offloading most weights to system memory while only moving active expert weights to the GPU per forward pass. The same machine couldn’t run a hypothetical dense 744B model at any usable speed.
For teams running local inference for privacy, cost, or offline reasons, this trade-off is often excellent.
MoE vs. Dense Models: When Each Makes Sense
MoE isn’t strictly better than dense models. The choice depends on what you’re optimizing for.
Advantages of MoE
- Higher knowledge capacity per compute unit: You get more effective intelligence per FLOP spent during inference
- Better scaling efficiency: Training MoE models to the same quality as dense models typically requires less compute
- Specialization: Experts naturally develop distinct capabilities, which can improve performance on diverse tasks
Limitations of MoE
- Memory footprint: Total parameter count is large, even if compute is efficient — you need storage for all experts
- Inference latency: Loading different experts per token can cause cache misses and memory bandwidth bottlenecks on some hardware
- Training complexity: Load balancing, router stability, and expert collapse require careful engineering
- Smaller batch size sweet spot: Dense models scale better with large batches; MoE models are often more efficient at smaller batch sizes
How GLM 5.2 Compares
Compared to a dense model like Llama 3.1 70B (which is a pure 70B dense model), GLM 5.2’s 40B active parameters represents a similar compute profile. But GLM 5.2’s 744B total parameter space means it can in principle hold far more knowledge, particularly across diverse domains and languages.
Against other MoE models — Mixtral 8x22B (141B total, ~39B active) or Qwen2-57B-A14B (57B total, 14B active) — GLM 5.2 sits at the larger end of the consumer-accessible MoE spectrum, trading more memory pressure for higher total capacity.
Running MoE Models in Production: Practical Considerations
If you’re evaluating whether to use GLM 5.2 or another MoE model in a real application, here are the factors that matter most.
Throughput and Batching
MoE models can have lower throughput than dense models at large batch sizes because different tokens in the same batch may route to different experts, reducing parallelism. For production inference at scale, this requires careful batching strategies or specialized serving infrastructure.
For low-to-medium traffic workloads — like an internal tool or a low-volume API — this is rarely a problem.
Quantization Effects
Quantization (reducing precision from FP16 to INT8 or INT4) affects MoE models differently than dense models. The routing mechanism is sensitive to precision loss, and aggressive quantization can cause router degradation where the gating network starts making poor expert selections.
Most quantized versions of GLM 5.2 available through Ollama or HuggingFace have been calibrated to minimize this, but it’s worth benchmarking quality on your specific tasks before committing to a heavily quantized version.
Context Length and Expert Utilization
Longer contexts don’t necessarily activate more experts — each token still picks K experts independently. But long-context prompts may create more diverse expert routing patterns across the sequence, which can affect KV cache behavior in some serving frameworks.
Accessing Advanced Models Without Managing Infrastructure
For most builders, the mechanics of MoE routing matter less than the practical question: how do I use capable models like GLM 5.2 without managing GPU clusters or quantization pipelines?
MindStudio addresses this directly. The platform gives you access to 200+ models — including large MoE models — through a single interface, without requiring API keys, separate accounts, or infrastructure setup.
When you’re building an AI agent or workflow in MindStudio, you can swap between model families with a dropdown. If you’re evaluating whether GLM 5.2 outperforms a dense alternative on your specific task, you can test both in the same workflow without spinning up separate environments.
One coffee. One working app.
You bring the idea. Remy manages the project.
This matters especially for teams that want to build AI-powered applications quickly. The time spent on model infrastructure is time not spent on the actual task — prompting, workflow logic, integration with business tools. MindStudio’s model layer handles the infrastructure so you can focus on what the agent should actually do.
If you’re running local models (Ollama, LMStudio, ComfyUI), MindStudio also supports those through its AI Media Workbench and local model integrations — useful if you’re running MoE models locally for privacy or cost reasons and want to wire them into automated workflows.
You can try MindStudio free at mindstudio.ai.
Frequently Asked Questions
What is the difference between total parameters and active parameters in a MoE model?
Total parameters refers to all the weights stored in the model across every expert, attention layer, and embedding. Active parameters refers to the subset that actually gets computed during a single forward pass — the experts selected by the router plus the shared layers. In GLM 5.2, 744B total parameters exist in storage, but only ~40B are used per token during inference. This gap is what makes MoE models memory-intensive but compute-efficient.
How does the MoE router decide which experts to use?
The router is a small learned network (usually a single linear projection) that takes the token’s current hidden state as input and outputs a score for each available expert. The top-K experts by score are selected, and their outputs are combined using weighted averaging. The routing weights are learned during training, not hardcoded — the model learns to associate certain types of inputs with certain experts over the course of training.
Can you actually run GLM 5.2 locally on consumer hardware?
Yes, with caveats. The full 744B model at FP16 requires far more memory than any consumer GPU. However, at 4-bit quantization, the model can be loaded into a combination of GPU VRAM and system RAM on a machine with 128–192GB of total memory. Tools like Ollama support this kind of CPU/GPU split inference. The speed will be slower than a cloud API, but for many use cases — especially offline or privacy-sensitive applications — it’s viable.
Is Mixture of Experts better than a dense model of the same active parameter count?
Generally yes, but it depends on the task and deployment context. A MoE model with 40B active parameters typically outperforms a dense 40B model because it draws on a larger total knowledge base (in GLM 5.2’s case, 744B total). The trade-off is increased memory requirements and potential latency from expert loading. For knowledge-diverse tasks, MoE tends to win. For narrow, specialized tasks where a fine-tuned dense model can be optimized directly, the advantage is less clear.
What happens if the router makes a bad expert selection?
The model’s output degrades, but not catastrophically — at least in well-trained models. During training, the load balancing loss and the routing mechanism itself are optimized to make reliable selections. In practice, routers in large MoE models like GLM 5.2 are quite stable. The main risk occurs with heavy quantization, where precision loss in the routing network can lead to poor expert assignment. This is why quantized MoE models require careful calibration.
How does MoE compare to model distillation for reducing inference costs?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
These are complementary approaches, not alternatives. Distillation creates a smaller dense model that mimics a larger model’s behavior — you reduce both total and active parameters. MoE reduces active parameters while keeping total parameters large. MoE preserves more of the original model’s knowledge but requires more memory. Distillation reduces memory requirements but may sacrifice capability. Some models combine both: train a large MoE model, then distill it into a smaller model for edge deployment.
Key Takeaways
- Mixture of Experts replaces dense feed-forward layers with multiple expert subnetworks, activating only a small number (typically 2) per token via a learned router.
- GLM 5.2’s 744B total parameters represent full knowledge capacity; its ~40B active parameters represent per-token compute cost — a roughly 18:1 ratio.
- The total parameter count determines memory requirements; the active parameter count determines compute and speed. Both matter, but for different reasons.
- MoE models are especially well-suited to diverse, knowledge-intensive tasks and workloads where compute efficiency matters more than peak throughput.
- Local inference of large MoE models is feasible at aggressive quantization with sufficient system RAM — the memory-compute split is the key trade-off to understand.
- If you want to use models like GLM 5.2 without managing infrastructure yourself, MindStudio provides access to a wide model library with no API setup required — try it free.
