How to Run Clover-35B-A3B-Preview Locally with SGLang
A deployment guide to Clover-35B-A3B's VRAM needs, block diffusion decoding, and setup with Transformers or the recommended SGLang backend.
What is Clover-35B-A3B-Preview?
Clover-35B-A3B-Preview is an open-weight diffusion language model from DiffuSpace, built on a 35B-parameter mixture-of-experts backbone with roughly 3B active parameters per token. It uses block diffusion instead of standard left-to-right autoregressive decoding, supports a 128K token context window, and is derived from Qwen3.6-35B-A3B. Running it locally requires a GPU with enough memory for the BF16 weights (about 69GB) plus room for inference state, and DiffuSpace recommends SGLang over plain Transformers for serving.
TL;DR
- Clover-35B-A3B-Preview is a mixture-of-experts diffusion language model with 35B total parameters, about 3B active per token, and a 128K context window.
- The model uses block diffusion generation, denoising masked token blocks in parallel rather than generating one token at a time, with defaults of
block_size=32andthreshold=0.95. - Expect to need around 69GB of GPU memory just for the BF16 weights, before accounting for KV cache and activation overhead during inference.
- A basic setup works with Transformers 5.19.0 and PyTorch 2.10+, but DiffuSpace’s own documentation recommends its SGLang implementation for serving and evaluation.
- Clover supports self-speculative decoding through the official SGLang and llama.cpp backends, which is not available in the base Hugging Face Transformers path.
- Generation includes built-in repetition and length controls (presence penalty, frequency penalty, length penalty, think-end bias) that are preconfigured and don’t need manual tuning for typical use.
- An expert selection mechanism called block coreset routing can restrict each block to a subset of experts covering a target share of routing mass, which may trade some quality for efficiency without shrinking the stored model.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
How much GPU memory does Clover-35B need?
The model card specifies approximately 69GB for the BF16 weights alone. That figure covers only the static weight storage, not the additional memory consumed during inference for the KV cache, activation buffers, and whatever batching or context length you’re running. In practice this means a single 80GB-class GPU (such as an A100 80GB or H100 80GB) is the realistic minimum for comfortable single-GPU serving, with limited headroom for long contexts or multiple concurrent requests. Anyone planning to use the full 128K context window should expect to need meaningfully more memory than the base weight footprint, since attention state and generation buffers scale with sequence length.
For setups without a single 80GB GPU, multi-GPU sharding via device_map or a serving framework’s native tensor parallelism becomes necessary. Lower-memory GPUs can’t hold the full BF16 weights on their own, and the model card doesn’t document a supported quantized format, so anyone wanting lower memory usage will need to rely on third-party quantization tools at their own risk, with no guarantee of matching the benchmarked behavior.
How does block diffusion generation work?
Unlike standard autoregressive language models that generate tokens strictly one after another, Clover generates text in blocks using a masked denoising process. Each block starts as a sequence of masked tokens. The model estimates confidence scores for each masked position and fills in any position whose confidence exceeds a threshold (default 0.95). If no position clears that bar, the single most confident position gets filled anyway, which guarantees the generation keeps moving forward even on difficult blocks. Once a block is fully resolved, it becomes part of the causal prefix for the next block.
This is implemented through what DiffuSpace calls bidirectional block attention: the model adds learned Gated DeltaNet parameters, including right-convolution weights and softmax controls, that let tokens see across the current noisy block while still respecting causality relative to everything generated before it. The default block size is 32 tokens. This architecture descends from Qwen3.6-35B-A3B’s hybrid Gated DeltaNet and full-attention MoE design, which has 40 layers, 256 routed experts, top-8 routing per token, and one shared expert per layer.
How do you set up Clover-35B with Transformers?
The quickest path to running Clover locally uses Hugging Face Transformers directly. The model card specifies Transformers 5.19.0 and PyTorch 2.10 or later as the tested combination:
pip install "transformers==5.19.0" accelerate safetensorsFrom there, loading follows a familiar pattern for causal LM inference, with trust_remote_code=True required since Clover’s block diffusion logic lives in custom model code:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Diffuspace/Clover-35B-A3B-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype=torch.bfloat16,
device_map="cuda:0",
).eval()Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
Prompts go through the standard chat template with enable_thinking=True available if you want the model’s reasoning phase exposed. Generation accepts the usual max_new_tokens alongside Clover-specific keyword arguments like block_size and threshold, which override whatever is set in generation_config.json by default. One hard limit to watch: any request that would push total prompt plus generated tokens past 128K gets rejected outright, including the space needed to resolve the final block.
This path is useful for testing and small-scale experimentation, but the model card is explicit that it’s a standalone PyTorch inference implementation, not the optimized serving path.
Why is SGLang the recommended backend?
DiffuSpace’s documentation states plainly that it recommends its official SGLang implementation for serving and evaluation rather than the base Transformers path. Two concrete reasons stand out. First, SGLang supports self-speculative decoding for Clover, a technique not available in the Hugging Face Transformers implementation, which can meaningfully speed up generation by letting the model draft and verify its own tokens rather than resolving each block through the full denoising loop every time. Second, the model card notes that small numerical differences between the Transformers code path and SGLang’s kernels can produce different generated text even under identical decoding settings. If you’re benchmarking the model or trying to reproduce published results, SGLang is the implementation that matches DiffuSpace’s own evaluation setup.
Setup instructions, supported decoding modes, and the self-speculative decoding configuration live in DiffuSpace’s official GitHub repository rather than the Hugging Face model card itself, since this is treated as the production-oriented serving path.
What generation controls does Clover use?
Long diffusion-based generations can drift into repetition, so Clover ships with several built-in controls, all preconfigured with sensible defaults:
| Parameter | Default | What it does |
|---|---|---|
presence_penalty | 0.9 | Discourages reusing tokens that already appeared |
frequency_penalty | 0.0 | Scales penalty by occurrence count (off by default) |
repetition_penalty | 1.05 | Reduces preference for previously generated tokens |
dllm_length_penalty | 10.0 | Gradually biases toward ending the response |
dllm_think_end_bias | 10.0 | Gradually biases toward closing the thinking phase |
Length and thinking-end controls ramp in gradually rather than triggering abruptly. The length penalty starts applying after 8,192 response tokens and reaches full strength over the next 8,192 tokens; the thinking-end bias starts at 4,096 tokens into the thinking phase and ramps over another 4,096. These are soft nudges, not hard cutoffs. The only hard limit is max_new_tokens. Since the defaults are already tuned, most users won’t need to touch these values for standard use.
What is expert selection in Clover?
Clover’s mixture-of-experts backbone normally routes each token to its top 8 experts out of 256 per layer. During block denoising, a separate mechanism called block-level expert selection (moe_block_coreset_enabled, on by default) can restrict an entire block to a shared subset of experts rather than letting every token in the block pick its own routing independently. The selector aggregates routing scores across all tokens in the block, ranks experts by that combined score, and keeps enough experts to cover a target share of the total routing mass, set by moe_block_coreset_top_p (default 0.9, meaning 90% of aggregated routing weight, not 90% of all experts).
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
This is distinct from sampling top_p and doesn’t change the size of the model on disk or in memory since all expert weights remain loaded. Lowering the threshold generally reduces the number of experts engaged per block, which may offer some computational benefit at a potential quality cost, though the model card doesn’t publish specific numbers quantifying that tradeoff.
Frequently Asked Questions
How much VRAM do I need to run Clover-35B-A3B-Preview?
The BF16 model weights require approximately 69GB on their own. Add meaningfully more for KV cache and activation memory depending on context length and batch size, which effectively requires an 80GB-class GPU for single-GPU inference.
Does Clover-35B support quantized formats to reduce memory use?
The official model card documents only the BF16 weight format and does not specify an officially supported quantized release, so reduced-memory deployment would depend on community or third-party quantization efforts.
What’s the difference between running Clover in Transformers versus SGLang?
The Transformers implementation is a standalone PyTorch inference path useful for testing, but it lacks self-speculative decoding and can produce slightly different outputs than SGLang due to kernel-level numerical differences. DiffuSpace recommends SGLang for serving and for reproducing benchmarked evaluation results.
What is self-speculative decoding in this context?
It’s a decoding optimization available in Clover’s official SGLang and llama.cpp implementations that lets the model generate draft tokens and verify them more efficiently than the standard block-by-block denoising loop, which is not available through the base Hugging Face Transformers code.
Can Clover-35B handle very long documents?
Yes, it supports up to 131,072 tokens total across the prompt and generated response combined. Requests that would exceed that limit, including space needed to resolve the final block, are rejected rather than silently truncated.