Clover-35B-A3B: A New Open-Source Diffusion Language Model Explained
Clover-35B-A3B-Preview pairs block diffusion with a 35B MoE backbone and 128K context. Here's how it works and what it takes to run.
What is Clover-35B-A3B-Preview?
Clover-35B-A3B-Preview is an open-weight diffusion language model released by DiffuSpace, built on top of Qwen3.6-35B-A3B. It uses a 35-billion-parameter mixture-of-experts (MoE) backbone with roughly 3 billion parameters active per token, a 128K-token context window, and a block diffusion decoding method instead of standard left-to-right autoregressive generation. It’s distributed under Apache 2.0 and runs through Hugging Face Transformers or DiffuSpace’s own SGLang and llama.cpp backends.
TL;DR
- Block diffusion decoding generates text in chunks rather than one token at a time: each block starts fully masked and the model fills in confident positions in parallel until the whole block resolves.
- The model inherits its MoE backbone from Qwen3.6-35B-A3B, with 40 layers, 256 routed experts, top-8 routing, and one shared expert per layer, giving it roughly 3B active parameters out of 35B total.
- Clover adds bidirectional block attention on top of Qwen’s hybrid Gated DeltaNet / full-attention design, using extra learned parameters so tokens can attend within a noisy block while still respecting a causal prefix.
- Running it in BF16 requires a GPU (or multi-GPU setup) with around 69 GB of memory just for weights, before accounting for KV cache and inference state.
- A block-level expert selection feature (
moe_block_coreset_top_p) can restrict routing to a shared subset of experts per block, trading some quality for fewer distinct experts activated during denoising. - DiffuSpace recommends its own SGLang implementation for serving and benchmarking, since the standalone Transformers path can produce slightly different outputs due to kernel-level numerical differences.
- The model supports a full 128K context window covering prompt and generation combined, with soft length and repetition controls to prevent long outputs from looping.
How does block diffusion generation differ from standard LLM decoding?
Most large language models generate text autoregressively: one token at a time, left to right, each token conditioned on everything before it. Clover instead generates in blocks. Each block (32 tokens by default) starts out entirely masked, like noise in an image diffusion model. The model then scores confidence for each masked position and fills in whichever tokens cross a threshold (0.95 by default) in parallel. If no position meets that bar on a given pass, the single most confident token gets filled anyway, so generation always makes progress rather than stalling.
Once a block is fully resolved, it becomes part of the causal prefix for the next block. That’s the “bidirectional block attention” piece: within an unresolved block, tokens can attend to each other in both directions (useful for filling in gaps out of strict left-to-right order), but once a block locks in, it behaves like a normal causal prefix for everything that follows. DiffuSpace implemented this by adding learned parameters, including right-convolution weights and softmax controls, to Qwen’s existing Gated DeltaNet layers rather than replacing the architecture outright.
The practical upshot: instead of committing to one token and never revisiting it, the model can resolve several tokens in a block simultaneously, which is the core bet behind diffusion-style language models as an alternative to pure autoregression.
What’s inside the architecture?
Clover is a text-only model derived from Qwen3.6-35B-A3B, and it keeps most of that base model’s bones intact:
- 40 layers using a hybrid of Gated DeltaNet (a linear-attention-style mechanism) and full attention.
- 256 routed experts per MoE layer, with top-8 token routing (each token is handled by 8 of the 256 experts) plus one shared expert that every token passes through.
- ~35B total parameters, with only about 3B active per token thanks to the MoE routing, which is what keeps inference compute manageable despite the large total parameter count.
- 128K token context window, covering prompt and generated output combined. Requests that would exceed this limit, including the space needed to finish resolving a final block, are rejected outright.
On top of this, DiffuSpace’s modifications add the bidirectional block attention parameters and the block diffusion generation logic itself, plus a set of generation controls built specifically for the quirks of block-based decoding.
What hardware do you need to run it locally?
The model ships in BF16 safetensors format, split across 26 files. According to the model card, you need a GPU with enough memory to hold approximately 69 GB of weights in BF16, plus additional headroom for inference state (KV cache, activations, and the diffusion block’s working memory). That rules out running it on a single consumer GPU with 24 GB or even 48 GB of VRAM without quantization; this is squarely a data-center-or-high-end-workstation model in its native precision, something like a single 80GB A100/H100 or a multi-GPU setup.
To get started with the reference implementation, you need:
pip install "transformers==5.19.0" accelerate safetensorsThe model card’s example loads it with device_map="cuda:0" and dtype=torch.bfloat16, applies a chat template with enable_thinking=True, and calls generate() with block_size=32, threshold=0.95, and max_new_tokens=24576. Generation defaults load automatically from the bundled generation_config.json, so most users won’t need to hand-tune block size or confidence threshold.
For production serving, DiffuSpace points users toward its own SGLang implementation instead of the plain Transformers path, since the two can produce slightly different generated text due to small numerical differences between PyTorch operations and SGLang’s custom kernels, even when decoding settings are identical.
How does expert selection work during block denoising?
One distinctive feature of Clover is block-level expert selection, controlled by two config settings: moe_block_coreset_enabled (on by default) and moe_block_coreset_top_p (default 0.9). During the denoising process for a block, instead of letting every token independently route to its own top-8 experts across all 256 available, Clover can aggregate the routing scores of all tokens’ candidate experts across the entire block, rank experts by that aggregate score, and keep only enough of them to cover 90% of the total routing mass.
That 90% refers to cumulative routing weight, not 90% of all 256 experts, so the actual number of experts retained per block can be much smaller. The purpose is to reduce the number of distinct experts invoked while processing a block, which can matter for memory bandwidth and serving efficiency on MoE hardware, though DiffuSpace notes lower top_p values can carry a quality tradeoff. This setting only affects which experts are used during inference. It doesn’t shrink the model’s stored or loaded weights, so it’s not a way to reduce the ~69 GB memory footprint.
Is Clover-35B worth running over a standard autoregressive model?
That depends on what you’re optimizing for. Clover’s pitch is state-of-the-art performance among open-source diffusion language models on the benchmarks DiffuSpace reports in its launch blog, with self-speculative decoding support in the SGLang and llama.cpp backends aimed at improving throughput. Diffusion-style decoding is still a less mature serving path than standard autoregressive transformers, so ecosystem support (quantized formats, inference frameworks, fine-tuning tooling) lags behind well-established dense and MoE autoregressive models.
For teams specifically interested in block diffusion as a decoding paradigm, Clover is one of the few openly released models at this scale (35B total, 3B active) that lets you test it directly rather than reading about it in a paper. For teams just looking for the best general-purpose open model to self-host, the hardware requirements (69GB+ for weights alone) and the newer, less battle-tested serving stack are worth weighing against more established options.
Frequently Asked Questions
What does “A3B” mean in Clover-35B-A3B?
It denotes the model’s mixture-of-experts structure: 35B total parameters with approximately 3B active per token, since MoE routing only runs each token through a subset of experts (top-8 of 256, plus one shared expert) rather than the full parameter set.
What base model is Clover built on?
Clover is derived from Qwen3.6-35B-A3B, retaining its hybrid Gated DeltaNet/full-attention MoE architecture (40 layers, 256 routed experts) and its Apache 2.0 license, with DiffuSpace adding block diffusion inference and bidirectional attention parameters on top.
How much VRAM does Clover-35B need?
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
Roughly 69 GB for the BF16 weights alone, before adding memory for KV cache and other inference state, meaning it generally requires a high-memory GPU like an 80GB A100/H100 or a multi-GPU configuration.
What’s the difference between the Transformers and SGLang implementations?
Both run the same model, but DiffuSpace recommends SGLang for serving and evaluation because its custom kernels can produce slightly different numerical results, and therefore different generated text, compared to the standalone PyTorch path in Transformers, even with identical settings.
Can I reduce the model’s memory footprint with expert selection settings?
No. The moe_block_coreset_top_p setting controls how many experts are actively used during block denoising for efficiency, but it doesn’t change how many weights are stored or loaded into memory, so it won’t lower the baseline ~69 GB requirement.
