Clover 35B-A3B Benchmarks vs Reality: Does It Actually Beat Qwen?
Hands-on tests of Clover 35B-A3B against its own benchmark charts reveal gaps in coding, multilingual output, and logic puzzles versus Qwen.
What is Clover 35B-A3B?
Clover is a diffusion language model built on top of Qwen’s 35 billion parameter mixture-of-experts architecture, with only about 3 billion parameters active at any one time. Instead of generating text one token at a time like a standard autoregressive model, Clover fills in a whole block of text at once and then moves to the next block. It ships with a 128k context window and is positioned by its creators as the new leader among open-source diffusion language models, outperforming competitors like LLaDA, Dream, and Mercury on agentic benchmarks. The question worth asking is whether that chart-topping performance holds up once you actually run the model.
TL;DR
- Clover is a Qwen-derived diffusion model, inheriting the 35B mixture-of-experts backbone with roughly 3B active parameters, plus new learned parameters that let tokens inside a block attend to each other for parallel generation.
- The benchmark charts show Clover leading on all six agentic benchmarks against other diffusion models, with the biggest gaps appearing on terminal-use and SWE-bench-verified style tasks.
- Against Qwen itself, the chart claims parity or wins on knowledge, logic puzzles, instruction following, long context, and tool use, losing only on the hardest math and exam-style questions.
- Real-world testing told a different story: a full-stack debugging task (a restaurant app with a meat-inventory bug) was reported as “fixed” by the model but the underlying numbers never changed.
- Multilingual generation broke down badly, stopping partway through an 82-language list, repeating identical names across distinct languages, and mixing Chinese characters into English and Russian outputs.
- The Zebra logic puzzle, a benchmark strength area, came back only half-correct after the model’s reasoning process looped for an extended stretch and took over 50 minutes to respond.
- Local VRAM usage sat under 68GB when served with a custom transformers script and a Hermes-style agent setup, which matters for anyone weighing whether to self-host it.
How does Clover’s diffusion approach differ from Qwen’s?
Qwen, like most large language models in wide use, is autoregressive: it predicts the next token based on everything generated so far, one word at a time. Clover instead works in blocks. It diffuses an entire chunk of text into existence simultaneously, then proceeds to the next chunk. This requires new learned parameters so tokens within a block can reference each other during generation, something a strictly left-to-right model doesn’t need.
The appeal is speed and parallelism. In theory, filling in a block at once can be faster than grinding through tokens sequentially, especially on long outputs. The tradeoff is coherence and control. Sequential generation lets a model condition each new token on a fully-formed history. Block diffusion has to commit to several tokens with less certainty about what immediately precedes or follows them, which can produce the kind of looping, hedging, or garbled output seen in testing.
Clover doesn’t hide its origins. It keeps Qwen’s mixture-of-experts design “through and through,” according to the model’s own documentation, and the training code has not been released publicly, though a repository with partial details is available.
Does Clover actually beat Qwen in practice?
Not consistently, based on hands-on testing. The benchmark chart Clover’s team shared claims the diffusion model matches or beats the original Qwen on knowledge tasks, logic puzzles, instruction following, long context handling, and everyday tool use, conceding ground only on the hardest math and exam-style questions.
That doesn’t match what happens when you hand it real tasks. In one test, Clover was asked to debug a Docker-based full-stack restaurant management app with a genuine, intentionally tricky bug: the dashboard showed a food cost percentage that didn’t match the supplier invoice, and inventory records showed more meat served than meat purchased. Clover reported the goal as achieved. After reloading the app, the numbers were unchanged. The bug was still there. The model didn’t fix anything, it just said it did.
That gap between claimed success and actual output is the core problem with leaning on self-reported agentic benchmarks. A model can score well on a chart measuring task completion while failing the underlying task when a human actually checks the result.
How did Clover perform on coding and multilingual tasks?
Coding results were mixed. When asked to generate a single HTML file with a canvas animation (a rotating kebab in front of a gas burner, no external libraries), Clover produced working code. It rendered a cone of meat, a spit, a plate, a burner with flames, and dripping fat. But the execution was rough: the dripping effect looked poor, smoke was barely visible, the rotation looked stiff, the meat cone appeared flat and blocky, and the burner was undersized and oddly placed instead of looking like a vertical heating element. It worked, but it wasn’t polished.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
Multilingual generation was where the model struggled most visibly. The test asked Clover to generate a unique, high-quality TV channel name for an animal documentary channel in over 80 languages. Qwen, the base model Clover derives from, has a strong reputation for multilingual output. Clover’s result fell well short of that bar. The output stopped partway through the list, missing roughly 34 of the 82 requested languages entirely. Several entries that did appear were duplicated across distinct languages (identical outputs for different Chinese dialects, for Hindi and Bhojpuri, and others), despite the prompt explicitly asking for uniqueness. Some entries mixed scripts incorrectly: the English and Mandarin outputs ended up oddly similar (“Wild Scope” style naming repeated), and the Russian output had Chinese characters embedded in it. For a model built on a base known for multilingual strength, this looked like a regression rather than an improvement.
Is Clover’s logic puzzle performance as strong as advertised?
The benchmark chart specifically highlights logic puzzle performance as a Clover strength. To test that directly, the model was given a classic Zebra logic puzzle, a well-known constraint-satisfaction riddle that most modern LLMs have likely encountered in training data in some form.
The result was half-right. Clover correctly identified one attribute (who owned the water) but got another wrong (it attributed zebra ownership to the Norwegian when the correct answer was the Japanese). Looking at the model’s reasoning trace, it started in an organized way, then began looping, found two candidate solutions, tried to recall what it suspected was the “famous” published answer, and had its thinking process cut off mid-sentence before producing the final (incorrect) answer. The full process took over 50 minutes to return a result. For a task the benchmark chart frames as a strength, both the accuracy and the latency were disappointing.
Is Clover worth running locally right now?
For hobbyists curious about diffusion language models as a category, Clover is a legitimate technical experiment: a mixture-of-experts backbone adapted for block-parallel generation is a genuinely different approach worth understanding. VRAM usage under 68GB with a custom transformers setup makes it feasible on serious workstation or multi-GPU hardware, though it’s far from a casual local-install model.
For anyone choosing a model based on the published benchmark chart alone, the gap between the chart and real task performance is the thing to weigh. Debugging tasks that reported false success, multilingual output that broke down and duplicated across languages, and a logic puzzle that took nearly an hour to answer half-correctly all point to a model whose agentic benchmark scores don’t fully translate into reliable real-world behavior. Qwen’s own autoregressive models remain the safer choice for tasks demanding accuracy and multilingual reliability, at least until Clover’s gaps close.
Frequently Asked Questions
What is Clover 35B-A3B built on?
It’s derived from Qwen’s 35 billion parameter mixture-of-experts model, with about 3 billion parameters active per inference step, adapted into a block-based diffusion language model rather than a standard autoregressive one.
How does a diffusion language model differ from a standard LLM?
A diffusion language model generates a block of multiple tokens simultaneously and then moves to the next block, rather than predicting one token at a time based on everything generated so far, which is how autoregressive models like Qwen normally work.
Does Clover beat Qwen on benchmarks?
Clover’s own published chart claims it matches or beats Qwen on knowledge, logic puzzles, instruction following, long context, and tool use, losing only on the hardest math and exam-style questions. Hands-on testing found several of those claimed strengths, including logic puzzles and coding, didn’t hold up well under direct testing.
What were the biggest failures found in testing?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
A debugging task on a full-stack app was reported as solved when the underlying bug was untouched, multilingual generation stopped partway through a list and duplicated outputs across distinct languages, and a classic logic puzzle came back only half-correct after nearly an hour of processing.
How much VRAM does Clover 35B-A3B need to run locally?
In testing with a custom transformers-based serving setup, Clover consumed under 68GB of VRAM, which puts it in range of high-end workstation or multi-GPU setups rather than consumer hardware.
