Meta Muse Spark 1.3: Why Its Benchmark Scores Don't Add Up
Muse Spark 1.3 tops the DeepSWE coding benchmark, but hands-on tests show weak real-world output. Here's why the scores don't match reality.
What is Muse Spark 1.3?
Muse Spark 1.3 is a model out of Meta that appeared this week alongside a cluster of other major releases, including Claude Fable 5.1, Gemini 3.8 Flash, and GPT6. On paper, it’s the most impressive of the bunch for coding: it posted a 75.4% score on the DeepSWE benchmark, which would put it above every other model released this week, including Gemini 3.8 Flash (73.7%), Claude Opus 5 (74%), and GPT6 (74.1%). The problem is that when the same model is asked to actually write code for a real project, the output doesn’t look like the work of a top-ranked coding model at all.
TL;DR
- Muse Spark 1.3 scored 75.4% on DeepSWE, technically the highest of any model released this week, yet it doesn’t appear on the public DeepSWE leaderboard at all.
- On Artificial Analysis, Muse Spark ranks third overall, ahead of GPT6 and Gemini 3.8 Flash, a placement that contradicts hands-on coding results.
- A side-by-side coding test (a Megabomb-style game clone) told a different story: Muse Spark produced a bare cube shooting other cubes, while lower-benchmarked models like Gemini 3.8 Flash and GPT6 produced playable, visually complete games.
- Cost is Muse Spark’s one clear advantage: it’s currently free to use through OpenRouter, with an API-equivalent cost around 55 cents per task, in line with Gemini 3.8 Flash.
- On Beauty Bench, an AI-judged SVG image generation test, Muse Spark ranked only 20th overall, far below its DeepSWE and Artificial Analysis placements would suggest.
- The gap raises a broader question about benchmark trust: a model can top a coding leaderboard and still underperform visibly weaker-ranked competitors on an actual coding task.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
How does Muse Spark 1.3 perform on DeepSWE?
DeepSWE is a benchmark that has historically tracked fairly well with how good a model actually feels to use for coding tasks. It’s one of the more trusted signals in a space full of noisy leaderboards. Muse Spark 1.3’s reported score of 75.4% would make it the top coding model available, surpassing Claude Opus 5 and Gemini 3.8 Flash, both of which sit around 74%, and GPT6’s 74.1%.
The catch: Muse Spark 1.3 doesn’t show up on the actual public DeepSWE leaderboard. An earlier version, Muse Spark 1.2, is listed further down, but 1.3’s chart-topping number appears disconnected from the leaderboard it’s supposedly dominating. That disconnect alone is enough to warrant skepticism before even testing the model directly.
How does it perform on Artificial Analysis?
Artificial Analysis is a composite benchmark that blends multiple individual test scores with different weightings to estimate a model’s overall capability. It’s been a reliable reference point for comparing frontier models against each other.
On this benchmark, Muse Spark 1.3 lands in third place, behind Claude Opus 5 and Claude Fable 5.1, but ahead of GPT6 and notably ahead of Gemini 3.8 Flash by a wide margin. Cost per task comes in around 55 cents, similar to Gemini 3.8 Flash’s pricing tier and dramatically cheaper than Fable 5.1’s $3.69 per task.
A model landing above GPT6 and well above Gemini 3.8 Flash on a respected composite benchmark should translate into visibly stronger real-world performance. That’s not what happened in practice.
What happened in the real-world coding test?
To get a practical read on these models, a common test used across releases is having each one build a clone of a simple arcade-style shooter game, referred to as a Megabomb clone. This test isolates a model’s ability to translate instructions into working, playable code with a coherent visual design, rather than just solving isolated coding problems.
The results lined up poorly with the benchmark rankings:
- Claude Fable 5.1 produced a fully playable game with distinct characters, enemy designs, camera angle controls, and reasonable visual polish, though it took around two hours and cost roughly $120 in usage credits to generate.
- Gemini 3.8 Flash produced a noticeably better-looking game than earlier Fable attempts, using only a small fraction of available credits.
- GPT6 produced what came across as the most complete and aesthetically finished game, with multiple character classes (a knight, a ranger, a mage) and convincing enemy designs, generated in about 12 minutes.
- Muse Spark 1.3, the model ranked first on DeepSWE and third on Artificial Analysis, produced a cube shooting at other cubes. No distinct characters, no real visual design, nothing resembling the finished output of the lower-ranked models.
This is the core tension: a model that benchmarks as the best coding model available generated the least functional, least polished result in a direct coding comparison against models it supposedly outranks.
Is Muse Spark 1.3 worth using?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
If cost is the priority, yes, at least for now. Muse Spark 1.3 is currently accessible for free through OpenRouter, and its estimated API cost per task (around 55 cents) is competitive with some of the cheaper frontier options. For lightweight tasks, experimentation, or anything where budget matters more than top-tier output, that’s a real advantage.
For serious coding work, the hands-on test suggests otherwise. A model that can’t produce a functional, visually coherent small game, while cheaper and lower-benchmarked competitors can, isn’t demonstrating the coding strength its DeepSWE and Artificial Analysis scores imply. On Beauty Bench, a separate test that scores SVG image generation (essentially code that draws an image) using an AI judge, Muse Spark 1.3 ranked 20th overall, a result far more consistent with its practical performance than its other benchmark placements.
Why do the benchmark scores and real-world results diverge?
There are a few possible explanations, none of which are confirmed, but all worth considering when a gap this large shows up:
Benchmarks like DeepSWE test specific, narrow coding problems, often algorithmic or bug-fixing tasks, that may not correlate well with a model’s ability to handle open-ended creative coding like building a game from scratch. A model could be tuned or particularly strong on the narrow distribution of problems a benchmark covers without generalizing to broader tasks.
There’s also the possibility of benchmark contamination or overfitting, where a model has effectively seen benchmark-like data during training, inflating its score without a corresponding gain in general capability. This isn’t asserted as fact for Muse Spark 1.3 specifically, but it’s a well-documented risk across the industry whenever a new model’s benchmark score seems disconnected from hands-on impressions.
Finally, composite benchmarks like Artificial Analysis apply weightings across many sub-tests. A model can score well in categories that don’t reflect the kind of coding work most builders actually care about, pulling its overall rank up while its practical coding output lags behind.
Whatever the cause, the lesson for anyone evaluating new models is the same: benchmark leaderboards are a starting point, not a verdict. Running your own test, even something as simple as asking a model to build a small game or app, remains the more reliable way to judge whether a model is actually good at the thing you need it for.
Frequently Asked Questions
What is DeepSWE and why does it matter?
DeepSWE is a coding benchmark used to evaluate how well AI models perform on software engineering tasks. It has generally correlated well with how models feel to use in real coding workflows, which is why a sudden mismatch with hands-on results is notable.
Why isn’t Muse Spark 1.3 on the public DeepSWE leaderboard?
Only an earlier version, Muse Spark 1.2, appears on the public leaderboard. Muse Spark 1.3’s reported 75.4% score, which would top the leaderboard, doesn’t appear listed alongside the other ranked models, making the figure difficult to independently verify.
How does Muse Spark 1.3 compare in cost to other models?
It’s currently available for free through OpenRouter, with an estimated API-equivalent cost of about 55 cents per task, similar to Gemini 3.8 Flash and much cheaper than Claude Fable 5.1’s $3.69 per task.
Does a high benchmark score guarantee good real-world coding performance?
One coffee. One working app.
You bring the idea. Remy manages the project.
Not necessarily. Muse Spark 1.3’s top DeepSWE score and third-place Artificial Analysis ranking didn’t translate into strong results on a hands-on game-building test, where it underperformed models ranked below it on both benchmarks.
What should builders do instead of relying on benchmarks alone?
Run direct comparisons on tasks relevant to your own use case, such as building a small app or feature, before choosing a model based on leaderboard position. Benchmarks are useful signals but can diverge sharply from practical output.
