Kimi K3.1 Pro Benchmarks: Does It Really Beat GPT-6 Astra?
Kimi K3.1 Pro claims to beat Opus 5.5 and GPT-6 Astra on paper. Independent tests tell a more complicated, cheaper story.
What is Kimi K3.1 Pro?
Kimi K3.1 Pro is a large language model from the Chinese AI lab Moonshot AI, released as an upgrade to Kimi K3 just three months after that model launched. Moonshot’s own benchmarks claim it matches or beats Claude Opus 5.5 and GPT-6 Astra on several hard tests, including Humanity’s Last Exam. Independent testing paints a less dramatic picture: strong for an open-weight model, mid-pack against the actual frontier, and notably cheap to run.
TL;DR
- Moonshot’s own charts show Kimi K3.1 Pro beating Claude Opus 5.5 and GPT-6 Astra on tests like Humanity’s Last Exam (70% vs 67.7% and 57.2%) and SWE Marathon, but nearly all of those numbers come from benchmarks Moonshot ran and in some cases designed itself.
- Independent rankings are more modest. Artificial Analysis puts it fourth overall with an intelligence score of 55, three points behind Opus 5.5’s 58, while Arena’s web dev leaderboard ranks it eighth, up from 17th for the previous Kimi K3.
- A hands-on coding benchmark (KingBench) scored it 53.7%, putting it sixth out of eleven models, behind Opus 5.5, GPT-6 Astra, GPT-6.1 Soul, Gemini 4 Argon, and even Claude Haiku 5.5.
- The model is enormous but efficient: roughly 2.8 trillion total parameters using a mixture-of-experts design that activates only about 104 billion per token, plus a 1 million token context window and image understanding.
- Pricing undercuts the competition hard: $2 per million input tokens and $10 per million output tokens through Moonshot’s API, versus roughly $4/$20 for Opus 5.5 and a reported $10/$50 for GPT-6 Astra.
- Real-world coding tests exposed gaps, including an emulator that only built one of three requested systems and a “Docker alternative” that can’t actually build or run containers, though the model was notably honest about these shortcomings rather than faking success.
- Open weights are expected by November 4, which would make it the highest-scoring open-weight model on at least one independent index, narrowing the gap between open and closed models from 12 points to roughly 3.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
How does Kimi K3.1 Pro actually work?
The model uses a mixture-of-experts architecture, meaning it has a very large total parameter count but only activates a fraction of it for any given piece of text. Out of roughly 2.8 trillion total parameters, around 104 billion are active per token. Think of it like a hospital with hundreds of specialists: a patient only sees the few doctors relevant to their condition, not the entire staff. This keeps inference costs and speed closer to what you’d expect from a much smaller model, despite the overall size being one of the largest ever made available outside the top American labs.
It also supports a 1 million token context window, roughly equivalent to several books worth of text in one conversation, and can process images alongside text. Moonshot ships it across several products: Kimi Work for office tasks, Kimi Code for programming (an open-source coding agent comparable to Claude Code or OpenAI’s Codex), and a browser extension. There’s also a “High Speed” variant that runs two to three times faster at roughly double the price.
What do Moonshot’s own benchmarks claim?
Moonshot’s launch materials lean heavily on comparisons to Claude Opus 5.5 and GPT-6 Astra, the flagship paid models from Anthropic and OpenAI. On Humanity’s Last Exam, a notoriously difficult expert-written test, Moonshot reports Kimi K3.1 Pro scoring 70% with tools, ahead of Opus 5.5 (67.7%) and well ahead of GPT-6 Astra (57.2%). On SWE Marathon, a long-horizon software engineering benchmark, it reports 51.7% versus 46.3% and 42.5% respectively. On Terminal Bench, Moonshot says the score jumped from 13% for K3 to 65.7% for K3.1 Pro, putting it close to Opus 5.5’s 66.4%.
Moonshot also highlights finance-specific work: a 65% reduction in hallucination rate compared to K3, and strong scores on an internal “Office QA Pro” benchmark aimed at real business documents and spreadsheets. The hallucination numbers are genuinely notable for financial use cases, where a fabricated figure in a model can be expensive, though the comparison numbers for competitors (Opus 5.5 at 6.8%, GPT-6 Astra at 4.2% on the same internal test) still show GPT-6 Astra ahead on factual reliability.
The catch is that almost all of these figures come from tests Moonshot ran itself, and several are benchmarks the company designed internally rather than standardized, third-party evaluations.
Is Kimi K3.1 Pro bench-maxed?
“Bench-maxed” describes a model that performs well on commonly used test sets, whether through deliberate optimization or incidental overlap with training data, but underdelivers in actual use. It’s a frequent accusation leveled at Chinese open-weight models, and it’s exactly the skepticism that greeted Kimi K3.1 Pro’s launch.
Two independent sources offer a reality check. Artificial Analysis, which runs its own cross-model intelligence index, scored Kimi K3.1 Pro at 55, an 11-point jump from K3, placing it fourth overall behind Opus 5.5 (58) and tied with Claude Sonnet 5.5 and Meta’s Muse Wonderful (56 each). That’s a strong result, just three points off the top.
Arena’s web development leaderboard, built from blind human preference votes, ranked it eighth, behind Opus 5.5, GPT-6 Astra, Sonnet 5.5, GPT-6.1 Soul, Claude Fable 5.1, Muse Wonderful, and another Sonnet variant. It’s a real jump from 17th place for the original K3, but it’s not the top-tier result Moonshot’s marketing implies.
A more hands-on evaluation using a coding benchmark called KingBench, which ran the model through six practical software tasks (a retro emulator, a container runtime, a 3D room designer, a keyboard configuration app, a Pokémon game clone, and a fitness tracking app) scored it 5.37 out of 10, or 53.7%. That put it sixth out of eleven models tested, behind Opus 5.5 (5.78), GPT-6 Astra (5.65), GPT-6.1 Soul (5.50), Gemini 4 Argon (5.43), and Claude Haiku 5.5 (5.42). Moonshot’s blog had claimed a seven-point lead over GPT-6 Astra; the independent test showed GPT-6 Astra clearly ahead instead.
So: somewhat bench-maxed, but not fabricated. The model is genuinely capable, just not quite at the level its own marketing suggests.
What did real coding tests reveal?
The KingBench evaluation surfaced a consistent pattern: Kimi K3.1 Pro tends to build a solid foundation but doesn’t finish the full scope of what’s asked. Given a task to build a Game Boy, DS, and 3DS emulator in Rust, it only implemented the Game Boy portion, though notably it was upfront about this rather than faking support for the other two systems, and the Game Boy core it did build passed all classic CPU instruction tests. A “Docker alternative” project correctly pulled and verified container images (even blocking 20 deliberately malicious ones) but had no actual build, exec, or networking functionality, despite the readme claiming otherwise.
Its best result came on a keyboard configuration app, where it correctly implemented a real hardware protocol (VIA/QMK) that most models get wrong entirely. Its weakest areas involved tasks requiring broader scope or polish, like a room designer that functioned well mechanically but didn’t look realistic, and a Pokémon clone with broken leveling and a save-wiping bug on the title screen.
The honesty about limitations lines up with Moonshot’s claimed hallucination improvements. The model seems to avoid confidently faking functionality it didn’t build, which is a meaningful trait even when the underlying work is incomplete.
Is Kimi K3.1 Pro worth it on price?
This is where the model’s case gets stronger. Through Moonshot’s API, Kimi K3.1 Pro costs $2 per million input tokens and $10 per million output tokens, a roughly one-third price cut from K3. Claude Opus 5.5 runs about $4 input and $20 output, making Kimi roughly half the price. GPT-6 Astra is reportedly around $10 input and $50 output, putting Kimi at roughly a fifth of the cost.
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
But cheap tokens don’t automatically mean a cheap bill. The KingBench run found that Kimi K3.1 Pro is a “chatty” agent that rereads large amounts of cached context on every step. Across six tasks using about 112 million tokens, the total run cost roughly $32, with the bulk of that coming from repeatedly rereading cached context rather than from generating new output. Compared against Claude Haiku 5.5 (a similarly priced model that scored slightly higher for about a quarter of the cost) and DeepSeek V4.1 Flash (which scored close behind for under $2), Kimi K3.1 Pro didn’t come out as the economical choice in pure API terms, despite its low headline price.
Where the pricing argument holds up better is through a Kimi subscription, where per-token costs aren’t the limiting factor, or once open weights are available and the model can be self-hosted, likely by cloud providers running racks of GPUs given its 2.8 trillion parameter scale.
Frequently Asked Questions
Does Kimi K3.1 Pro actually beat GPT-6 Astra?
On Moonshot’s own benchmarks, yes, in several categories. On independent tests, no: Arena’s leaderboard and a hands-on coding benchmark both placed GPT-6 Astra ahead of Kimi K3.1 Pro, though the gap isn’t enormous.
What is bench-maxing and does it apply here?
Bench-maxing refers to a model being tuned to score well on popular benchmarks without matching that performance in real use. Kimi K3.1 Pro shows some signs of this gap between Moonshot’s internal numbers and independent evaluations, but it’s not a hollow result. Independent tests still place it as a genuinely competitive, if not top-tier, model.
How big is Kimi K3.1 Pro?
It has around 2.8 trillion total parameters but uses a mixture-of-experts design that activates only about 104 billion parameters per token, which keeps it faster and cheaper to run than its full size would suggest.
When are the open weights released?
Moonshot has said the full K3.1 Pro weights will be available by November 4, under its own license (similar to MIT with added conditions for very large companies). The company has also said it removed cyber-related training data from the open release, so the downloadable version may not exactly match the hosted API version.
Is Kimi K3.1 Pro a good choice for developers right now?
It depends on the use case. Paying per token through the API, cheaper or similarly priced alternatives like Claude Haiku 5.5 or DeepSeek V4.1 Flash scored competitively or better in independent testing. It looks more attractive on a flat-rate Kimi subscription, or once open weights allow self-hosting, where it’s positioned to be one of the strongest open-weight models available.
