Open-Weight Models vs Claude and GPT for AI Coding: Who Actually Wins?
Open models match Claude and GPT on short coding tasks but stall on long agentic workflows. Here's what the ticket-by-ticket data shows.
Open models win on short tasks and lose on long ones. That’s the actual split.
On short, well-scoped coding tasks, open-weight models like Kimi K3.1 Pro perform about as well as closed frontier models, often at a quarter of the cost. On long-running agentic coding workflows, where a model has to plan, build, review, and self-correct across dozens of steps in a real codebase, open models fall apart. A controlled test building the same game across three AI coding setups found Claude finished all 12 tickets in 14 hours, GPT finished 11 of 12 in roughly 28 hours, and Kimi K3.1 Pro completed only 5 of 12 in 44 hours, repeatedly stalling and violating explicit project rules.
TL;DR
- Short tasks are now a toss-up: for discrete, well-bounded coding jobs, open models match frontier models closely enough that the cost difference makes open models the better choice for most routine work.
- Long-horizon agentic coding is where the gap shows up: once a workflow requires sustained adherence to rules, planning context, and multi-step review across hours of work, open models lose track of constraints that closed models hold onto.
- A head-to-head build test quantified the gap: Claude Opus 5.5 completed 12 of 12 tickets in about 14 hours, GPT6 Astra completed 11 of 12 in roughly 28 hours, and Kimi K3.1 Pro completed only 5 of 12 in 44 hours.
- Open models can end up more expensive in practice: Kimi’s repeated rule violations forced so much re-iteration and review-cycle rework that total token spend exceeded what Claude used, despite Kimi’s lower per-token price.
- Benchmark choice matters a lot: Kimi K3.1 Pro Max edges out Claude Opus 5.5 and GPT6 Astra on Terminal Bench 4.0, but on long-horizon benchmarks like Terminal Bench Science and Frontier Code 2.0, the gap between open and closed models widens sharply.
- Even frontier models aren’t solved for long-horizon work: one cited study put GPT 5.6 Sol at 41.6% on a long-horizon benchmark versus Kimi K3 at 22.1%, a real gap, but neither number represents mastery.
- Analysts expect open models to take over most routine agentic steps within a few years: estimates circulating in the industry suggest open-weight models could handle roughly 90% of enterprise agentic workflow steps within two to five years, mainly for short, repeatable steps rather than end-to-end complex builds.
Nothing gets lost.
Why do open models do well on benchmarks but struggle in real use?
Benchmarks like Terminal Bench 4.0 tend to measure performance on tasks that are complex but bounded: a single hard problem with a clear success condition, finished in one pass or a short loop. Open models like Kimi K3.1 Pro, GLM, and DeepSeek variants can match or beat frontier models on exactly this kind of task, which is why their release benchmarks look so strong.
Real agentic coding work looks different. A typical workflow involves planning a feature against a larger codebase, implementing it under a set of standing rules, having a separate pass review and play-test the result, and only then merging. That loop might run for an hour or more per ticket, and a real project runs dozens of these loops back to back. The model has to keep track of project-specific constraints (what files are read-only, what the review bar is, how the ticket fits into the broader plan) across all of it.
This is where the disconnect shows up. Benchmarks that specifically test long-horizon performance, like Terminal Bench Science and Frontier Code 2.0, show a much bigger gap between open and closed models than headline benchmarks suggest. That gap lines up with what shows up in hands-on testing: models that look competitive on a single hard problem lose the thread once the task stretches across many steps and much more context.
How big is the gap in practice?
A controlled test built the same application (a small video game called Northstar, broken into 12 tickets covering a full game loop) using three separate setups: Claude Opus 5.5 for planning and review with Sonnet 5.5 as the workhorse model, GPT6 Astra for planning and review with GPT 6.1 Sol as the workhorse, and Kimi K3.1 Pro running the same full process end to end. Each setup used an identical harness: a structured workflow of planning, implementation, review, play-testing, and merge, with the same starting assets and rules.
The results:
- Claude: 12 of 12 tickets completed in about 14 hours, averaging roughly an hour per full planning-build-review-merge cycle. The resulting build had a working combat loop and solid physics.
- GPT: 11 of 12 tickets completed in roughly twice the time Claude took. The build looked rougher (leftover hitboxes, an unfinished HUD) but the core loop worked.
- Kimi K3.1 Pro: 5 of 12 tickets completed in 44 hours, with multiple multi-hour stalls. Features like the between-raid upgrade shop, present in both the Claude and GPT builds, never got built because so much time went into re-iterating on earlier tickets.
The recurring failure mode with Kimi wasn’t bad code quality in isolation. It was an inability to consistently follow standing project rules, things as basic as “don’t touch this read-only file.” Each violation triggered a review rejection and another iteration cycle, which ate tokens and time without producing forward progress. Over tens of millions of tokens of testing, that pattern repeated enough that Kimi ended up costing more in practice than Claude, despite a much lower per-token price.
Is it worth using open models for coding at all?
Yes, for a specific category of work. For short, well-defined tasks, including most individual bug fixes, scoped feature edits, and discrete steps in a larger business automation, open models perform at parity with frontier models for a fraction of the cost. That makes them the practical default for high-volume, routine agentic work, especially in settings where cost per token matters or where data needs to stay on local infrastructure.
The weak point is specifically long-running, multi-step agentic coding where a model has to drive an entire build process: planning against an existing codebase, implementing within a rule set, reviewing its own work, and iterating until it clears a real quality bar. For that kind of work, closed frontier models currently hold a clear and measurable edge, and it isn’t close enough to ignore if the goal is a working, polished result rather than a rough prototype.
What does this mean for how teams should mix models?
The practical pattern that falls out of this is tiering by task length and complexity rather than picking one model family for everything. Routine, short agentic steps, the kind that make up most individual actions inside a larger business workflow, are a good fit for open models today, and that share is likely to keep growing. Long-horizon, high-stakes builds where a single agent or small team of agents has to sustain context and rule adherence over many hours still favor frontier closed models, even accounting for their higher per-token cost.
This also isn’t a static picture. Open models have closed ground on short tasks faster than most expected, and industry estimates suggest they’ll keep taking share of routine agentic workflow steps over the next few years. The long-horizon gap is the part that hasn’t closed yet, and it’s also the part that benchmarks optimized for single-shot problem solving don’t capture well.
Frequently Asked Questions
What is the main weakness of open-weight models in AI coding?
Their main weakness is sustaining attention to rules and context across long, multi-step agentic workflows. They perform well on individual, bounded coding tasks but lose track of project constraints (file restrictions, review standards, planning context) as a task stretches across many hours and steps.
Does Kimi K3.1 Pro beat Claude or GPT on any coding benchmarks?
Yes. On Terminal Bench 4.0, Kimi K3.1 Pro Max scored slightly better than both Claude Opus 5.5 and GPT6 Astra. The gap reverses on benchmarks focused on long-horizon tasks, like Terminal Bench Science and Frontier Code 2.0, where closed models perform notably better.
Are open models actually cheaper for coding in practice?
For short, discrete tasks, yes, often significantly so. For long-running agentic builds, the cost advantage can disappear or reverse: repeated rule violations and review failures force extra iteration cycles that consume more total tokens than a closed model would have used to finish the same work correctly.
How much better are frontier models at long-horizon coding tasks?
One cited benchmark study put GPT 5.6 Sol at 41.6% versus Kimi K3 at 22.1% on a long-horizon task set, a roughly two-to-one gap. Frontier models still aren’t considered solved for long-horizon work either; they just degrade less than open models do.
Will open models eventually close this gap?
Industry estimates suggest open-weight models could handle a large majority of routine enterprise agentic workflow steps within the next few years, driven by cost pressure and data-sensitivity concerns that favor local or open deployment. That shift applies mainly to short, repeatable steps rather than full end-to-end long-horizon builds, where closed models currently hold a clearer advantage.
