GPT-6 Astra Benchmarks Explained: What the Scores Really Mean
GPT-6 Astra's scores on Terminal Bench, Frontier Math, and ARC-AGI-3 explained, and why these benchmarks measure real capability, not marketing.
What do GPT-6 Astra’s benchmark scores actually measure?
GPT-6 Astra’s headline numbers on Terminal Bench Science, Agent’s Last Exam, Frontier Math Tier 4, and ARC-AGI-3 measure whether a model can plan, verify, and execute multi-step work against tasks it has never seen, with wrong answers actively penalized rather than just ungraded. These aren’t trivia quizzes. They’re built from real scientific datasets, industrial software, and math problems that professional mathematicians expected AI to score near zero on. The jump in Astra’s scores over the past year reflects a genuine change in what these systems can do unsupervised, not just a marketing framing of old capabilities.
TL;DR
- Terminal Bench Science tests models on real research tasks, like finding a periodic dip in 25,000 stellar brightness readings or diagnosing lake drainage from nearly 3 GB of satellite imagery, with penalties for wrong hypotheses.
- Agent’s Last Exam, built by UC Berkeley, covers thousands of expert-curated tasks across 55 industries, including machining software where a single detected collision drops the score to zero.
- Frontier Math Tier 4 was designed to be nearly unsolvable. A year ago, models scored close to zero and a mathematician said he hoped AI would score zero on it; Astra reportedly hit 98% with reasoning enabled and 83% without any chain of thought.
- ARC-AGI-3 shows Astra solving novel game levels using fewer actions than the human baseline that successfully completed them, reversing what the benchmark’s creators expected to be a durable human advantage.
- Hallucination rates dropped sharply compared to prior OpenAI models in tests replaying situations where earlier models had confabulated with real users.
- Cost and token efficiency matter alongside raw scores: Astra reportedly beats rival models on several benchmarks while using fewer tokens per task, lowering effective API cost.
- Not every third-party benchmark favors Astra uniformly, which is itself useful signal that these scores reflect real differences in task design rather than a single model sweeping everything.
Why does Terminal Bench Science matter more than it sounds?
The name is forgettable, but the tasks are not trivial. Terminal Bench Science hands a model messy, large real-world datasets, tens of thousands of rows of stellar brightness data, gigabytes of daily satellite images of lakes in Greenland, or hundreds of MRI scans, and asks it to write code that extracts a specific, verifiable scientific finding. The model has to identify a repeating signal in noisy data, reason through partial obstructions like cloud cover across a multi-month season, or measure and label an injury on a medical image. Crucially, it’s graded on held-out data it never saw and penalized for incorrect hypotheses, so guessing is expensive. A model doing “better than expert” on this kind of task for a few dollars in compute is a different claim than doing well on a static multiple-choice exam.
What makes Agent’s Last Exam a harder test than it sounds?
Agent’s Last Exam, designed at UC Berkeley, deliberately avoids abstract puzzles in favor of economically grounded work. It spans thousands of tasks across 55 industries with outcomes that can be objectively verified. One example has the model operating industrial machining software, where a hidden grading system samples 10,000 points on a reference surface and requires critical points to land within 0.3 millimeters, with any detected tool collision zeroing the score outright. Another task involves molten plastic processing parameters. Another has the model reconstructing maps, monsters, and battle layouts from an older video game, judged floor by floor by a separate vision model. According to reporting on the benchmark, Astra set a new state-of-the-art score here while also using fewer tokens than rival models, which brings down the effective cost per task even when list prices look similar.
Why did Frontier Math Tier 4 go from near-zero to near-solved?
Frontier Math, built by Epoch AI, was explicitly designed to be brutally hard. Its hardest tier, Tier 4, drew a comment from a mathematics professor at the time who said he could barely solve some of these problems himself and hoped AI systems would score zero. For a while, they basically did. Gemini 2.5 Pro scored near zero, and GPT-5, released about a year prior, reportedly landed in the 10 to 20% range. GPT-6 Astra’s reported score of 98% with reasoning, and 83% with no chain-of-thought scratchpad at all, represents a genuine step change rather than incremental tuning. A separate claim, attributed to an assistant professor at Stanford, describes the model improving on a known result about prime number gaps by a full order of magnitude, associated with mathematician Terence Tao’s prior work, and introducing a new starting approach to a problem that had been open since the 1930s. That kind of result, if it holds up under scrutiny, matters more than a percentage score because it points to actual mathematical contribution rather than pattern-matching against known problem types.
Is ARC-AGI-3 the most meaningful benchmark here?
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
It might be, because of what it was built to prevent. ARC-AGI-3 places models in game-like environments built from abstract patterns the model has never encountered, designed specifically to catch models that can’t reason on the fly. Earlier models famously failed this kind of test inefficiently, burning through random actions without making progress. The benchmark’s creators measured a “human baseline”: the average number of actions taken by humans who successfully solved each level. Astra reportedly beat that human baseline roughly 96% of the time, using about 50% fewer actions on average. The benchmark’s own creators had hypothesized, before testing, that action efficiency would remain a lasting dividing line between humans and AI. That hypothesis didn’t hold. François Chollet, the researcher behind the ARC-AGI series, has said the goal is to keep building harder versions until no task remains that humans can do and AI can’t, and has suggested that timeline may be arriving sooner than he expected.
Does a better benchmark score mean fewer hallucinations?
Not automatically, but in Astra’s case the two reportedly moved together. OpenAI tested the model against specific situations where earlier models had hallucinated in real user interactions, and Astra reportedly produced substantially fewer hallucinations in the same scenarios, described as somewhere between a three-fold and ten-fold reduction depending on the case. That matters because higher benchmark scores sometimes come from a model that reasons more confidently while still fabricating details elsewhere. A drop in hallucination rate alongside a jump in verified task performance is a stronger signal than either number alone.
How do outside companies’ internal benchmarks fit in?
Third-party validation outside OpenAI’s own test suite adds useful texture. Cognition AI reportedly rated Astra as state-of-the-art for coding compared to Anthropic’s Claude Opus 5.1. Jane Street, evaluating models on internal trading and coding benchmarks, reportedly found Astra ahead on coding tasks specifically, while judging Claude’s “trading intuition” as a clearer step forward for that narrower skill. That’s a meaningful distinction: it suggests real differentiation between models by task type rather than one model uniformly dominating every category. Similarly, on GDPval, a benchmark tracked by Artificial Analysis and originally built by OpenAI, results reportedly varied enough that Astra on some settings scored lower than smaller models from other labs on that particular test, which is a useful reminder that no single benchmark tells the whole story.
Frequently Asked Questions
What is GPT-6 Astra?
GPT-6 Astra is OpenAI’s latest flagship model, positioned as a major capability jump rather than an incremental update, based on reported performance across science, coding, math, and agentic benchmarks compared to rivals like Anthropic’s Claude Opus 5.1.
Why do these benchmarks matter more than older ones like MMLU?
Newer benchmarks like Terminal Bench Science, Agent’s Last Exam, and ARC-AGI-3 use verifiable, expert-curated, real-world tasks with penalties for wrong answers, rather than static multiple-choice questions. That design makes it harder for a model to score well through memorization or lucky pattern-matching.
Did GPT-6 Astra really score 98% on Frontier Math Tier 4?
That figure comes from reporting on the model’s release and refers to performance with reasoning enabled. Without a reasoning scratchpad, the reported score was around 83%, still a large jump from prior models that scored near zero on this tier a year earlier.
Is GPT-6 Astra available to everyone?
According to reporting at launch, Astra rolled out first to API customers and Pro subscribers, with Plus subscribers gaining access afterward. Free-tier users were not given access at launch.
Does beating a benchmark mean the model has fewer flaws overall?
No. Reported testing still shows the model hallucinating, just less often than prior models in comparable scenarios, and it reportedly still has quirks around memory and long-conversation reliability. Benchmark wins indicate capability gains on specific measured tasks, not the elimination of all failure modes.
