GPT-6 Astra Aced a Physics Test by Catching the Experts' Own Mistakes
GPT-6 Astra scored 98% on the TheoPhysEval physics benchmark after experts found their answer key, not the AI, had the errors.
What happened with GPT-6 Astra and the physics benchmark?
A group of researchers from institutions including Harvard, the University of Texas, Google DeepMind, and top Indian research institutes built a new benchmark called TheoPhysEval: 50 unpublished, research-level theoretical physics problems, each roughly a day’s work for a world-class expert. When they tested OpenAI’s GPT-6 Astra (run inside the Codex coding environment, with access to paper search and self-written verification scripts), the model appeared to score around 88%. On closer review, several of its “wrong” answers turned out to be correct. The official answer key had subtle errors that the model’s own reasoning had caught. After the test-writers fixed their solutions, Astra’s score rose to 49 out of 50, or 98%.
TL;DR
- TheoPhysEval is a 50-question theoretical physics benchmark built from brand-new, unpublished problems specifically so no model could have memorized the answers during training.
- GPT-6 Astra, running in OpenAI’s Codex tool with paper-search and code-verification access, was initially scored at around 88% before corrections.
- After researchers consulted the physicists who wrote the questions, they found the official answer key was wrong in several cases, and Astra’s original answers were correct.
- Once corrected, Astra’s score jumped to 49 out of 50 (98%), though it still failed at least one problem outright and performed worse when cut off from its tools.
- A similar pattern showed up on Frontier Math, a hard benchmark from Epoch AI, where an AI-assisted review reportedly flagged fatal errors in roughly a third of the problems.
- A separate study using GPT-6.1 “Soul” scanned nearly 800 math papers for false claims and found major errors in about 6% of older papers, a rate that dropped sharply in papers from recent months.
- The core implication is that some benchmark scores may have understated real AI capability, because models were being marked wrong against faulty answer keys.
Why does it matter that the answer key was wrong?
Benchmarks only work if the ground truth is reliable. TheoPhysEval was designed with that in mind: every problem is new and unpublished, so a model can’t have seen the solution anywhere during training. That rules out the usual objection that a model is just pattern-matching to something it memorized.
But the researchers found a different vulnerability. Even world-class physicists make mistakes on research-level problems, the kind where a single sign error can silently invalidate pages of derivation. According to the paper, the team “initially assessed that Astra had made errors in several additional problems,” but after consulting the experts who authored those questions, they found “subtle errors in the original solutions.” In every one of those cases, once the expert solution was corrected, Astra’s original answer was right.
That matters beyond this one test. If an answer key is wrong, every model graded against it, not just GPT-6 Astra, was scored unfairly. A model that produced the correct answer would have been marked down. That means published scores on flawed benchmarks could be systematically underestimating how capable some models actually are. It also flips the usual assumption that human-authored answer keys are the trustworthy baseline against which AI is judged.
The TheoPhysEval paper is careful not to oversell this: Astra still failed some problems, performance dropped when it lost access to its tools, and at least one problem went unsolved by every model tested. The benchmark isn’t “solved.” But catching expert-level errors on unpublished, research-grade physics is a different kind of result than simply getting a high score.
Is this pattern showing up outside physics too?
Yes, and it predates the physics story. OpenAI researcher Noam Brown pointed to an earlier, similar episode involving Frontier Math, a benchmark built by Epoch AI using problems from professional mathematicians designed to take hours or days to solve. When it launched in late 2024, leading models solved less than 2% of its problems, and it was pitched as a test that should hold up for years.
By mid-2025, Epoch AI reported running an AI-assisted review of Frontier Math and finding that the review had flagged fatal errors in roughly a third of the problems, with the team believing most of those flags were valid. That’s a dramatic failure rate for a benchmark built specifically to be rigorous and hard to game.
A related but distinct effort looked at published research rather than benchmark questions. Researchers led by Jasper Dekoninck and Martin Vechev used GPT-6.1 “Soul” to check nearly 800 math papers for false claims, using a method based on counterexamples: if a paper claims something is always true, the model only needs to find one case where it fails. The process reportedly cost around $2,500 in compute for hundreds of papers, far cheaper than hiring human reviewers to do equivalent spot-checks. The team found that around 6% of papers from before August had major errors serious enough to undermine their central claim, and another 30% had minor, fixable mistakes. Human researchers checking the AI’s flags estimated the model’s major-error calls were correct a bit less than 90% of the time. Interestingly, papers from more recent months, when AI tool usage was more common among authors, showed a major-error rate closer to 1.5%, though the researchers stopped short of attributing that drop entirely to AI assistance.
How are scientists actually using this in practice?
Benchmarks aside, individual researchers describe a more direct shift. Gabriele Carcassi, a physicist at the University of Michigan, has talked about a conjecture related to topologically convex spaces that he’d been stuck on for a year, one a collaborating mathematician had tried and failed to resolve over six months of work. He gave the problem to an AI model and reportedly got a correct answer in about two minutes, using a mathematical technique (the tube lemma) he hadn’t previously applied to it. His own framing is useful: the AI isn’t a flawless genius and everything it produces still needs checking, but its breadth of knowledge across fields lets it make connections that a single human specialist might miss.
A higher-profile example came from OpenAI, which said one of its models produced a proof for the cycle double cover conjecture, a roughly 50-year-old open problem in graph theory, reportedly using 64 parallel AI agents and finishing in under an hour. Mathematician Thomas Bloom reviewed the proof and called it short, elementary, and plausible as something that could have been found decades earlier, suggesting humans may have abandoned the right approach too early while the model simply kept grinding through variations.
That said, the picture isn’t uniformly positive. When OpenAI later released a large batch of AI-generated math papers at once, the reaction from mathematicians was mixed: some treated it as a milestone, others were critical, and a handful of papers had to be retracted due to errors. The capability is real, but so is the need for human verification at scale.
How do you test an AI that might be better than the people grading it?
This is the uncomfortable question the TheoPhysEval result raises. Normally, benchmark quality depends on the test being harder than the model, not harder than the people who wrote it. When a model’s “wrong” answer turns out to be right, you need experts capable of recognizing that, which gets harder as problems move further into specialized research territory.
One proposed direction is leaning on problems with automatically verifiable answers: code that either runs correctly or doesn’t, or formal math proofs checked step by step by verification software with no ambiguity. That removes the dependency on a human-authored answer key entirely. It’s part of why benchmark saturation, where a new hard test gets beaten within months instead of years, has become a recurring theme in AI evaluation, and why some researchers are now treating AI-assisted auditing of benchmarks and papers as a standard step rather than an afterthought.
Frequently Asked Questions
What is TheoPhysEval?
It’s a benchmark of 50 unpublished, research-level theoretical physics problems built by researchers from institutions including Harvard, UT, and Google DeepMind, designed to test AI models on problems a world expert would need about a day to solve.
Did GPT-6 Astra actually beat human physicists?
Not exactly. It outperformed the benchmark’s original answer key, which contained errors that the physicists who wrote the questions later confirmed and corrected. Astra still failed some problems and performed worse without tool access.
What is Frontier Math and how is it related?
Frontier Math is a separate, very difficult math benchmark from Epoch AI. An AI-assisted review of it reportedly flagged fatal errors in about a third of its problems, a pattern similar to what happened with TheoPhysEval.
Does this mean AI benchmark scores have been wrong all along?
It means some scores may have underestimated real capability, since models that gave correct answers could have been marked wrong against faulty answer keys. It doesn’t mean all benchmarks are unreliable, but it does raise doubts about how answer keys for very hard, expert-level tests get validated.
Is AI actually helping reduce errors in published research?
There’s early evidence pointing that way. A study scanning nearly 800 math papers found a lower rate of major errors in more recent papers, coinciding with higher reported AI tool use among authors, though the researchers were cautious about claiming direct causation.
