The 'Stolen Thoughts' Paper: How AI Chain-of-Thought Gets Leaked
Researchers found a way to pull raw chain-of-thought reasoning from proprietary LLM APIs, exposing credentials and enabling model distillation.
What is the “Stolen Thoughts” paper about?
Researchers published a paper, informally nicknamed “Stolen Thoughts,” describing a method for extracting the raw, unfiltered chain-of-thought reasoning that large language models generate internally before producing an answer. Normally, that reasoning is hidden or summarized before it reaches the user. The paper shows this protection can be bypassed without directly attacking the model or triggering the anti-distillation safeguards that AI labs have built to stop competitors from copying their reasoning patterns.
TL;DR
- The raw chain-of-thought that reasoning models generate is usually filtered before users see it, but researchers found a way to recover the unfiltered version straight from an API.
- Raw reasoning traces have shown up containing real credentials, including emails, passwords, API keys, and personal identifiers, which is one reason labs keep them hidden.
- The technique reportedly works without triggering anti-distillation defenses, meaning it doesn’t attack the stronger model head-on or set off the usual protections.
- Prefilling a smaller model’s reasoning with a small slice of tokens from a stronger model’s output appears to nudge the smaller model’s answers toward the stronger model’s style, hinting at possible distillation.
- The findings suggest anti-distillation protections may not be reliable, since researchers say they’ve warned labs including Anthropic about the vulnerability.
- The story surfaced alongside other AI news, including Claude making unusual progress on the Riemann hypothesis’s lower bound and Cursor’s new “Grok bot” agent running on its own virtual machine.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Why do AI companies hide raw chain-of-thought in the first place?
Reasoning models like the “thinking” variants of Claude, GPT, and Gemini generate an internal scratchpad before writing a final answer. That scratchpad is the chain-of-thought: a step-by-step trace of the model working through a problem. What most users see in a chat interface is not that raw trace. It’s a cleaned-up, summarized version.
There are three main reasons labs keep the raw version private.
First, safety. Unfiltered reasoning can be messy, weird, or say things the model wouldn’t want to state directly in an answer. Letting people see it unfiltered risks confusing or misleading users, or exposing reasoning that looks alarming out of context even when the final answer is fine.
Second, security. Raw reasoning traces have been shown to contain things like email addresses, passwords, credit card numbers, dates of birth, and API keys. If a user pastes sensitive information into a prompt, the model may reference it while “thinking out loud” internally, even though it correctly withholds that information from the final visible response. One example: a model working through how to respond to a user who pasted an API key explicitly reasoned about not repeating the key back, which is exactly the kind of internal step nobody wants exposed in a leak.
Third, and maybe most commercially important: distillation. Chain-of-thought traces reveal how a model thinks, not just what it answers. If a competing lab can see that reasoning, they can train their own, cheaper model to imitate it. That’s a shortcut to matching a frontier model’s capabilities without the enormous cost of training one from scratch.
How does the chain-of-thought extraction actually work?
The paper doesn’t rely on a traditional jailbreak or a direct attack on the target model. Instead, it works around the summarization layer that sits between the raw reasoning and what the API returns to a user, extracting the underlying trace without tripping the safeguards designed to detect distillation attempts.
One of the more revealing techniques described involves prefilling a smaller model’s reasoning process with a tiny slice, around one percent of the tokens, from a stronger model’s chain-of-thought. That small nudge appears to shift the smaller model’s final answers toward the style and conclusions of the stronger model, even though the actual answer text itself was never directly copied.
This was demonstrated using Kimi K2, a model from the Chinese lab Moonshot AI, prefilled with a fragment of reasoning from Anthropic’s Opus model. The result skewed toward Opus-like answers. That’s not conclusive proof that Kimi was trained on Anthropic’s outputs, but it’s the kind of behavioral fingerprint that makes distillation suspicion hard to dismiss.
Why does this matter for AI safety and competition?
Labs like Anthropic and OpenAI have leaned on hidden chain-of-thought as a form of both safety infrastructure and competitive moat. If a rival lab can quietly extract reasoning traces from a proprietary API, two things follow.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
One, the safety case weakens. Hidden reasoning was partly justified as a way to prevent users from seeing raw, unfiltered model cognition that might contain errors, biases, or sensitive leaked data. If that barrier can be bypassed by outside researchers, it can presumably be bypassed by anyone else with the same technique, including bad actors looking to scrape credentials that show up accidentally in reasoning traces.
Two, the economic case weakens. A huge amount of the value in frontier models sits in the quality of their reasoning. That’s expensive to produce: it requires massive training runs, careful reinforcement learning, and enormous compute budgets. If a smaller or newer lab can extract meaningful signal from a frontier model’s chain-of-thought and use it to shape their own model’s outputs, the head start that expensive frontier labs are paying for erodes faster than they’d like.
According to the coverage of the paper, the researchers didn’t just publish and walk away. They reportedly warned Anthropic and other labs about the vulnerability before or alongside publication, which is standard responsible-disclosure practice, but it also signals that the labs may not yet have a reliable fix.
Is it possible to fully stop this kind of leak?
Based on what’s been shown so far, a complete fix looks difficult. The extraction method works by going around the summarization layer rather than breaking the underlying model, which means patching one specific exploit doesn’t necessarily close the door on the next variant. Anti-distillation safeguards are designed to catch direct attempts to query or copy a model’s behavior at scale. A technique that never directly attacks the stronger model, and only borrows a sliver of its reasoning to steer a separate model’s output, doesn’t look like the kind of attack those safeguards were built to catch.
That suggests the current generation of protections around chain-of-thought is more of a speed bump than a wall. Labs can keep tightening how much of the raw trace ever reaches an API response, but as long as any signal from that reasoning leaks through, in this case through prefilling techniques, there’s a path for smaller models to absorb some of it.
What does this mean for people building with AI APIs?
For developers, the practical takeaway is less about replicating the extraction technique and more about treating chain-of-thought as sensitive by default. If your application logs, stores, or displays raw reasoning traces from any model, treat those logs the same way you’d treat logs containing user input: assume credentials, personal data, or other sensitive fragments could show up in there unexpectedly.
It also reframes how to think about model provenance. If reasoning traces can leak or influence other models even without direct API scraping, benchmarking claims about a new model being trained “from scratch” deserve some skepticism. The line between an original model and a distilled one is getting blurrier, and that has real implications for anyone choosing between vendors based on claimed independence or capability.
Frequently Asked Questions
What is chain-of-thought in AI models?
Chain-of-thought is the internal, step-by-step reasoning a model produces while working through a problem before generating its final answer. Reasoning-focused models like Claude, GPT’s reasoning variants, and Gemini’s thinking modes rely heavily on this internal process.
Why don’t users normally see the raw chain-of-thought?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
Labs summarize or filter it before it reaches the user interface, mainly for safety, to avoid exposing sensitive data like credentials that can accidentally appear in the reasoning, and to make it harder for competitors to distill the model’s reasoning patterns.
What does “distillation” mean in this context?
Distillation is training a smaller or newer model to mimic the behavior of a stronger, more expensive model. Access to raw chain-of-thought traces can make distillation easier and cheaper, since it exposes not just answers but the reasoning steps behind them.
Does the Stolen Thoughts paper prove specific labs copied each other’s models?
No. It demonstrates that prefilling a model’s reasoning with fragments from another model’s chain-of-thought shifts its answers toward that model’s style, which is suggestive of distillation-like behavior but not direct proof that any specific lab trained on another’s outputs.
Can this kind of chain-of-thought leak be fully patched?
Current evidence suggests it’s difficult. The method sidesteps the usual anti-distillation safeguards rather than attacking them directly, so closing one gap doesn’t guarantee the underlying issue is solved.