Did Moonshot's Kimi K3 Distill Claude or GPT? Examining the Claims
A technical look at whether Moonshot's Kimi K3 could have been distilled from US frontier models, based on how distillation actually works.
Drafted with Claude from source material, checked by automated verification, and reviewed before release. How we make these.
What actually happened with Moonshot’s K3 release
Moonshot AI released K2, a non-reasoning model, last July. It scored around 19 on the Artificial Intelligence Index. Earlier this year, K2.6 scored 44 on the same benchmark. Then Moonshot shipped K3, which landed within a few points of the most capable model on the index at the time, reportedly a model referred to as “Fable 5” sitting near 60%. That jump, from a mid-tier open model to near-frontier performance in under a year, is what triggered accusations that Moonshot distilled its model from US frontier labs, possibly by scraping outputs through API access. The claim deserves a real technical answer, not just a geopolitical reaction.
TL;DR
- Distillation has a precise technical meaning: training a smaller “student” model to reproduce the full probability distribution (soft labels) a larger “teacher” model assigns to outputs, not just copying finished text.
- True distillation requires access to logits or log probabilities, something commercial APIs from Anthropic, OpenAI, and Google do not expose. You typically get only the final sampled token.
- What critics usually describe is “sequence-level distillation” or output harvesting, which means scraping a model’s finished responses and fine-tuning on them. This transfers style and tone, not underlying capability.
- Anthropic reported roughly 16 million exchanges with Claude through about 24,000 accounts it linked to activity originating from China, which is evidence of terms-of-service violations, not proof of capability transfer.
- The timeline makes deep distillation implausible: the referenced frontier model was pulled from public access days after release due to export controls, then widely available for about eighteen days before K3 shipped, far too short a window to run pre-training and mid-training on a multi-trillion-parameter target.
- Capability comes mostly from pre-training, mid-training, and large-scale reinforcement learning, stages that require months of locked data and tens of millions of rollouts, not something a few weeks of API scraping can replicate.
- On-policy distillation is real and effective, but it still needs teacher log probabilities, which paid frontier APIs simply don’t hand out.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
What is distillation, technically?
Distillation as a formal technique traces back to a 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, “Distilling the Knowledge in a Neural Network.” The core idea: a large, expensive-to-run teacher model produces not just a final answer but a full probability distribution over every possible output, called soft labels. In image classification, a teacher might assign 90% probability to “this is a 2” but also small nonzero probabilities to other digits like 7 or 3, based on visual similarity. Hinton called this “dark knowledge” because those secondary probabilities encode generalization patterns that hard labels (a flat “it’s a 2, everything else is zero”) throw away.
To make that signal usable, you divide the model’s raw outputs (logits) by a temperature parameter, which flattens the distribution into something trainable. A smaller student model is then trained to match that softened distribution, not just the final label. If it succeeds, it inherits a compressed version of the teacher’s behavior.
Large language models work the same way at the token level. Every generation step involves the model scoring every token in its vocabulary and picking the most likely one. Real distillation means training a student to match that entire distribution across the vocabulary, not just the single token that got sampled.
Why can’t you distill a model through its public API?
This is the crux of the controversy. Commercial APIs from Anthropic, OpenAI, and similar providers give you the sampled output, the text the model actually generated. They do not expose the underlying log probabilities across the vocabulary. Some APIs will hand you log probabilities for the token that was chosen, but not the full distribution the classic distillation method requires.
Without that distribution, you cannot run the textbook Hinton-style distillation. What you can do is collect a large volume of finished outputs and fine-tune a model on them as ordinary training data. This is called sequence-level distillation, described in a 2016 paper by Kim and Rush. It’s a legitimate, well-established technique, used since early days of open-source fine-tuning projects like Alpaca and Vicuna, which trained on outputs scraped from OpenAI’s API.
The catch: sequence-level distillation mostly transfers style, tone, and surface-level imitation. It does not transfer the deeper reasoning capability baked into a frontier model through its training pipeline. Anthropic reported observing this kind of activity at industrial scale, noting roughly 16 million exchanges with Claude through about 24,000 accounts it attributed to origins in China, widely understood as an example of output harvesting rather than genuine capability distillation. It’s a real terms-of-service problem. It’s not the same thing as stealing a model’s core competence.
Where does model capability actually come from?
Modern large language model training runs through four broad stages: pre-training, mid-training, supervised fine-tuning, and reinforcement learning. Pre-training and mid-training establish base capability and are extremely expensive; companies lock training data months before a model ships and reuse the same pre-trained base across multiple downstream versions. Mid-training can inject targeted skills, like math or code proficiency, more cheaply than a full pre-training run, but it still requires a real corpus, not scraped chat transcripts.
Supervised fine-tuning shapes behavior and style. The heaviest lift today, and where labs report spending the most post-training effort, is large-scale reinforcement learning, where a model practices tasks and gets rewarded for successful attempts. A full RL run for a frontier-class model reportedly needs on the order of tens of millions of rollouts (complete task attempts). That volume is simply not achievable by hitting a paid API a few thousand or even a few million times.
There’s a more sophisticated variant called on-policy distillation, where a student model generates its own answers and a teacher model scores every token using its log probabilities, acting as a reward model. Research has shown this approach can be dramatically more compute-efficient than plain RL, reportedly cutting GPU hours by an order of magnitude in one benchmark comparison. But it has the same hard requirement as classic distillation: you need the teacher’s log probabilities, not just its text output. Paid commercial APIs don’t provide that.
Does the timeline support the distillation accusation?
This is where the geopolitical claim runs into a hard scheduling problem. The frontier model in question was released, then pulled from public access within days due to US export controls, before becoming widely available again roughly three weeks before Moonshot shipped K3. That leaves a window of about eighteen days of API access, at most, before K3’s release.
A model in the multi-trillion-parameter range needs pre-training and mid-training data locked in months ahead of launch. Eighteen days of API scraping, even at industrial scale, could not feed a pre-training run that size, and even a best-case scenario of full API access during that window would only support sequence-level distillation, transferring style rather than capability, because no public API from Anthropic or comparable frontier labs exposes the log probabilities that real capability transfer depends on.
None of this rules out that Moonshot used some form of imitation learning, output harvesting for stylistic fine-tuning, or synthetic data generation, practices common across the industry and not unique to any one country’s labs. But the specific claim, that K3’s frontier-level jump came from deep distillation of a US model’s capabilities via API, doesn’t hold up against how distillation technically works or against the release timeline.
Frequently Asked Questions
What is the difference between distillation and simply copying outputs?
Distillation, in the technical sense, trains a student model to match a teacher’s full probability distribution over possible outputs, transferring generalization patterns. Copying outputs (sequence-level distillation or output harvesting) means fine-tuning on a teacher’s finished text, which transfers writing style and tone but not the deeper reasoning capability encoded in the teacher’s training.
Can you distill a model through a public API alone?
Not in the full technical sense. Public APIs from major labs return sampled text, not the log probabilities across the full vocabulary that classic distillation requires. You can do sequence-level distillation (fine-tuning on scraped outputs), but that’s a much weaker technique that mostly copies style.
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
Why did Anthropic report millions of exchanges linked to accounts from China?
Anthropic reported observing large-scale automated querying, described as roughly 16 million exchanges through around 24,000 accounts, consistent with output harvesting for sequence-level distillation or dataset building. It’s evidence of terms-of-service abuse, not proof that a competitor cloned a model’s underlying capability.
Why does the K3 release timeline matter for the distillation claim?
The frontier model cited in the controversy was briefly available, pulled due to export controls, then reintroduced weeks before K3 shipped. Pre-training and mid-training data for a model of that scale need to be locked months in advance, so an eighteen-day API window couldn’t plausibly supply the data needed for genuine capability transfer.
Is on-policy distillation a shortcut around these limits?
On-policy distillation is real and efficient, letting a student learn from a teacher’s token-by-token scoring of the student’s own outputs. But it still requires access to the teacher’s log probabilities, which commercial APIs don’t expose, so it’s not a workaround for distilling a closed frontier model through normal API access.
