Are Neural Networks Actually Interpretable? Nick Cammarata's Claim Explained
Nick Cammarata argues AI neurons aren't tangled messes but readable patterns that grow like trees, challenging years of interpretability pessimism.
What is Nick Cammarata’s “neural networks are already clean” claim?
Nick Cammarata, a former OpenAI interpretability researcher now at Goodfire, published a research essay on October 15th arguing that the inside of neural networks is not the chaotic, overlapping mess the field has assumed for years. His claim is that most neurons inside AI models have a single, legible job once you look at them correctly, and that the field’s main evidence for confusion and “superposition” was built on misreadings of its own famous examples. If correct, this reframes one of the hardest open problems in AI: understanding what’s actually happening inside a model when it produces an answer.
TL;DR
- Nick Cammarata, co-author of the 2020 “Zoom In” interpretability paper and now a researcher at Goodfire, argues that neural network neurons are largely readable rather than randomly tangled.
- The famous cat-and-car neuron from the Inception V1 image model, long cited as proof that neurons encode unrelated concepts, turns out to respond to a shared V-shape pattern found in both cat faces and car fronts.
- Cammarata reexamined a 2020 circuit he co-published that appeared to show a car concept hidden inside dog-related neurons, and found both groups shared a simpler trait: the subjects faced the camera.
- He describes neurons as growing through recursive generalization, starting narrow (like a yellow rapeseed field) and branching outward into related concepts (tennis balls, dogs, broccoli) in traceable steps, not random jumps.
- Some of the apparent confusion came from flawed research tools, including visualization methods that defaulted to generating dog imagery and grading systems that rewarded predictive guesses over genuinely true explanations.
- On the open-weight model Qwen 332B, Cammarata reports roughly 90% of neurons have one clear function, a sharp jump from a 2023 OpenAI estimate that only about 1.7% of neurons in another model could be explained.
- Reactions from figures including Sam Altman and OpenAI researcher Rune suggest the essay is being treated as a potentially significant moment for AI interpretability, though it remains a single researcher’s essay rather than a peer-reviewed consensus finding.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Why does the “AI black box” problem matter?
AI companies can train powerful models without fully understanding how those models arrive at their outputs. A neural network is built from layers of artificial neurons, numerical switches that activate in patterns learned during training on massive datasets. Nobody writes instructions for what any individual neuron should represent. The network adjusts billions of internal values on its own until it gets good at predicting text, classifying images, or generating code.
This creates a real accountability gap. An engineer can explain why a bridge beam is load-bearing. A pharmacologist can trace how a drug interacts with the body. But ask why a large language model generated a specific answer, and the honest response from most researchers has been: we don’t fully know. As these systems get deployed in hospitals, schools, financial systems, and government, that gap becomes a practical safety problem, not just an academic curiosity. Interpretability research exists specifically to close it, by trying to map what individual neurons and circuits inside a model actually represent.
What is superposition, and why did researchers believe AI was a tangled mess?
Early interpretability work found something discouraging: individual neurons often seemed to fire for multiple, unrelated concepts. The canonical example came from Inception V1, an older image classification model, where one neuron appeared to activate both for cat faces and for the fronts of cars. Researchers called this property polysemanticity, and in some layers of that model nearly every neuron looked this way.
The field’s explanation became known as superposition: a network has far fewer neurons than there are concepts in the world, so it compresses multiple unrelated ideas into the same neuron, on the theory that concepts which rarely co-occur won’t interfere with each other. If true, this is bad news for interpretability, because it implies that studying a single neuron tells you almost nothing about what the model is actually representing. The field’s workaround was to train a second model, a sparse autoencoder, to try to disentangle the first one’s internal representations, a translator for a signal assumed to be fundamentally scrambled. Supporting this pessimism were hard numbers: a 2023 Anthropic study gave typical neurons in a small model a near-zero interpretability score, and a related OpenAI project (the one Cammarata started) found only about 1.7% of neurons could be explained even under generous criteria.
How did Cammarata reinterpret the famous cat-and-car neuron?
Cammarata went back to the Inception V1 neuron that had become the textbook example of polysemanticity. Instead of treating “cats and cars” as two disconnected categories crammed into one slot, he asked a simpler question: what single visual pattern could explain both? The answer was a V-shape. A cat’s face narrows to a point formed by its ears and chin. The front of a car, especially a sloped hood meeting a grille, forms a similar pointed angle. When he checked what else activated the neuron, the pattern held: V-shaped flames, V-shaped water jets, V-shaped fox faces.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
He also ran a confirming check by looking at what suppresses the neuron, pushing its activation in the opposite direction. The answer was round, O-shaped imagery, the geometric opposite of a V. A neuron encoding a genuinely random mix of concepts wouldn’t need a coherent opposite. A shape detector would. Notably, when a sparse autoencoder (the standard “untangling” tool) was run on this same layer, it split the neuron into separate cat and car components, in Cammarata’s view making a single coherent concept look more fragmented than it actually was.
He applied the same scrutiny to a 2020 circuit he co-published with Chris Olah, which appeared to show a car-related concept embedded among dog-detecting neurons, long treated as a flagship example of concept-hiding in neural networks. Revisited, the dogs and the car shared a simpler commonality: they were all posed facing the camera. According to Cammarata, no second genuine example of that original “hidden car in dog neurons” phenomenon has been published since.
Why do neurons look confusing if they’re not actually randomly mixed?
Cammarata’s answer is that neurons develop the way branches grow on a tree, not the way items pile up in a junk drawer. He retrained an image model from scratch and tracked a single neuron through the entire training process. It began by responding to yellow rapeseed fields. Because some of those training photos happened to contain dogs, the neuron’s attention extended to dogs sitting in grass, then to other green-and-yellow round objects like tennis balls and apples, then to shaggy dogs that often appeared with tennis balls, and eventually to broccoli. By the end of training, the original field imagery had faded, leaving a neuron that fires for tennis balls, terriers, and broccoli, a combination that looks arbitrary in isolation but makes sense as a traceable chain of generalization.
Some apparent confusion also turned out to be a tooling problem rather than a property of the network itself. Common visualization methods, which generate an idealized image to show what a neuron “wants to see,” tended to default toward dog-like features in deeper layers regardless of what the neuron actually represented, because dog features were an easy way to trigger high activation. Grading systems for neuron explanations compounded the issue by scoring guesses on whether they predicted firing patterns rather than whether they were actually true, similar to marking a test answer correct without checking the reasoning behind it.
What did this look like in a modern language model?
Cammarata’s main test case was Qwen 332B, an open-weight language model from Alibaba with 32 billion parameters. In his audit, roughly 90% of its neurons appeared to have one clear, identifiable function, with another 7% serving two related purposes, leaving only 2 to 4% unexplained. That stands in sharp contrast to the 1.7% figure from the 2023 OpenAI project.
One illustrative example: a neuron that initially responded to the letters “IY,” generalized to the word “DIY,” and eventually generalized further to the full phrase “do-it-yourself,” a phrase that doesn’t literally contain the letters “IY” anymore. That’s the same tree-like growth pattern seen in the image model, playing out with language concepts instead of visual ones. Cammarata also tested model designs and sizes: an updated architecture called ConvNext, trained on identical data, produced neurons that stayed more narrowly focused rather than sprawling outward, and smaller versions of models tended to form broader single concepts per neuron rather than cramming unrelated ideas together, both patterns consistent with organized growth rather than random compression.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
Is this claim widely accepted yet?
Not formally. This is a single researcher’s essay, not a peer-reviewed paper with independent replication. That said, the reaction from well-known figures in the field, including an OpenAI researcher calling it a potential turning point and Sam Altman publicly welcoming the work, suggests it’s being taken seriously rather than dismissed. The real test will be whether other labs can reproduce the interpretability rates Cammarata reports on other models and architectures.
Frequently Asked Questions
Who is Nick Cammarata?
He’s a former OpenAI researcher who worked on the company’s interpretability team, co-authored the influential 2020 “Zoom In” paper with Chris Olah, and later led a project using AI to explain the internal workings of other AI models. He now works at Goodfire, an AI lab focused on interpretability that raised $150 million at a $1.25 billion valuation.
What does “superposition” mean in AI?
Superposition is the theory that neural networks, having fewer neurons than there are concepts to represent, compress multiple unrelated ideas into single neurons. Cammarata’s essay challenges how much superposition actually explains the field’s famous confusing examples, arguing many were cases of a single coherent pattern rather than a jumble of unrelated ideas.
Does this mean AI models are now fully understood?
No. Cammarata reports high interpretability rates for specific neurons in specific models he examined, including Qwen 332B, but a meaningful percentage of neurons remain unexplained, and the claim hasn’t been independently replicated across the broader range of AI models in wide use.
Why does interpretability research matter for AI safety?
If researchers can reliably read what’s happening inside a model, they can better check whether it’s being honest, whether it has learned unwanted behaviors, and whether it’s reasoning the way developers intend, especially as these systems take on more autonomous, high-stakes tasks.
What’s the significance of the cat-and-car neuron example?
It was one of the most cited pieces of evidence that AI neurons encode unrelated, tangled concepts. Cammarata’s reinterpretation, that the neuron detects a shared V-shape pattern rather than two unrelated categories, undercuts a foundational example used to argue neural networks are inherently difficult to interpret.
