Neural Networks Are Clean

TL;DR

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

A Goodfire researcher argues that neural networks may be more interpretable than influential examples of “polysemantic” neurons suggested. The report presents results from a revised InceptionV1 model and preliminary estimates for Qwen3-32B, but its conclusions and neuron counts remain the author’s analysis, not a settled consensus.

A Goodfire researcher has published a report arguing that neural networks are more legible than mechanistic interpretability research has often assumed, saying prominent examples of neurons that appear to perform unrelated tasks may have been misread. The report presents a revised InceptionV1 model whose author says is broadly interpretable, alongside preliminary estimates for Qwen3-32B; these are research claims, not an established field-wide conclusion.

The report challenges superposition, a proposed explanation for why a neuron can appear to respond to several unrelated concepts. Under that account, a model with limited neurons may encode many features in overlapping directions, making individual units hard to interpret. The author argues that some apparent mixtures instead arise from ordinary generalization and composition, or from shortcomings in model architecture and analysis tools.

As evidence, the author describes changes to InceptionV1, an image-recognition model. They report that architectural improvements, without changing the model’s size or training data, made 99% to 100% of neurons in its final layer easily interpretable, and that the model was broadly legible throughout. The author also says versions trained across a 300-fold range in size remained broadly legible. These are findings reported by the author; the supplied material does not provide an independent replication.

For language models, the report gives a preliminary audit of Qwen3-32B. The author estimates roughly 90% of neuron poles have one main job, about 7% have two related jobs, and 2% to 4% are not yet understood. The report cautions that estimates depend on choices such as activation thresholds and how researchers define a single “job.”

How Interpretability Research Could Change

If the report’s interpretation holds up, researchers may be able to study a model’s neurons and circuits directly more often, rather than first training a separate model to represent its internal features. That could change how mechanistic interpretability work is organized and how researchers test explanations of model behavior.

The issue matters beyond a debate over terminology. Interpretability research aims to understand the internal processes behind model outputs, which may help researchers investigate capabilities, failures and safety risks. However, finding many apparently understandable neurons would not by itself show that a model’s full behavior is understood or that it is safe. The report’s estimates are about neuron-level interpretability, not a general measure of model reliability.

From Polysemantic Neurons to Direct Study

Mechanistic interpretability has used two broad approaches: studying a network’s existing neurons and circuits, or hypothesizing features and using methods such as sparse autoencoders to extract a more separable representation. The second approach gained prominence partly because some networks appeared to contain neurons with multiple unrelated functions.

One familiar vision example involved a unit reported to respond to both cat faces and car fronts. The Goodfire author says such cases helped motivate the superposition explanation, but argues they do not establish that unrelated features are arbitrarily packed together. The report also revisits a 2019 circuit interpreted as a car feature spreading into dog-related neurons, saying the pattern may instead reflect composition and generalization. The author says they know of no second published circuit with the same shape.

The report contrasts its newer claims with earlier studies that found many neurons difficult to interpret, including work on small language models and automated explanations. It attributes the difference to improved tools and modern models, while arguing that some earlier assessments may have been affected by architecture and analysis limitations.

“Neural networks are already clean.”

— Goodfire report author

Questions About the Report’s Estimates

The report’s findings are presented by its author, and the supplied source material does not establish whether other researchers have replicated them or whether the field agrees with the reinterpretation of superposition. The Qwen3-32B figures are explicitly preliminary, and the author says the share classified as having one main job varies with methodological choices.

It is also unclear how far the findings generalize across model architectures, training methods and tasks. Neurons that can be assigned an understandable role do not necessarily make every circuit or model behavior understandable. The report does not establish that direct neuron analysis can replace feature-extraction methods in all cases.

Replication and Broader Model Tests

The next test is whether researchers can replicate the InceptionV1 analysis and apply comparable methods to other models. Broader evaluations would help establish whether the reported legibility reflects a general pattern or particular architecture, tools and definitions.

Further work could also clarify how neuron-level judgments change with activation thresholds and how those judgments relate to understanding circuits and model behavior. The supplied report does not announce a specific follow-up study or publication schedule, so the timing of independent tests remains unknown.

Key Questions

What does the report mean by saying neural networks are clean?

The author argues that many networks’ neurons and circuits can be studied directly, and that apparent mixtures of unrelated concepts have sometimes been misinterpreted. “Clean” is the report’s characterization, not a settled scientific classification.

What is superposition in this debate?

Superposition is an explanation for how a network with limited neurons might represent multiple features in overlapping directions. The report argues that some examples cited in support of it instead reflect ordinary generalization, composition or model-training problems.

What did the report say about InceptionV1?

The author says architectural improvements made 99% to 100% of neurons in the model’s final layer easily interpretable, without changing model size or training data. The supplied material does not report independent replication of that result.

Are the Qwen3-32B neuron figures definitive?

No. The report describes them as preliminary estimates and says results depend on thresholds and on how researchers define a neuron’s main job.

Does neuron interpretability mean a model is safe?

No. The report concerns how understandable neurons may be. It does not establish that the model’s complete behavior is understood or that the model is reliable or safe.

Source: hn