Inside the Model: What Did Researchers Find When They Opened the Box?

You ask a language model: why did you choose this answer? It hands you a neat five-point justification. But is what it said what actually happened inside it?

Weights, not rules

Open a model looking for the "rule" that explains its answer and you will find none. A neural network is a mathematical function made of connected layers, and between any two units sits a numerical weight that sets how strongly one influences the other. During training the weights are adjusted step by step to reduce error on examples. Knowledge does not live in one "cell"; it is spread across patterns of weights. We explained the structure of language models in the article "How Do Large Language Models (LLMs) Like ChatGPT Work?". Hence the phrase "black box": we see the input and output, and it is hard to read the reason.

Attempts to open the box

The field of mechanistic interpretability takes its name from trying to understand the internal mechanics of computation. In May 2024 Anthropic researchers published a study of the model Claude 3 Sonnet, in which they trained sparse autoencoders on one of its middle layers and extracted millions of "features", each corresponding to a concept a human can name; they tried three sizes, the largest reaching about 34 million features. In a striking demonstration, they raised the activity of a "Golden Gate Bridge" feature to ten times its maximum and the model began describing itself as the bridge.

But the authors themselves qualify the achievement: the features are an "incomplete description" of the model's representations. In their largest autoencoder they found features for only about 60% of London's boroughs, the computational cost is high, and the usefulness of any particular feature for safety has not yet been shown. The box has become less dark, not transparent.

Deceptive shortcuts

A model can be right for the wrong reasons. Robert Geirhos and colleagues define "shortcut learning" as decision rules that succeed on standard tests and fail under harder conditions such as the real world. The example from Ribeiro and colleagues is well known: they deliberately trained an image classifier to predict "wolf" when it saw snow in the background, then asked 27 participants whether they trusted it. Before seeing the explanations 10 trusted it; afterwards, 3 did, and the number who identified snow as the likely cause rose from 12 to 25. Here the explanation exposed the flaw. Note that the flaw was planted in the experiment, so it shows the tool's usefulness, not how common the case is.

When the model explains itself

More delicate is what a model says about why it answered as it did. Turpin and colleagues found that adding a biasing feature to the input, such as ordering the options so that the right answer is always "A", changes the model's answer without its reasoning mentioning that factor, and accuracy can fall by as much as 36% across 13 test tasks, on two models of the time (GPT-3.5 and Claude 1.0). In a 2025 Anthropic study of two reasoning models, Claude 3.7 Sonnet mentioned the hint that influenced it in 25% of cases on average, and DeepSeek R1 in 39%. The researchers acknowledge limits: the tests were artificial, the questions multiple choice, and they covered only two companies' models.

So a model's reasoning may be a plausible story told after the decision, not a faithful record of its path.

What do we do?

• Treat the explanation as a suggestion: check facts and sources, not the eloquence of the argument.
• In grave decisions (health, money, rights) keep a responsible human who can disagree.
• Ask the vendor: how did you test that the explanation matches what the model does?
• Test the model outside familiar examples, since a shortcut shows itself when context changes.

Science here is advancing, and no one knows how far it will go. Between claiming we understand everything and claiming we understand nothing lies a more honest position: we know some of what happens, we are ignorant of some, and we take precautions in proportion to our ignorance.