A protein language model learned biology nobody taught it

The obvious place to look is individual neurons, and that turns out to be a dead end. Examining single neurons in ESM-2 surfaces only a few dozen per layer with any clear conceptual alignment, across roughly fifteen recognizable concepts. The model plainly knows more than that. It just isn't storing what it knows one concept per neuron.

Simon and Zou trained sparse autoencoders on ESM-2's internal embeddings, separating overlapping signals into individual features that can be examined one at a time. They then matched those features against known protein annotations from Swiss-Prot to see which corresponded to established biology.

Thousands of features is far more than anyone can inspect by hand, so they built an automated interpretation pipeline: a language model proposes a description of what each feature responds to, and a separate validation step checks whether the description holds up against the feature's actual activation pattern.

Up to 2,548 interpretable features per layer, corresponding to as many as 143 distinct biological concepts. Compare that to the few dozen legible neurons in the same layers. Same model, same weights — the difference is entirely in how you look.

Superposition, confirmed across scales. The pattern holds as models get bigger, and larger models capture more interpretable concepts rather than fewer. Whatever ESM-2 is doing, packing many concepts into shared dimensions is a persistent feature of it, not an artifact of one model size.

Concepts with no name yet. Some features track coherent patterns spanning evolutionarily distinct protein families — real structure in the model's representations that existing annotation databases don't capture. In practice this runs in both directions: the same features can flag annotations missing from current databases.

Finding a feature that correlates with a biological concept shows the model has a representation. It doesn't show the model uses it. Establishing that requires intervening — changing the feature mid-computation and watching what the model does differently.

That intervention is more delicate than it sounds. Sparse autoencoders reconstruct activations imperfectly, so simply swapping in a modified reconstruction degrades the model with the autoencoder's own error. The authors avoided this by splitting each embedding into the part the autoencoder reconstructs and the leftover residual, editing only the reconstruction — clamping chosen features to target values — then adding the residual back before letting the forward pass continue. The model's own imperfectly-captured signal survives untouched; only the feature under test changes.

The paper states it directly: the steering experiments were all conducted using NNsight. NNsight's intervention API is what makes this read-modify-write pattern tractable — pull activations at a chosen layer, compute on them outside the model, write the modified values back, and let normal processing resume, without reimplementing ESM-2's forward pass to insert an edit.

The results are the paper's clearest causal evidence. In one experiment, steering a periodic glycine feature at a single amino acid position changed the model's predictions for neighbouring positions in a coherent, interpretable way. The model wasn't just storing the concept. It was propagating it.

This is one of the clearest bridges between mechanistic interpretability and a science domain outside natural language. The methods came from work on language models; the substrate is protein sequence; the toolkit transferred largely intact.

For computational biologists, it's a demonstration that a protein model's representations can be decomposed into components you can name, check against the literature, and manipulate. For interpretability researchers, it's evidence that superposition and sparse decomposition aren't quirks of text models.