From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models
Citations
0
Open access
No
Source
crossref
OpenAlex
Not enriched
DOI
10.1101/2025.02.06.636901
Abstract
Abstract Protein language models (pLMs) are powerful predictors of protein structure and function, learning through unsupervised training on millions of protein sequences. pLMs are thought to capture common motifs in protein sequences, but the specifics of pLM features are not well understood. Identifying these features would not only shed light on how pLMs work, but potentially uncover novel protein biology––studying the model to study the biology. Motivated by this, we train sparse autoencoders (SAEs) on the residual stream of a pLM, ESM-2. By characterizing SAE features, we determine that pLMs use a combination of generic features and family-specific features to represent a protein. In addition, we demonstrate how known sequence determinants of properties such as thermostability and subcellular localization can be identified by linear probing of SAE features. For predictive features without known functional associations, we hypothesize their role in unknown mechanisms and provide visualization tools to aid their interpretation. Our study gives a better understanding of the limitations of pLMs, and demonstrates how SAE features can be used to help generate hypotheses for biological mechanisms. We release our code , model weights and feature visualizer .
Collections
Add to collection
Paper intelligence
Analysis has not been completed yet.
No graph connections yet.
Sync citations or add papers to shared collections to build this network.
Knowledge graph
Citation network
Explore references, papers that cite this work and related papers in your Codex library.
References
0No references have been linked yet.
Cited by
0No saved paper is currently linked as citing this work.
Related papers
0Add papers to shared collections or enrich their topics to find related work.
Research workspace
Attach the paper PDF, extract its text, classify its contents and create semantic embeddings.
Paper resources
No PDF assets have been attached to this paper yet.