Challenges in Mechanistically Interpreting Model Representations

Summary

Mechanistic interpretability (MI) aims to understand AI models by reverse-engineering the exact algorithms neural networks learn. Most works in MI so far have studied behaviors and capabilities that are trivial and token-aligned. However, most capabilities important for safety and trust are not that trivial, which advocates for the study of hidden representations inside these networks as the unit of analysis. We formalize representations for features and behaviors, highlight their importance and evaluation, and perform an exploratory study of dishonesty representations in `Mistral-7B-Instruct-v0.1'. We justify that studying representations is an important and under-studied field, and highlight several challenges that arise while attempting to do so through currently established methods in MI, showing their insufficiency and advocating work on new frameworks for the same.

Plain-language summary

Mechanistic interpretability (MI) aims to understand AI models by reverse-engineering the exact algorithms neural networks learn. Most works in MI so far have studied behaviors and capabilities that are trivial and token-aligned. However, most capabilities important for safety and trust are not that trivial, which advocates for the study of hidden representations inside these networks as the unit of analysis. We formalize representations for features and behaviors, highlight their importance and evaluation, and perform an exploratory study of dishonesty representations in `Mistral-7B-Instruct-v0.1'. We justify that studying representations is an important and under-studied field, and highlight several challenges that arise while attempting to do so through currently established methods in MI, showing their insufficiency and advocating work on new frameworks for the same.

Research problem

However, most capabilities important for safety and trust are not that trivial, which advocates for the study of hidden representations inside these networks as the unit of analysis. In terms of scaling, while Lieberum et al. (2023) show that some circuit analysis ideas can scale to larger models, it is still not clear if this remains true in a more general case. Figures adapted from these works and taken from a similar illustration in Zou et al. (2023). tain feature in n-d data such that f : Rn → {0, 1}, an important open problem is how to study the emergence and effect of such a black-box feature f .

Methodology

Challenges in Mechanistically Interpreting Model Representations Satvik Golechha * 1 James Dao * 2 Abstract Mechanistic interpretability (MI) aims to under- stand AI models by reverse-engineering the exact algorithms neural networks learn. A number of transformer model capabilities and properties have since been studied mechanistically, such as indirect-object identification (Wang et al., 2022), grokking (Nanda et al., 2023), and the greater than operation (Hanna et al., 2023). They advocate studying population-level representations inside a model as the center of analysis instead of neurons and circuits in the bottom-up approach of MI. They show that a sim- ple method can find linear representations for a number of complex behaviors relevant to AI safety such as honesty, harmlessness, and power-seeking. ngineer- ing (Zou et al., 2023) push for a top-down approach to trans- parency inspired from a similar perspective in cognitive neuroscience (Barack & Krakauer, 2021). They show that a sim- ple method can find linear representations for a number of complex behaviors relevant to AI safety such as honesty, harmlessness, and power-seeking.

Main findings

REPRESENTATIONS AS THE RIGHT LEVEL OF ABSTRACTION Since studying all the logits in a model’s long-term genera- tion is intractable, and next-token-based MI has significant generalization and scaling issues, we believe that repre- sentations form just the right level of abstraction to study, allowing for human-interpretable model explanations while being tractable. Thus, a fundamental framework for interpreting representations helps toward understanding and improving the model with respect to all of these phenomena. With results from various commonly used MI methods, we demonstrate that the most important questions around studying representations still 1The code for this work is available here. remain unanswered. We find that this trivial hypothesis does not hold. One anomaly we find across multiple datapoints is MLP-30, which contributes to honesty when the model is injected with dishonesty.

Key contributions

Limitations

• While these are much more complicated behaviors than those for which circuit- level analyses have been successful, this method fails to answer “how a model works” and does not yield concrete, verifiable interpretations.
• We find that this trivial hypothesis does not hold.
• Note that the contribution of model components before layer 15 does not change. focus at MLP and attention layers due to computational constraints.
• On the other hand, when an MLP layer m gets (ra + d) in- stead of ra, the output of the MLP layer changes from y = Wout(Win(ra) ∗ SiLU(Wgate(ra))) to y = Wout(Win(ra + d) ∗ SiLU(Wgate(ra + d))), both of which cannot be de- composed linearly due to the SiLU non-linearity.

Models and methods

Datasets

Evaluation metrics

Keywords