Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
Abstract
Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. However, it is unclear whether LMs perform these tasks by cheating with answers memorized from pretraining corpus, or, via a multi-step reasoning mechanism. In this paper, we try to answer this question by exploring a mechanistic interpretation of LMs for multi-step reasoning tasks. Concretely, we hypothesize that the LM implicitly embeds a reasoning tree resembling the correct reasoning process within it. We test this hypothesis by introducing a new probing approach (called MechanisticProbe) that recovers the reasoning tree from the model’s attention patterns. We use our probe to analyze two LMs: GPT-2 on a synthetic task (k-th smallest element), and LLaMA on two simple language-based reasoning tasks (ProofWriter & AI2 Reasoning Challenge). We show that MechanisticProbe is able to detect the information of the reasoning tree from the model’s attentions for most examples, suggesting that the LM indeed is going through a process of multi-step reasoning within its architecture in many cases. Show more
Permanent link
https://doi.org/10.3929/ethz-b-000653493Publication status
publishedExternal links
Book title
Proceedings of the 2023 Conference on Empirical Methods in Natural Language ProcessingPages / Article No.
Publisher
Association for Computational LinguisticsEvent
Organisational unit
09684 - Sachan, Mrinmaya / Sachan, Mrinmaya
Funding
ETH-19 21-1 - Neuro-cognitive Model Inspired from Human Language Processing (ETHZ)
More
Show all metadata
ETH Bibliography
yes
Altmetrics