Are Language Models Efficient Reasoners? A Perspective from Logic Programming
OPEN ACCESS
Loading...
Author / Producer
Date
2025-10-29
Publication Type
Conference Paper
ETH Bibliography
yes
Citations
Altmetric
OPEN ACCESS
Data
Rights / License
Abstract
Modern language models (LMs) exhibit strong deductive reasoning capabilities, yet standard evaluations emphasize correctness while overlooking a key aspect of human-like reasoning: efficiency. In real-world reasoning scenarios, much of the available information is irrelevant, and effective deductive inference requires identifying and ignoring such distractions. We propose a framework for assessing LM reasoning efficiency through the lens of logic programming, introducing a simple method to align proofs written in natural language -- as generated by an LM -- with shortest proofs found by executing the logic program. Efficiency is quantified by measuring how well a model avoids unnecessary inference. Empirically, we construct a dataset of math word problems injected with various number of irrelevant axioms that vary in semantic overlap with the goal theorem. We find that current LMs show marked accuracy declines under such conditions -- even with minimal, domain-consistent distractions -- and the proofs they generate frequently exhibit detours through irrelevant inferences.
Permanent link
Publication status
published
Editor
Book title
Journal / series
Volume
Pages / Article No.
22305
Publisher
OpenReview
Event
39th Annual Conference on Neural Information Processing Systems (NeurIPS 2025)
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Language model; Logic programming; Proof; Arithmetic reasoning; Math word problem; Search; Reasoning; Efficiency
Organisational unit
09682 - Cotterell, Ryan / Cotterell, Ryan
Notes
Postersession on December 5, 2025