On the Proper Treatment of Tokenization in Psycholinguistics
OPEN ACCESS
Loading...
Author / Producer
Date
2024-11
Publication Type
Conference Paper
ETH Bibliography
yes
Citations
Altmetric
OPEN ACCESS
Data
Rights / License
Abstract
Language models are widely used in computational psycholinguistics to test theories that relate the negative log probability (the surprisal) of a region of interest (a substring of characters) under a language model to its cognitive cost experienced by readers, as operationalized, for example, by gaze duration on the region. However, the application of modern language models to psycholinguistic studies is complicated by the practice of using tokenization as an intermediate step in training a model. Doing so results in a language model over *token* strings rather than one over character strings. Vexingly, regions of interest are generally misaligned with these token strings. The paper argues that token-level language models should be (approximately) marginalized into character-level language models before they are used in psycholinguistic studies to compute the surprisal of a region of interest; then, the marginalized character-level language model can be used to compute the surprisal of an arbitrary character substring, which we term a focal area, that the experimenter may wish to use as a predictor. Our proposal of marginalizing a token-level model into a character-level one solves this misalignment issue independently of the tokenization scheme. Empirically, we discover various focal areas whose surprisal is a better psychometric predictor than the surprisal of the region of interest itself.
Permanent link
Publication status
published
Book title
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Journal / series
Volume
Pages / Article No.
18556 - 18572
Publisher
Association for Computational Linguistics
Event
29th Conference on Empirical Methods in Natural Language Processing (EMNLP 2024)
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Organisational unit
09682 - Cotterell, Ryan / Cotterell, Ryan
Notes
Funding
Related publications and datasets
Is supplemented by: https://github.com/rycolab/psycho-toke