Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
OPEN ACCESS
Loading...
Author / Producer
Date
2021-07
Publication Type
Journal Article
ETH Bibliography
yes
Citations
Altmetric
OPEN ACCESS
Data
Rights / License
Abstract
Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, only the contextualized BERT embeddings outperform the baselines in our experiments, which raises the need for further research. Finally, EEG data shows to be particularly promising when limited training data is available.
Permanent link
Publication status
published
External links
Editor
Book title
Journal / series
Volume
15
Pages / Article No.
659410
Publisher
Frontiers Media
Event
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
EEG; natural language processing; frequency bands; brain activity; machine learning; multi-modal learning; physiological data; neural network
Organisational unit
09588 - Zhang, Ce (ehemalig) / Zhang, Ce (former)