On-device audio-visual multi-person wake word spotting
OPEN ACCESS
Loading...
Author / Producer
Date
2023-12
Publication Type
Journal Article
ETH Bibliography
yes
Citations
Altmetric
OPEN ACCESS
Data
Abstract
Audio-visual wake word spotting is a challenging multi-modal task that exploits visual information of lip motion patterns to supplement acoustic speech to improve overall detection performance. However, most audio-visual wake word spotting models are only suitable for simple single-speaker scenarios and require high computational complexity. Further development is hindered by complex multi-person scenarios and computational limitations in mobile environments. In this paper, a novel audio-visual model is proposed for on-device multi-person wake word spotting. Firstly, an attention-based audio-visual voice activity detection module is presented, which generates an attention score matrix of audio and visual representations to derive active speaker representation. Secondly, the knowledge distillation method is introduced to transfer knowledge from the large model to the on-device model to control the size of our model. Moreover, a new audio-visual dataset, PKU-KWS, is collected for sentence-level multi-person wake word spotting. Experimental results on the PKU-KWS dataset show that this approach outperforms the previous state-of-the-art methods.
Permanent link
Publication status
published
External links
Editor
Book title
Journal / series
Volume
8 (4)
Pages / Article No.
1578 - 1589
Publisher
Institution of Engineering and Technology
Event
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
audio-visual fusion; human-computer interfacing; speech processing