Open access
Date
2023-12Type
- Journal Article
ETH Bibliography
yes
Altmetrics
Abstract
Audio-visual wake word spotting is a challenging multi-modal task that exploits visual information of lip motion patterns to supplement acoustic speech to improve overall detection performance. However, most audio-visual wake word spotting models are only suitable for simple single-speaker scenarios and require high computational complexity. Further development is hindered by complex multi-person scenarios and computational limitations in mobile environments. In this paper, a novel audio-visual model is proposed for on-device multi-person wake word spotting. Firstly, an attention-based audio-visual voice activity detection module is presented, which generates an attention score matrix of audio and visual representations to derive active speaker representation. Secondly, the knowledge distillation method is introduced to transfer knowledge from the large model to the on-device model to control the size of our model. Moreover, a new audio-visual dataset, PKU-KWS, is collected for sentence-level multi-person wake word spotting. Experimental results on the PKU-KWS dataset show that this approach outperforms the previous state-of-the-art methods. Show more
Permanent link
https://doi.org/10.3929/ethz-b-000603879Publication status
publishedExternal links
Journal / series
CAAI Transactions on Intelligence TechnologyVolume
Pages / Article No.
Publisher
Institution of Engineering and TechnologySubject
audio-visual fusion; human-computer interfacing; speech processingMore
Show all metadata
ETH Bibliography
yes
Altmetrics