Pan, Z., Tao, R., Xu, C., & Li, H. (2022). Selective Listening by Synchronizing Speech With Lips. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30, 1650–1664. https://doi.org/10.1109/taslp.2022.3153258
Abstract:
A speaker extraction algorithm seeks to extract the speech of a target speaker from a multi-talker speech mixture when given a cue that represents the target speaker, such as a pre-enrolled speech utterance, or an accompanying video track. Visual cues are particularly useful when a pre-enrolled speech is not available. In this work, we don't rely on the target speaker's pre-enrolled speech, but rather use the target speaker's face track as the speaker cue, that is referred to as the auxiliary reference, to form an attractor towards the target speaker. We advocate that the temporal synchronization between the speech and its accompanying lip movements is a direct and dominant audio-visual cue. Therefore, we propose a self-supervised pre-training strategy, to exploit the speech-lip synchronization cue for target speaker extraction, which allows us to leverage abundant unlabeled in-domain data. We transfer the knowledge from the pre-trained model to the attractor encoder of the speaker extraction network. We show that the proposed speaker extraction network outperforms various competitive baselines in terms of signal quality, perceptual quality, and intelligibility, achieving state-of-the-art performance.
License type:
Attribution 4.0 International (CC BY 4.0)
Funding Info:
This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - AME Programmatic Funding Scheme
Grant Reference no. : A18A2b0046
This research / project is supported by the National Research Foundation Singapore - Human-Robot Interaction Phase 1
Grant Reference no. : 192 25 00054
This research / project is supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy - Germany’s Excellence Strategy (University Allowance, University of Bremen)
Grant Reference no. : EXC 2077
This research / project is supported by the Chinese University of Hong Kong, Shenzhen - N/A
Grant Reference no. : UDF01002333, UF02002333