Zhou, X., Yılmaz, E., Long, Y., Li, Y., Li, H. (2020). Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition. Interspeech 2020, 1042–1046. https://doi.org/10.21437/interspeech.2020-2488
Abstract:
Code-switching (CS) occurs when a speaker alternates words of two or more languages within a single sentence or across sentences. Automatic speech recognition (ASR) of CS speech has to deal with two or more languages at the same time. In this study, we propose a Transformer-based architecture with two symmetric language-specific encoders to capture the individual language attributes, that improve the acoustic representation of each language. These representations are combined using a language-specific multi-head attention mechanism in the decoder module. Each encoder and its corresponding attention module in the decoder are pre-trained using a large monolingual corpus aiming to alleviate the impact of limited CS training data. We call such a network a multi-encoder-decoder (MED) architecture. Experiments on the SEAME corpus show that the proposed MED architecture achieves 10.2% and 10.8% relative error rate reduction on the CS evaluation sets with Mandarin and English as the matrix language respectively.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation Singapore - National Robotics Programme: AI Speech Lab
Grant Reference no. : AISG-100E-2018-006
This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - AME Programmatic Funding Scheme
Grant Reference no. : A18A2b0046
This research / project is supported by the National Natural Science Foundation of China - NA
Grant Reference no. : 61701306
This research / project is supported by the National Research Foundation Singapore - National Robotics Programme: Human-Robot Interaction Phase 1
Grant Reference no. : 1922500054