J. H. M. Wong, M. Huzaifah, H. B. Sailor, S. Sun, K. M. Tan, B. Wang, Q. Wang, W. Zhang, X. Zou, N. F. Chen, and A. T. Aw, "Diversity and complementarity of speech encoders across diverse tasks in a multi-modal large language model," in Proc. ASRU, Honolulu, USA, Dec 2025
Abstract:
A Large Language Model (LLM) can be extended to understand speech inputs by using a speech encoder to compute embeddings from the speech, which are then used with a text prompt. Diverse information is expressed in speech and a wide variety of tasks can be performed. Different speech encoders may specialise toward different information types and tasks. This complementarity can be leveraged upon by using multiple speech encoders. This paper presents a comprehensive analysis of the diversity and complementarity between open-source speech encoders, when used in a multi-modal LLM framework. Experiments identify the encoders that excel in each type of downstream task, thereby guiding future system design. The diversity between encoders is measured, showing that Whisper tends to behave more differently. Diversity between encoders is compared across tasks, showing that semantic tasks tend to yield more diverse predictions. Early and late fusion show that complementarity can yield improvements.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation - National Large Language Models Funding Initiative
Grant Reference no. :