J. Zhuang, T. He, Y. Hou, Y. S. Ong, and Q. Zhang, “Learning Dynamic Frame Semantics for Multimodal Sentiment Analysis,”, IEEE TAFFC, 2026.
Abstract:
In this paper, we investigate how the mathematization of frame semantics can enhance multimodal sentiment analysis (MSA). Existing deep learning approaches to MSA typically focus on single semantic frames. In contrast, frame semantics, a well-recognized theory from linguistics, posits that language cognition and interpretation essentially rely on the coherence of multiple frames divided according to lexical units and
encapsulated with diverse background knowledge and conceptual structures. To facilitate MSA with frame semantics, we propose a new model, namely Dynamic-Aware Frame Semantic Analysis (DAFSA). DAFSA first models multimodal sequential features (e.g., text, audio, and vision) as multimodal semantic frames. Then, DAFSA can extract the representations that capture the subtle emotional expressions by dynamically learning and summarizing the coherent structures from intra- and crossmodal frames. Experimental results demonstrate that DAFSA significantly outperforms state-of-the-art approaches to MSA and foundation models whose size is larger than DAFSA up to 100×, showcasing the effectiveness of modeling frame semantics in MSA.
License type:
Publisher Copyright
Funding Info:
This work was supported in part by the National Key R&D Program of China under Grant 2024YFA1012700, the Liaoning Provincial Central-Guided Local Sci-Tech Development Program under Grant 2025JH6/101000005, the Dalian Science and Technology Innovation Fund under Grant 2024JJ12GX020, the Liaoning Provincial Natural Science Foundation Program under Grant 2024-MSBA-05, the Liaoning Provincial Science and Technology Program under Grant 2024JH2/102600046, the 111 Project under Grant D23006, the Aviation Transformation Programme-Airport (ATP Airport) of Singapore (Award No. ATP AIRPORT/2026/ARES/FLASH), and the National Research Foundation, Singapore under its NRF AI-for-Science (AI4S) Challenge Grant (NRF-AI4SCH-2025-0007). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.