Dissecting multimodality in VideoQA transformer models by impairing modality fusion

Dissecting multimodality in VideoQA transformer models by impairing modality fusion

ICML'24: Proceedings of the 41st International Conference on Machine Learning

https://proceedings.mlr.press/v235/rawal24a.html

Ishaan Singh Rawal, Alexander Matyasko, Shantanu Jaiswal, Basura Fernando, Cheston Tan

21 July 2024

Rawal, I.S., Matyasko, A., Jaiswal, S., Fernando, B. Tan, C.. (2024). Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:42213-42244 Available from https://proceedings.mlr.press/v235/rawal24a.html.

While VideoQA Transformer models demonstrate competitive performance on standard benchmarks, the reasons behind their success are not fully understood. Do these models capture the rich multimodal structures and dynamics from video and text jointly? Or are they achieving high scores by exploiting biases and spurious features? Hence, to provide insights, we design QUAG (QUadrant AveraGe), a lightweight and non-parametric probe, to conduct dataset-model combined representation analysis by impairing modality fusion. We find that the models achieve high performance on many datasets without leveraging multimodal representations. To validate QUAG further, we design QUAG-attention, a less-expressive replacement of self-attention with restricted token interactions. Models with QUAG-attention achieve similar performance with significantly fewer multiplication operations without any finetuning. Our findings raise doubts about the current models' abilities to learn highly-coupled multimodal representations. Hence, we design the CLAVI (Complements in LAnguage and VIdeo) dataset, a stress-test dataset curated by augmenting real-world videos to have high modality coupling. Consistent with the findings of QUAG, we find that most of the models achieve near-trivial performance on CLAVI. This reasserts the limitations of current models for learning highly-coupled multimodal representations, that is not evaluated by the current datasets (project page: https://dissect-videoqa.github.io).

Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)

This research / project is supported by the National Research Foundation, Singapore - NRF Fellowship Award
Grant Reference no. : NRF-NRFF14-2022-0001

This research / project is supported by the Agency for Science, Technology and Research - Central Research Fund
Grant Reference no. :

https://oar.a-star.edu.sg/communities-collections/articles/21773

NA

Institute of High Performance Computing