Comparing Classification and Generation Approaches to Situated Reasoning with Vision-language Pre-trained Models

Comparing Classification and Generation Approaches to Situated Reasoning with Vision-language Pre-trained Models

European Conference on Computer Vision - Machine Visual Common Sense (ECCV-MVCS) workshop (2022)

https://eccv2022.ecva.net/

Xin Huang, Hui Li Tan, Jung Jae Kim

multimodal learning, Visual Question Answering

23 October 2022

0

Situated Reasoning in Real-World Videos (STAR) is a new benchmark for evaluating situated reasoning ability through situation abstraction and logic-grounded question answering on real-world videos. In this paper, we present our submission to the STAR challenge which achieves the top-1 result for the situated question reasoning. We investigated two approaches to utilizing a vision-language pre-trained model, including classification and generation methods, and we show that the generation method outperforms the classification method for all question types of the challenge. We also compared the methods with other baselines, including another vision-language pre-trained model, and discuss why different vision-language pre-trained models show significant performance gap for the STAR challenge.

Publisher Copyright

This research / project is supported by the A*STAR - AME Programmatic Funding
Grant Reference no. : A18A2b0046

https://oar.a-star.edu.sg/communities-collections/articles/18771

ECCV_2022_MVCS_STAR_challenge_final

Institute for Infocomm Research