Ee, Y. K., Zhang, H., Matyasko, A., & Fernando, B. (2025). Deduce and Select Evidences with Language Models for Training-Free Video Goal Inference. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 5937–5947. https://doi.org/10.1109/wacv61041.2025.00579
Abstract:
We introduce ViDSE, a Video framework that Deduce and Selects visual Evidence for training-free video goal inference using language models. Unlike approaches that directly apply vision-language models (VLM) or combine VLM+LLM to process dense video visuals, ViDSE explicitly selects relevant visual evidence (e.g., frames) based on the hypothesis deduced by the LLM. This approach not only im-proves accuracy but also reveals the logical process behind the model's decisions, enhancing explainability. Our exper-iments demonstrate that this selection process significantly reduces ambiguity in the subsequent inference reasoning stage and outperforms VLM-only and VLM+LLM models on goal inference tasks such as CrossTask and COIN. We further validate ViDSE’ s generalizability and robustness on action recognition tasks, such as ActivityNet and UCF-101, under training-free and open-vocabulary conditions. We observe that ViDSE easily generalizes to other video tasks (e.g., action recognition) requiring filtering of redundant and irrelevant information.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation, Singapore - NRF Fellowship
Grant Reference no. : NRF-NRFF14-2022-0001
This research / project is supported by the Agency for Science, Technology and Research (A*STAR), Science and Engineering Research Council - Central Research Fund
Grant Reference no. :
This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - Centre for Frontier AI Research (CFAR)
Grant Reference no. :