Parmar, P., Peh, E., & Fernando, B. (2025). Learning to Visually Connect Actions and Their Effects. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 1477–1487. https://doi.org/10.1109/wacv61041.2025.00151
Abstract:
We introduce the novel concept of visually Connecting Actions and Their Effects (CATE) in video understanding. CATE can have applications in areas like task planning and learning from demonstration. We identify and explore two different aspects of the concept of CATE: Action Selection (AS) and Effect-Affinity Assessment (EAA), where video understanding models connect actions and effects at semantic and fine-grained levels, respectively. We design various baseline models for AS and EAA. Despite the intuitive nature of the task, we observe that models struggle, and humans outperform them by a large margin. Our experiments show that in solving AS & EAA, models learn intuitive properties like object tracking and encoding pose-related features without explicit supervision. We demon-strate that CATE can be an effective self-supervised task for learning video representations from unlabeled videos. The study aims to showcase the fundamental nature and versatility of CATE, with the hope of inspiring advanced formulations and models. Dataset & Code will be publicly released.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation, Singapore - NRF Fellowship
Grant Reference no. : NRF-NRFF14-2022-0001
This research / project is supported by the Agency for Science, Technology and Research (A*STAR), Science and Engineering Research Council - Central Research Fund
Grant Reference no. :
This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - Centre for Frontier AI Research (CFAR)
Grant Reference no. :