Sugandhika, C., Li, C., Rajan, D., & Fernando, B. (2025). Situational Scene Graph for Structured Human-Centric Situation Understanding. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 9215–9225. https://doi.org/10.1109/wacv61041.2025.00893
Abstract:
Graph based representation has been widely used in modelling spatio-temporal relationships in video understanding. Although effective, existing graph-based approaches focus on capturing the human-object relationships while ignoring fine-grained semantic properties of the action components. These semantic properties are crucial for understanding the current situation, such as where does the action takes place, what tools are used and functional properties of the objects. In this work, we propose a graph-based representation called Situational Scene Graph (SSG) to encode both human-object relationships and the corresponding semantic properties. The semantic details are represented as predefined roles and values inspired by situation frame, which is originally designed to represent a single action. Based on our proposed representation, we introduce the task of situational scene graph generation and propose a multi-stage pipeline Interactive and Complementary Network (InComNet) to address the task. Given that the existing datasets are not applicable to the task, we further introduce a SSG dataset whose annotations consist of semantic role-value frames for human, objects and verb predicates of human-object relations. Finally, we demonstrate the effectiveness of our proposed SSG representation by testing on different downstream tasks. Experimental results show that the unified representation can not only benefit predicate classification and semantic role-value classification, but also benefit reasoning tasks on human-centric situation understanding. Code and data are available at https://github.com/LUNAProject22/SSG.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation, Singapore - NRF Fellowship
Grant Reference no. : NRF-NRFF14-2022-0001
This research / project is supported by the Agency for Science, Technology and Research, Science and Engineering Research Council - Central Research Fund (CRF)
Grant Reference no. :