D. Roy, R. Rajendiran and B. Fernando, "Interaction Region Visual Transformer for Egocentric Action Anticipation," 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2024, pp. 6726-6736, doi: 10.1109/WACV57701.2024.00660.
Abstract:
Human-object interaction (HOI) and temporal dynamics along the motion paths are the most important visual cues for egocentric action anticipation. Especially, interaction regions covering objects and the human hand reveal significant visual cues to predict future human actions. However, how to incorporate and capture these important visual cues in modern video Transformer architecture remains a challenge, especially because integrating inductive biases into Transformers is hard. We leverage the effective MotionFormer that models motion dynamics to incorporate interaction regions using spatial cross-attention and further infuse contextual information using trajectory cross-attention to obtain an interaction-centric video representation for action anticipation. We term our model InAViT which achieves state-of-the-art action anticipation performance on large-scale egocentric datasets EPICKTICHENS100 (EK100) and EGTEA Gaze+. On the EK100 evaluation server, InAViT is on top of the public leader board (at the time of submission) where it outperforms the second-best model by 3.3% on mean-top5 recall. We will release the code.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation, DSO National Laboratories - AI Singapore Program
Grant Reference no. : AISG2-RP-2020-016, AISG-RP2019-010
This research / project is supported by the Agency for Science, Technology and Research (A*STAR), Science and Engineering Research Council - Central Research Fund (CRF)
Grant Reference no. :
This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - Centre for Frontier AI Research (CFAR)
Grant Reference no. :