GCM: Efficient video recognition with glance and combine module

Page view(s)
Checked on Jan 11, 2023
GCM: Efficient video recognition with glance and combine module
GCM: Efficient video recognition with glance and combine module
Other Titles:
Pattern Recognition
Publication Date:
11 August 2022
Zhou, Y., Huang, Z., Yang, X., Ang, M., & Ng, T. K. (2023). GCM: Efficient video recognition with glance and combine module. Pattern Recognition, 133, 108970. https://doi.org/10.1016/j.patcog.2022.108970
In this work, we present an efficient and powerful building block for video action recognition, dubbed \textit{Glance and Combine Module} (GCM). In order to obtain a broader perspective of the video features, GCM introduces an extra glancing operation with a larger receptive field over both the spatial and temporal dimensions, and combines features with different receptive fields for further processing. We show in our ablation studies that the proposed GCM is much more efficient than other forms of 3D spatio-temporal convolutional blocks. We build a series of GCM networks by stacking GCM repeatedly, and train them from scratch on the target datasets directly. On the Kinetics-400 dataset which focuses more on appearance rather than action, our GCM networks can achieve similar accuracy as others without pre-training on ImageNet. For the more action-centric recognition datasets such as Something-Something (V1 & V2) and Multi-Moments in Time, the GCM networks achieve state-of-the-art performance with less than two thirds the computational complexity of other models. With only 19.2 GFLOPs of computation, our GCMNet_15 can obtain 63.9% top-1 classification accuracy on Something-Something-V2 validation set under single-crop testing. On the fine-grained action recognition dataset FineGym, we beat the previous state-of-the-art accuracy achieved with 2-stream methods by more than 6% using only RGB input.
License type:
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
Funding Info:
There was no specific funding for the research done
Files uploaded:

File Size Format Action
pattern-recognition-gcm-final.pdf 658.16 KB PDF Request a copy