MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction

Page view(s)
115
Checked on May 11, 2024
MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction
Title:
MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction
Journal Title:
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI:
Authors:
Keywords:
Publication Date:
10 July 2023
Citation:
Wang Jing, Aixin Sun, Zhang Hao, LI X.L., MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction, ACL 2023
Abstract:
Given a query, the task of Natural Language Video Localization (NLVL) is to localize a temporal moment in an untrimmed video that semantically matches the query. In this paper, we adopt a proposal-based solution that generates proposals (i.e., candidate moments) and then select the best matching proposal. On top of modeling the cross-modal interaction between candidate moments and the query, our proposed Moment Sampling DETR (MS-DETR) enables efficient moment-moment relation modeling. The core idea is to sample a subset of moments guided by the learnable templates with an adopted DETR framework. To achieve this, we design a multi-scale visual-linguistic encoder, and an anchor-guided moment decoder paired with a set of learnable templates. Experimental results on three public datasets demonstrate the superior performance of MS-DETR.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the A*STAR - RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP)
Grant Reference no. : N.A
Description:
ISSN:
Nil
Files uploaded:

File Size Format Action
acl-2023-ms-detr.pdf 3.54 MB PDF Open