Natural Language Video Localization: A Revisit in Span-based Question Answering Framework

Page view(s)
5
Checked on Oct 23, 2022
Natural Language Video Localization: A Revisit in Span-based Question Answering Framework
Title:
Natural Language Video Localization: A Revisit in Span-based Question Answering Framework
Other Titles:
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publication Date:
23 February 2021
Citation:
Zhang, H., Sun, A., Jing, W., Zhen, L., Zhou, J. T., & Goh, R. S. M. (2021). Natural Language Video Localization: A Revisit in Span-based Question Answering Framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1–1. doi:10.1109/tpami.2021.3060449
Abstract:
Natural Language Video Localization (NLVL) aims to locate a target moment from an untrimmed video that semantically corresponds to a text query. Existing approaches mainly solve the NLVL problem from the perspective of computer vision by formulating it as ranking, anchor, or regression tasks. These methods suffer from large performance degradation when localizing on long videos. In this work, we address the NLVL from a new perspective, \ie span-based question answering (QA), by treating the input video as a text passage. We propose a video span localizing network (VSLNet), on top of the standard span-based QA framework (named VSLBase), to address NLVL. VSLNet tackles the differences between NLVL and span-based QA through a simple yet effective query-guided highlighting (QGH) strategy. QGH guides VSLNet to search for the matching video span within a highlighted region. To address the performance degradation on long videos, we further extend VSLNet to VSLNet-L by applying a multi-scale split-and-concatenation strategy to locate the target moment accurately. Extensive experiments show that the proposed methods outperform the state-of-the-art methods; VSLNet-L addresses the issue of performance degradation on long videos. Our study suggests that the span-based QA framework is an effective strategy to solve the NLVL problem.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the A*STAR - Human-Robot Collaborative AI for Advanced Manufacturing and Engineering (AME)
Grant Reference no. : A18A2b0046
Description:
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works
ISSN:
0162-8828
1939-3539
Files uploaded:

File Size Format Action
2021-tpami-natural-language-video-localization-a-revisit-in-span-based-question-answering-framework.pdf 4.92 MB PDF Request a copy