Zhang, H., Sun, A., Jing, W., Zhou, J. T. (2020). Span-based Localizing Network for Natural Language Video Localization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.585
Abstract:
Given an untrimmed video and a text query,
natural language video localization (NLVL) is
to locate a matching span from the video that
semantically corresponds to the query. Existing solutions formulate NLVL either as a ranking task and apply multimodal matching architecture, or as a regression task to directly
regress the target video span. In this work,
we address NLVL task with a span-based QA
approach by treating the input video as text
passage. We propose a video span localizing network (VSLNet), on top of the standard
span-based QA framework, to address NLVL.
The proposed VSLNet tackles the differences
between NLVL and span-based QA through
a simple and yet effective query-guided highlighting (QGH) strategy. The QGH guides
VSLNet to search for matching video span
within a highlighted region. Through extensive experiments on three benchmark datasets,
we show that the proposed VSLNet outperforms the state-of-the-art methods; and adopting span-based QA framework is a promising
direction to solve NLVL.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - Advanced Manufacturing and Engineering (AME) Programmatic Fund
Grant Reference no. : A18A2b0046, A18A1b0045