Goel, A., Fernando, B., Keller, F., & Bilen, H. (2023, October 1). Who are you referring to? Coreference resolution in image narrations. 2023 IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/iccv51070.2023.01399
Abstract:
Coreference resolution aims to identify words and phrases which refer to the same entity in a text, a core task in natural language processing. In this paper, we extend this task to resolving coreferences in long-form narrations of visual scenes. First, we introduce a new dataset with annotated coreference chains and their bounding boxes, as most existing image-text datasets only contain short sentences without coreferring expressions or labeled chains. We propose a new technique that learns to identify coref-erence chains using weak supervision, only from image-text pairs and a regularization using prior linguistic knowledge. Our model yields large performance gains over several strong baselines in resolving coreferences. We also show that coreference resolution helps improve grounding narratives in images.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation - NRF Fellowship
Grant Reference no. : NRF-NRFF14-2022-0001