Goel, A., Fernando, B., Keller, F., & Bilen, H. (2023). Semi-supervised multimodal coreference resolution in image narrations. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2023.emnlp-main.682
Abstract:
In this paper, we study multimodal coreference resolution, specifically where a series of sentences ie narration is paired with an image. This poses significant challenges due to fine-grained image-text alignment, the inherent ambiguity present in narrative language and unavailability of large annotated training data.
To tackle these challenges, we present a data efficient semi-supervised approach that utilizes image-narration pairs to resolve coreferences and narrative grounding in a multimodal context. Our approach incorporates losses for both labeled and unlabeled data within a cross-modal framework. Through rigorous evaluation, we demonstrate that our proposed approach outperforms strong baselines both quantitatively and qualitatively for the tasks of coreference resolution and narrative grounding.
License type:
Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)
Funding Info:
This research / project is supported by the National Research Foundation, Singapore - NRF Fellowship
Grant Reference no. : NRF-NRFF14-2022-0001