Implicit representation of an image can map arbitrary coordinates in the continuous domain to their corresponding color values, presenting a powerful capability for image reconstruction. Nevertheless, existing implicit representation approaches only focus on building continuous appearance mapping, ignoring the continuities of the semantic information across pixels. Consequently, achieving the desired reconstruction results becomes challenging when the semantic information within input image is corrupted, such as when a large region is missing. To address the issue, we suggest learning semantic-aware implicit representation (SAIR), that is, we make the implicit representation of each pixel rely on both its appearance and semantic information (e.g., which object does the pixel belong to). To this end, we propose a framework with two modules: (1) a semantic implicit representation (SIR) for a corrupted image. Given an arbitrary coordinate in the continuous domain, we can obtain its respective text-aligned embedding indicating the object the pixel belongs. (2) an appearance implicit representation (AIR) based on the SIR. Given an arbitrary coordinate in the continuous domain, we can reconstruct its color whether or not the pixel is missed in the input. We validate the novel semantic-aware implicit representation method on the image inpainting task, and the extensive experiments demonstrate that our method surpasses state-of-the-art approaches by a significant margin.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority - Trust Tech Funding Initiative
Grant Reference no. : DTC-RGC-04
This research / project is supported by the National Research Foundation, Singapore, and DSO National Laboratories - AI Singapore Programme
Grant Reference no. : AISG2-GC-2023-008
This research / project is supported by the Agency for Science, Technology and Research - Career Development Fund
Grant Reference no. : C233312028
Description:
This is a post-peer-review, pre-copyedit version of an article published in Lecture Notes in Computer Science. The final authenticated version is available online at: http://dx.doi.org/10.1007/978-3-031-73235-5_18