K Chan, F Liu, G Lin, CS Foo, W Lin. Robust-PIFu: Robust Pixel-aligned Implicit Function for 3D Human Digitalization from a Single Image. ICLR 2025.
Abstract:
Existing methods for 3D clothed human digitalization perform well when the input image is captured in ideal conditions that assume the lack of any occlusion.
However, in reality, images may often have occlusion problems such as incomplete observation of the human subject’s full body, self-occlusion by the human
subject, and non-frontal body pose. When given such input images, these existing methods fail to perform adequately. Thus, we propose Robust-PIFu, a pixelaligned implicit model that capitalized on large-scale, pretrained latent diffusion
models to address the challenge of digitalizing human subjects from non-ideal
images that suffer from occlusions.
Robust-PIfu offers four new contributions. Firstly, we propose a ‘disentangling’
latent diffusion model. This diffusion model, pretrained on billions of images,
takes in any input image and removes external occlusions, such as inter-person
occlusions, from that image. Secondly, Robust-PIFu addresses internal occlusions like self-occlusion by introducing a ‘penetrating’ latent diffusion model.
This diffusion model outputs multi-layered normal maps that by-pass occlusions
caused by the human subject’s own limbs or other body parts (i.e. self-occlusion).
Thirdly, in order to incorporate such multi-layered normal maps into a pixelaligned implicit model, we introduce our Layered-Normals Pixel-aligned Implicit Model, which improves the structural accuracy of predicted clothed human
meshes. Lastly, Robust-PIFu proposes an optional super-resolution mechanism
for the multi-layered normal maps. This addresses scenarios where the input image is of low or inadequate resolution. Though not strictly related to occlusion,
this is still an important subproblem. Our experiments show that Robust-PIFu outperforms current SOTA methods both qualitatively and quantitatively. Our code
will be released to the public.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the Agency for Science, Technology and Research (A*STAR) - Manufacturing, Trade, and Connectivity Programmatic Funds
Grant Reference no. : M23L7b0021
This research / project is supported by the Ministry of Education - Academic Research Fund Tier 2
Grant Reference no. : MOE-T2EP20223-0001