Lin, D., Cheng, Y., Guo, A., Mao, S., & Li, Y. (2025). PEVA-Net: Prompt-enhanced view aggregation network for zero/few-shot multi-view 3D shape recognition. Neurocomputing, 129590. https://doi.org/10.1016/j.neucom.2025.129590
Abstract:
Large vision-language models have impressively promote the performance of 2D visual recognition under zero/few-shot scenarios. In this paper, we focus on exploiting the large vision-language model, i.e., CLIP, to address zero/few-shot 3D shape recognition based on multi-view representations. The key challenge for both tasks is to generate a discriminative descriptor of the 3D shape represented by multiple view images under the scenarios of either without explicit training (zero-shot 3D shape recognition) or training with a limited number of data (few-shot 3D shape recognition). We analyze that both tasks are relevant and can be considered simultaneously. Specifically, leveraging the descriptor which is effective for zero-shot inference to guide the tuning of the aggregated descriptor under the few-shot training can significantly improve the few-shot learning efficacy. Hence, we propose Prompt-Enhanced View Aggregation Network (PEVA-Net) to simultaneously address zero/few-shot 3D shape recognition. Under the zero-shot scenario, we propose to leverage the prompts built up from candidate categories to enhance the aggregation process of multiple view-associated visual features. The resulting aggregated feature serves for effective zero-shot recognition of the 3D shapes. Under the few-shot scenario, we first exploit a ViT encoder to aggregate the view-associated visual features into a global descriptor. To tune the encoder, we propose a self-feature-distillation scheme via a feature distillation loss by treating the zero-shot descriptor as the guidance signal for the few-shot descriptor. This scheme can significantly enhance the few-shot learning efficacy. Without any pre-training process, our PEVA-Net can produce the state-of-the-art zero-shot 3D shape recognition performance on ModelNet40, ModelNet10 and ShapeNetCore 55 datasets with the accuracy of 84.48%, 93.50% and 74.65%. Under the 16-shot setting of ModelNet40, the proposed PEVA-Net also sets the state-of-the-art recognition accuracy of 90.64%. Extensive ablation experiments are conducted to analyze the superiority of the proposed PEVA-Net.
License type:
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
Funding Info:
This research / project is supported by the Agency for Science, Technology and Research - RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP)
Grant Reference no. : I2001E0073