Singh, A., & Jayavelu, S. (2024). Robust Representation Learning With Self-Distillation For Domain Generalization. 2024 IEEE International Conference on Image Processing (ICIP), 1179–1185. https://doi.org/10.1109/icip51287.2024.10648135
Abstract:
Despite the recent success of deep neural networks, there remains a need for effective methods to enhance domain generalization using vision transformers. In this paper, we propose a novel domain generalization technique called Robust Representation Learning with Self-Distillation (RRLD) comprising i) intermediate-block self-distillation and ii) augmentation guided self-distillation to improve the generalization capabilities of transformer-based models on unseen domains. This approach enables the network to learn robust and general features that are invariant to different augmentations and domain shifts while effectively mitigating overfitting to source domains. To evaluate the effectiveness of our proposed method, we perform extensive experiments on PACS [1] and OfficeHome [2] benchmark datasets, as well as an industrial wafer semiconductor defect dataset [3]. The results demonstrate that RRLD achieves robust and accurate generalization performance. We observe an average accuracy improvement in the range of 1.2% to 2.3% over the state-of-the-art on the three datasets.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the Agency for Science, Technology and Research - Advanced Manufacturing and Engineering (AME) Programmatic Fund
Grant Reference no. : A1898b0043