Hu, D., Liang, J., Wang, X. & Foo, C.-S. (2024). Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:19304-19326
Abstract:
Unsupervised domain adaptation (UDA) has seen substantial efforts to improve model accuracy for an unlabeled target domain with the help of a labeled source domain. However, UDA models often exhibit poorly calibrated predictive uncertainty on target data, a problem that remains under-explored and poses risks in safety-critical UDA applications. The calibration problem in UDA is particularly challenging due to the absence of labeled target data and severe distribution shifts between domains. In this paper, we approach UDA calibration as a target-domain-specific unsupervised problem, different from mainstream solutions based on covariate shift. We introduce Pseudo-Calibration (PseudoCal), a novel post-hoc calibration framework. Our innovative use of inference-stage mixup synthesizes a labeled pseudo-target set capturing the structure of the real unlabeled target data. This turns the unsupervised calibration problem into a supervised one, easily solvable with temperature scaling. Extensive empirical evaluations across 5 diverse UDA scenarios involving 10 UDA methods consistently demonstrate the superior performance and versatility of PseudoCal over existing solutions.
License type:
Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)
Funding Info:
This research / project is supported by the Singapore Ministry of Education - Academic Research Fund Tier 1
Grant Reference no. : A-8001229-00-00
This research / project is supported by the Nataional Natural Science Foundation of China - NA
Grant Reference no. : 62276256
This research / project is supported by the Beijing Municipal Science & Technology Commission (BSTC) and the Administrative Commission of Zhongguancun Science Park - Beijing Nova Program
Grant Reference no. : Z211100002121108