Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Zhang, Y., He, R., Liu, Z., Bing, L., & Li, H. (2021). Bootstrapped Unsupervised Sentence Representation Learning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 5168–5180. https://doi.org/10.18653/v1/2021.acl-long.402
Abstract:
As high-quality labeled data is scarce, unsupervised sentence representation learning has attracted much attention. In this paper, we propose a new framework with a two-branch Siamese Network which maximizes the similarity between two augmented views of each sentence. Specifically, given one augmented view of the input sentence, the online network branch is trained by predicting the representation yielded by the target network of the same sentence under another augmented view. Meanwhile, the target network branch is bootstrapped with a moving average of the online network. The proposed method significantly outperforms other state-of-the-art unsupervised methods on semantic textual similarity (STS) and classification tasks. It can be adopted as a post-training procedure to boost the performance of the supervised methods. We further extend our method for learning multilingual sentence representations and demonstrate its effectiveness on cross-lingual STS tasks. Our code is available at https://github.com/yanzhangnlp/BSL.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation Singapore and Agency for Science, Technology and Research (A*STAR) - Human Robot Collaborative AI for Advanced Manufacturing and Engineering (AME)
Grant Reference no. : A18A2b0046
This research / project is supported by the National Research Foundation Singapore - National Robotics Programme: Human-Robot Interaction Phase 1
Grant Reference no. : 1922500054