Zhenglin Wan, Xingrui Yu, David Mark Bossens, Yueming Lyu, Qing Guo, Flint Xiaofeng Fan, Yew-Soon Ong, and Ivor W. Tsang. 2025. Diversifying policy behaviors via extrinsic behavior curiosity. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25), Vol. 267. JMLR.org, Article 2460, 62135–62154.
Abstract:
Imitation learning (IL) has shown promise in various applications (e.g. robot locomotion) but is often limited to learning a single expert policy, constraining behavior diversity and robustness in unpredictable real-world scenarios. To address this, we introduce Quality Diversity Inverse Reinforcement Learning (QD-IRL), a novel framework that integrates quality-diversity optimization with IRL methods, enabling agents to learn diverse behaviors from limited demonstrations. This work introduces Extrinsic Behavioral Curiosity (EBC), which allows agents to receive additional curiosity rewards from an external critic based on how novel the behaviors are with respect to a large behavioral archive. To validate the effectiveness of EBC in exploring diverse locomotion behaviors, we evaluate our method on multiple robot locomotion tasks. EBC improves the performance of QD-IRL instances with GAIL, VAIL, and DiffAIL across all included environments by up to 185%, 42%, and 150%, even surpassing expert performance by 20% in Humanoid. Furthermore, we demonstrate that EBC is applicable to Gradient-Arborescence-based Quality Diversity Reinforcement Learning (QD-RL) algorithms, where it substantially improves performance and provides a generic technique for learning behavioral diverse policies. The source code of this work is provided at https://github.com/vanzll/EBC.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority - Trust Tech Funding Initiative
Grant Reference no. : NA
This research / project is supported by the National Research Foundation, Singapore - National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP-2024- 004)
Grant Reference no. : AISG-NMLP-2024- 004