Tan, B. C. Z., Chin, D. W. K., Liu, Z., Chen, N. F., & Lee, R. K.-W. (2025). Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PD. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 1550–1575. https://doi.org/10.18653/v1/2025.emnlp-main.81
Abstract:
Large Language Models (LLMs) can struggle to balance gullibility to misinformation and resistance to valid corrections in persuasive dialogues, a critical challenge for reliable deployment. We introduce **DuET-PD** (**Du**al **E**valuation for **T**rust in **P**ersuasive **D**ialogues), a framework evaluating multi-turn stance-change dynamics across dual dimensions: persuasion type (corrective/misleading) and domain (knowledge via MMLU-Pro, and safety via SALAD-Bench). We find that even a state-of-the-art model like GPT-4o achieves only 27.32% accuracy in MMLU-Pro under sustained misleading persuasions. Moreover, results reveal a concerning trend of increasing sycophancy in newer open-source models. To address this, we introduce Holistic DPO, a training approach balancing positive and negative persuasion examples. Unlike prompting or resist-only training, Holistic DPO enhances both robustness to misinformation and receptiveness to corrections, improving Llama-3.1-8B-Instruct’s accuracy under misleading persuasion in safety contexts from 4.21% to 76.54%. These contributions offer a pathway to developing more reliable and adaptable LLMs for multi-turn dialogue. Code is available at https://github.com/Social-AI-Studio/DuET-PD.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the AISG, IMDA, A*STAR - National Large Language Models Funding Initiative
Grant Reference no. : AISG-NMLP-2024-005, AISG-NMLP-2024-003, AISG-NMLP-2024- 004
This research / project is supported by the MOE - Academic Research Fund Tier 2
Grant Reference no. : T2EP20222-0036