Tang, Y., Puduppully, R., Liu, Z., & Chen, N. (2023). In-context Learning of Large Language Models for Controlled Dialogue Summarization: A Holistic Benchmark and Empirical Analysis. Proceedings of the 4th New Frontiers in Summarization Workshop. https://doi.org/10.18653/v1/2023.newsum-1.6
Abstract:
Large Language Models (LLMs) have shown significant performance in numerous NLP tasks, including summarization and controlled text generation. A notable capability of LLMs is in-context learning (ICL), where the model learns new tasks using input-output pairs in the prompt without any parameter update. However, the performance of LLMs in the context of few-shot abstractive dialogue summarization remains underexplored. This study evaluates various state-of-the-art LLMs on the SAMSum dataset within a few-shot framework. We assess these models in both controlled (entity control, length control, and person-focused planning) and uncontrolled settings, establishing a comprehensive benchmark in few-shot dialogue summarization. Our findings provide insights into summary quality and model controllability, offering a crucial reference for future research in dialogue summarization.
License type:
Attribution 4.0 International (CC BY 4.0)
Funding Info:
This research / project is supported by the National Research Foundation - CNRS@CREATE Program on Intelligent Modelling for Decision-making in Critical Urban Systems
Grant Reference no. : EC-2022-041
This research / project is supported by the A*STAR - Industry Alignment Pre-Positioning Fund
Grant Reference no. : H19/01/a0/023 - DCOF