CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

Page view(s)
36
Checked on Jul 08, 2024
CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
Title:
CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
Journal Title:
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Keywords:
Publication Date:
10 December 2023
Citation:
Li, M., Shi, T., Ziems, C., Kan, M.-Y., Chen, N., Liu, Z., & Yang, D. (2023). CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2023.emnlp-main.92
Abstract:
Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs’ annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline.
License type:
Attribution 4.0 International (CC BY 4.0)
Funding Info:
This research / project is supported by the National Research Foundation - AI Singapore Programme
Grant Reference no. : AISG2-GC-2022-005

Acknowledge grants from A*STAR Computing and Information Science (ACIS) Scholarship, NSF Graduate Research Fellowship under Grant No. DGE-2039655 and Office of Naval Research.
Description:
ISBN:
2023.emnlp-main.92