GCRC: A New MRC Dataset from Gaokao Chinese for Explainable Evaluation

Page view(s)
74
Checked on Dec 04, 2024
GCRC: A New MRC Dataset from Gaokao Chinese for Explainable Evaluation
Title:
GCRC: A New MRC Dataset from Gaokao Chinese for Explainable Evaluation
Journal Title:
ACL
DOI:
Keywords:
Publication Date:
04 August 2021
Citation:
Hongye Tan, Xiaoyue Wang, Yu Ji, Ru Li, Xiao-Li Li, Zhiwei Hu, Yunxiao Zhao and Xiaoqi Han, GCRC: A New Challenging MRC Dataset from Gaokao Chinese for Explainable Evaluation, ACL (Findings), 2021.
Abstract:
Recently, driven by numerous publicly available machine reading comprehension (MRC) datasets, RC systems have made some progress. These datasets, however, have two major limitations: 1) the defined tasks are relatively simple, and 2) they do not provide explainable evaluation which is critical to objectively and comprehensively review the reasoning capabilities of current RC systems. In this paper, we propose GCRC, a new dataset with challenging and high-quality multi-choice questions, collected from Gaokao Chinese (Chinese subject from the National College Entrance Examination of China). We have manually labelled three types of evidence to evaluate RC systems’ reasoning process: 1) sentence-level relevant supporting facts in an article required for answering a given question, 2) error reason of a distractor (i.e., an incorrect option) for explaining why a distractor should be eliminated, which is an important reasoning step for multi-choice questions, and 3) types of reasoning skills required for answering questions. Extensive experiments show that our proposed dataset is more challenging and very useful for identifying the limitations of existing RC systems in an explainable way, facilitating researchers to develop novel machine learning and reasoning approaches to tackle this challenging research problem.
License type:
Publisher Copyright
Funding Info:
This research is supported by core funding from: Institute for Infocomm Research
Grant Reference no. : Nil
Description:
ISSN:
Nil
Files uploaded:

File Size Format Action
acl-tanhongye.pdf 1.15 MB PDF Open