Hongye Tan, Xiaoyue Wang, Yu Ji, Ru Li, Xiao-Li Li, Zhiwei Hu, Yunxiao Zhao and Xiaoqi Han, GCRC: A New Challenging MRC Dataset from Gaokao Chinese for Explainable Evaluation, ACL (Findings), 2021.
Abstract:
Recently, driven by numerous publicly available machine reading comprehension (MRC) datasets, RC systems have made some progress. These datasets, however, have two major limitations: 1) the defined tasks are relatively simple, and 2) they do not provide explainable evaluation which is critical to objectively and comprehensively review the reasoning capabilities of current RC systems. In this paper, we propose GCRC, a new dataset with challenging and high-quality multi-choice questions, collected from
Gaokao Chinese (Chinese subject from the National College Entrance Examination of China). We have manually labelled three types of evidence to evaluate RC systems’ reasoning process: 1) sentence-level relevant supporting facts in an article required for answering a given question, 2) error reason of a distractor (i.e., an incorrect option) for explaining why a distractor should be eliminated, which is an important reasoning step for multi-choice questions, and 3) types of reasoning skills required for answering questions. Extensive experiments show that our proposed dataset is more challenging and very useful for identifying the limitations of existing RC systems in an explainable way, facilitating researchers to develop novel machine learning and reasoning approaches to tackle this challenging research problem.
License type:
Publisher Copyright
Funding Info:
This research is supported by core funding from: Institute for Infocomm Research
Grant Reference no. : Nil