J. Wong and N. Chen, "Obtaining objective labels and analysing annotator subjectivity by using a Rasch model for ordinal speech processing," in Proc. ASRU, Honolulu, USA, Dec 2025
Abstract:
In datasets for subjective tasks, disagreement between annotators is often accommodated by recording annotations from multiple annotators for each datapoint. However, it is more convenient to train and evaluate models against a scalar reference. In ordinal tasks, where outputs follow a monotonic order, the standard approach of computing the scalar reference as either the mean, median, or majority vote of the multiple annotations does not consider the differing bias between groups of annotators and assumes linearity of the output. This paper proposes to compute the scalar reference using a Rasch model. This expresses differing annotator bias, avoids linear assumptions, and allows control of the set of confounding variables that should influence the reference. Demonstrations on MSP-Podcast emotion recognition and speechocean762 spoken language assessment show how to use the Rasch model to compute a scalar reference, analyse confounders in the dataset, and compare multiple trained models.
License type:
Publisher Copyright
Funding Info:
There was no specific funding for the research done