论文标题
人类标签变化的“问题”：关于数据，建模和评估的地面真相
The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation
论文作者
论文摘要
标记中的人类变异通常被认为是噪声。机器学习的注释项目（ML）旨在最大程度地降低人类标签变化，并以最大化数据质量和最大化机器学习指标的假设。但是，这种传统的做法假定存在一个基础真理，并且忽略了由于分歧，注释或多个合理的答案而导致的标记中存在真正的人类变异。在这个立场论文中，我们认为这个人类标签变化的大开放问题持续存在，并且需要更多的关注才能向前推进我们的领域。这是因为人类标签变化会影响ML管道的所有阶段：数据，建模和评估。但是，很少有作品共同考虑所有这些维度。现有研究是分散的。我们调和了以前提出的人类标签变体的不同概念，提供了一个具有未聚集标签的公共可用数据集的存储库，描述了到目前为止提出的方法，确定差距并提出了前进的方法。随着数据集变得越来越多的可用性，我们希望这种对“问题”的合成观点将导致关于可能从根本上设计新方向的可能策略的公开讨论。
Human variation in labeling is often considered noise. Annotation projects for machine learning (ML) aim at minimizing human label variation, with the assumption to maximize data quality and in turn optimize and maximize machine learning metrics. However, this conventional practice assumes that there exists a ground truth, and neglects that there exists genuine human variation in labeling due to disagreement, subjectivity in annotation or multiple plausible answers. In this position paper, we argue that this big open problem of human label variation persists and critically needs more attention to move our field forward. This is because human label variation impacts all stages of the ML pipeline: data, modeling and evaluation. However, few works consider all of these dimensions jointly; and existing research is fragmented. We reconcile different previously proposed notions of human label variation, provide a repository of publicly-available datasets with un-aggregated labels, depict approaches proposed so far, identify gaps and suggest ways forward. As datasets are becoming increasingly available, we hope that this synthesized view on the 'problem' will lead to an open discussion on possible strategies to devise fundamentally new directions.
