论文标题
关于人类标签不确定性的影响
On the Ramifications of Human Label Uncertainty
论文作者
论文摘要
人类在数据标记过程中表现出分歧。我们将这种分歧称为人类标签不确定性。在这项工作中，我们研究了人类标签不确定性的后果（HLU）。我们对现有的不确定性估计算法的评估（随着HLU的存在）表明，现有的不确定性指标和算法本身的局限性是响应HLU的。同时，我们观察到预测性不确定性和普遍性的不当作用。为了减轻不适当的效果，我们引入了一种新型的基于基于标签的标签稀释训练方案，而无需大量的人类标签。具体而言，我们首先选择了具有低感知质量的样品子集，该样品由图像的统计规律性排名。然后，我们将单独的标签分配给该子集中的每个样本，以获得带有稀释标签的训练集。我们的实验和分析表明，基于NSS的标签稀释的培训减轻了HLU引起的不当作用。
Humans exhibit disagreement during data labeling. We term this disagreement as human label uncertainty. In this work, we study the ramifications of human label uncertainty (HLU). Our evaluation of existing uncertainty estimation algorithms, with the presence of HLU, indicates the limitations of existing uncertainty metrics and algorithms themselves in response to HLU. Meanwhile, we observe undue effects in predictive uncertainty and generalizability. To mitigate the undue effects, we introduce a novel natural scene statistics (NSS) based label dilution training scheme without requiring massive human labels. Specifically, we first select a subset of samples with low perceptual quality ranked by statistical regularities of images. We then assign separate labels to each sample in this subset to obtain a training set with diluted labels. Our experiments and analysis demonstrate that training with NSS-based label dilution alleviates the undue effects caused by HLU.
