论文标题
评估和表征人类理由
Evaluating and Characterizing Human Rationales
论文作者
论文摘要
评估机器生成原理质量的两种主要方法是：1）使用人类原理作为黄金标准； 2）基于理由如何影响模型行为的自动指标。但是，一个空旷的问题是，人类原理与这些自动指标的票据如何。分析各种数据集和模型，我们发现人类理由不一定在这些指标上表现良好。为了解开这一发现，我们提出了改进的指标，以说明模型依赖性基线性能。然后，我们提出了两种方法，以进一步表征理由质量，一种基于模型再培训，一种基于“保真度曲线”来揭示诸如无关和冗余之类的属性。我们的工作导致了可行的建议，以评估和表征理由。
Two main approaches for evaluating the quality of machine-generated rationales are: 1) using human rationales as a gold standard; and 2) automated metrics based on how rationales affect model behavior. An open question, however, is how human rationales fare with these automatic metrics. Analyzing a variety of datasets and models, we find that human rationales do not necessarily perform well on these metrics. To unpack this finding, we propose improved metrics to account for model-dependent baseline performance. We then propose two methods to further characterize rationale quality, one based on model retraining and one on using "fidelity curves" to reveal properties such as irrelevance and redundancy. Our work leads to actionable suggestions for evaluating and characterizing rationales.
