论文标题
框架：评估自由文本理由的理由标签一致性指标
FRAME: Evaluating Rationale-Label Consistency Metrics for Free-Text Rationales
论文作者
论文摘要
遵循人类的交流方式，自由文本的理由旨在使用自然语言来解释神经语言模型（LM）行为。但是，自由文本的理由的不受约束的本质使它们容易幻觉，因此必须拥有自由文本理由质量的指标很重要。现有的自由文本基本原理指标衡量了LM预测标签的基本原理的一致性，但是没有评估此类指标的可靠性的协议。因此，我们提出了框架，这是评估自由文本理性的评估理由标签一致性（RLC）指标的框架。框架基于三个公理：（1）良好的指标应产生最高分数的参考原理，这可以通过构造最大化RLC； （2）良好的指标应对理由的语义扰动适当敏感； （3）良好的指标应该对LM的任务性能的变化具有鲁棒性。在三个文本分类数据集中，我们表明现有的RLC指标无法满足所有三个帧公理，因为它们是通过模型预处理实现的，该预处理弄乱了度量标准的信号。然后，我们引入了一个非预先启动的RLC指标，该指标在（1）和（3）上大大优于基线，同时在（2）上进行了竞争性。最后，我们讨论使用RLC评估自由文本理由的局限性。
Following how humans communicate, free-text rationales aim to use natural language to explain neural language model (LM) behavior. However, free-text rationales' unconstrained nature makes them prone to hallucination, so it is important to have metrics for free-text rationale quality. Existing free-text rationale metrics measure how consistent the rationale is with the LM's predicted label, but there is no protocol for assessing such metrics' reliability. Thus, we propose FRAME, a framework for evaluating rationale-label consistency (RLC) metrics for free-text rationales. FRAME is based on three axioms: (1) good metrics should yield highest scores for reference rationales, which maximize RLC by construction; (2) good metrics should be appropriately sensitive to semantic perturbation of rationales; and (3) good metrics should be robust to variation in the LM's task performance. Across three text classification datasets, we show that existing RLC metrics cannot satisfy all three FRAME axioms, since they are implemented via model pretraining which muddles the metric's signal. Then, we introduce a non-pretraining RLC metric that greatly outperforms baselines on (1) and (3), while performing competitively on (2). Finally, we discuss the limitations of using RLC to evaluate free-text rationales.
