论文标题
调查AUC指标的故障模式，并探索用于评估安全系统中系统的替代方法
Investigating the Failure Modes of the AUC metric and Exploring Alternatives for Evaluating Systems in Safety Critical Applications
论文作者
论文摘要
随着与黑匣子模型使用相关的安全要求的重要性越来越重要，对模型的选择性答案能力的评估至关重要。为此目的，曲线下的面积（AUC）用作度量。我们发现AUC的局限性；例如，具有较高AUC的模型在执行选择性答案时并不总是更好。我们提出了三个固定确定限制的替代指标。在实验十种模型时，我们使用新指标的结果表明，更新和更大的预训练模型并不一定在选择性答案中表现出更好的性能。我们希望我们的见解将有助于开发出针对安全至关重要的应用量身定制的更好模型。
With the increasing importance of safety requirements associated with the use of black box models, evaluation of selective answering capability of models has been critical. Area under the curve (AUC) is used as a metric for this purpose. We find limitations in AUC; e.g., a model having higher AUC is not always better in performing selective answering. We propose three alternate metrics that fix the identified limitations. On experimenting with ten models, our results using the new metrics show that newer and larger pre-trained models do not necessarily show better performance in selective answering. We hope our insights will help develop better models tailored for safety-critical applications.
