论文标题
关于敏感性和准确性之间的关系
On the Relation between Sensitivity and Accuracy in In-context Learning
论文作者
论文摘要
在上下文学习（ICL）对提示的过敏性遭受过敏性，因此在实际情况下它不可靠。我们研究ICL对多种扰动类型的敏感性。首先，我们发现标签偏差掩盖了真正的灵敏度，因此先前的工作可能会大大低估ICL灵敏度。其次，我们观察到ICL敏感性和准确性之间存在很强的负相关性：对扰动敏感的预测不太可能正确。在这些发现的激励下，我们提出了\ textsc {sensel}，这是一种避免敏感预测的几个选择性预测方法。十个分类数据集上的实验表明，\ textsc {sensel}始终优于两个常用的基于置信的基于置信的基准基准基于弃权决策。
In-context learning (ICL) suffers from oversensitivity to the prompt, making it unreliable in real-world scenarios. We study the sensitivity of ICL with respect to multiple perturbation types. First, we find that label bias obscures the true sensitivity, and therefore prior work may have significantly underestimated ICL sensitivity. Second, we observe a strong negative correlation between ICL sensitivity and accuracy: predictions sensitive to perturbations are less likely to be correct. Motivated by these findings, we propose \textsc{SenSel}, a few-shot selective prediction method that abstains from sensitive predictions. Experiments on ten classification datasets show that \textsc{SenSel} consistently outperforms two commonly used confidence-based and entropy-based baselines on abstention decisions.
