论文标题
关于使用预训练的语言模型对虚假相关性的鲁棒性的实证研究
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models
论文作者
论文摘要
最近的工作表明，诸如BERT之类的预训练的语言模型可以改善数据集中的虚假相关性的鲁棒性。对这些结果感兴趣，我们发现他们成功的关键是从虚假相关性不存在的少数反例中概括。当这种少数族裔的例子稀缺时，预训练的模型的性能与从头开始训练的模型一样差。在极少数的情况下，我们建议使用多任务学习（MTL）来改善概括。我们对自然语言推断和释义识别的实验表明，MTL具有正确的辅助任务可显着提高挑战性示例的性能，而不会损害分布性能。此外，我们表明，MTL的收益主要来自少数族裔的概括。我们的结果强调了数据多样性对于克服虚假相关性的重要性。
Recent work has shown that pre-trained language models such as BERT improve robustness to spurious correlations in the dataset. Intrigued by these results, we find that the key to their success is generalization from a small amount of counterexamples where the spurious correlations do not hold. When such minority examples are scarce, pre-trained models perform as poorly as models trained from scratch. In the case of extreme minority, we propose to use multi-task learning (MTL) to improve generalization. Our experiments on natural language inference and paraphrase identification show that MTL with the right auxiliary tasks significantly improves performance on challenging examples without hurting the in-distribution performance. Further, we show that the gain from MTL mainly comes from improved generalization from the minority examples. Our results highlight the importance of data diversity for overcoming spurious correlations.
