论文标题
揭示更浅的启发式方法：使用三段论模式探索基于变压器的预训练的语言模型的自然语言推理能力
Uncovering More Shallow Heuristics: Probing the Natural Language Inference Capacities of Transformer-Based Pre-Trained Language Models Using Syllogistic Patterns
论文作者
论文摘要
在本文中，我们探讨了基于变压器的预训练的语言模型（PLM）使用的浅启发式方法，这些模型（PLMS）对自然语言推断（NLI）进行了微调。为此，我们基于三段式构建或拥有数据集，并评估了数据集中的许多模型的性能。我们发现证据表明这些模型在很大程度上依赖于某些浅启发式方法，在前提和假设之间涉及对称性和不对称性。我们认为，在我们的研究中缺乏概括，这正在成为该领域的活泼辩论的话题，这意味着PLM目前没有学习NLI，而是伪造的启发式方法。
In this article, we explore the shallow heuristics used by transformer-based pre-trained language models (PLMs) that are fine-tuned for natural language inference (NLI). To do so, we construct or own dataset based on syllogistic, and we evaluate a number of models' performance on our dataset. We find evidence that the models rely heavily on certain shallow heuristics, picking up on symmetries and asymmetries between premise and hypothesis. We suggest that the lack of generalization observable in our study, which is becoming a topic of lively debate in the field, means that the PLMs are currently not learning NLI, but rather spurious heuristics.
