论文标题
机器阅读，快速慢：何时“理解”语言？
Machine Reading, Fast and Slow: When Do Models "Understand" Language?
论文作者
论文摘要
目前，自然语言理解（NLU）中最根本的两个挑战是：（a）如何以“正确”的原因确定基于深度学习的模型是否在NLU基准上得分很高； （b）了解这些原因甚至是什么。我们研究了关于两个语言“技能”的阅读理解模型的行为：核心分辨率和比较。我们为从系统中预期的推理步骤提出了一个定义，该系统将“缓慢阅读”，并将其与五个模型的各种规模的伯特家族的行为进行比较，这是通过显着分数和反事实解释观察到的。我们发现，对于比较（而不是核心），基于较大编码器的系统更有可能依靠“正确”的信息，但即使他们在概括方面也很难，表明他们仍然学习特定的词汇模式，而不是比较的一般原则。
Two of the most fundamental challenges in Natural Language Understanding (NLU) at present are: (a) how to establish whether deep learning-based models score highly on NLU benchmarks for the 'right' reasons; and (b) to understand what those reasons would even be. We investigate the behavior of reading comprehension models with respect to two linguistic 'skills': coreference resolution and comparison. We propose a definition for the reasoning steps expected from a system that would be 'reading slowly', and compare that with the behavior of five models of the BERT family of various sizes, observed through saliency scores and counterfactual explanations. We find that for comparison (but not coreference) the systems based on larger encoders are more likely to rely on the 'right' information, but even they struggle with generalization, suggesting that they still learn specific lexical patterns rather than the general principles of comparison.
