论文标题
询问不讲：在上下文表示中探索潜在本体论
Asking without Telling: Exploring Latent Ontologies in Contextual Representations
论文作者
论文摘要
经过验证的上下文编码器（例如Elmo和Bert）的成功引起了这些模型所学的知识：如果没有明确的监督，他们是否会学会编码有意义的语言结构观念？如果是这样，该结构如何编码？为了进行研究，我们介绍了潜在子类学习（LSL）：对基于分类器的探测方法的修改，该方法诱导了探针输入的潜在分类（或本体学）。无需访问细颗粒的金标签，LSL提取了以可解释和可量化形式的输入表示形式的新兴结构。在实验中，我们发现了熟悉类别的有力证据，例如Elmo中的人格概念以及新的本体论区别，例如偏爱对核心论证上的细粒语义角色。我们的结果为预审慎的编码器中新兴结构提供了独特的新证据，包括与早期方法无法接近的现有注释的偏离。
The success of pretrained contextual encoders, such as ELMo and BERT, has brought a great deal of interest in what these models learn: do they, without explicit supervision, learn to encode meaningful notions of linguistic structure? If so, how is this structure encoded? To investigate this, we introduce latent subclass learning (LSL): a modification to existing classifier-based probing methods that induces a latent categorization (or ontology) of the probe's inputs. Without access to fine-grained gold labels, LSL extracts emergent structure from input representations in an interpretable and quantifiable form. In experiments, we find strong evidence of familiar categories, such as a notion of personhood in ELMo, as well as novel ontological distinctions, such as a preference for fine-grained semantic roles on core arguments. Our results provide unique new evidence of emergent structure in pretrained encoders, including departures from existing annotations which are inaccessible to earlier methods.
