论文标题
从上下文化的单词表示中无监督的句法信息蒸馏
Unsupervised Distillation of Syntactic Information from Contextualized Word Representations
论文作者
论文摘要
表明在各种语义和句法任务上表现出色的单词表示形式，例如Elmo和Bert。在这项工作中，我们应对神经语言表示语义和结构之间无监督的分离的任务：我们旨在学习上下文化矢量的转换，从而丢弃了词汇语义，但可以保留结构信息。为此，我们会自动生成结构上相似但语义上不同的句子组，并使用度量学习方法学习一种强调向量中编码的结构组件的转换。我们证明了我们的转换簇媒介通过结构特性，而不是通过词汇语义。最后，我们通过证明它们在几次解析设置中胜过原始上下文表示的表现来证明我们的蒸馏表示的实用性。
Contextualized word representations, such as ELMo and BERT, were shown to perform well on various semantic and syntactic tasks. In this work, we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations: we aim to learn a transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information. To this end, we automatically generate groups of sentences which are structurally similar but semantically different, and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics. Finally, we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in a few-shot parsing setting.
