论文标题
神经语言模型的语言分析
Linguistic Profiling of a Neural Language Model
论文作者
论文摘要
在本文中，我们调查了在微调过程之前和之后通过神经语言模型（NLM）学到的语言知识，以及这些知识如何在几个分类问题中影响其预测。我们使用一系列探测任务，每个任务都对应于从不同级别的语言注释中提取的独特句子级特征。我们表明，伯特能够编码广泛的语言特征，但是在接受特定的下游任务培训时，它往往会丢失此信息。我们还发现，伯特（Bert）编码不同类型的语言特性的能力对其预测具有积极影响：句子的可读性语言信息越多，它的能力就越高，它的能力越高，它的能力将预测分配给该句子的预期标签。
In this paper we investigate the linguistic knowledge learned by a Neural Language Model (NLM) before and after a fine-tuning process and how this knowledge affects its predictions during several classification problems. We use a wide set of probing tasks, each of which corresponds to a distinct sentence-level feature extracted from different levels of linguistic annotation. We show that BERT is able to encode a wide range of linguistic characteristics, but it tends to lose this information when trained on specific downstream tasks. We also find that BERT's capacity to encode different kind of linguistic properties has a positive influence on its predictions: the more it stores readable linguistic information of a sentence, the higher will be its capacity of predicting the expected label assigned to that sentence.
