论文标题
审前的语言模型胚胎学：阿尔伯特的诞生
Pretrained Language Model Embryology: The Birth of ALBERT
论文作者
论文摘要
虽然已经彻底检查了经过审慎的语言模型（LMS）的行为，但很少研究训练期间发生的事情。因此，我们研究了从一组随机初始化的参数到全能语言模型的发展过程，我们将其称为验证语言模型的胚胎学。我们的结果表明，阿尔伯特学会了在训练过程中以不同的学习速度重建和预测语音不同部分（POS）的令牌。我们还发现，语言知识和世界知识通常不会随着训练的进展而没有改善，也没有下游任务的表现。这些发现表明，在预训练期间，对预算模型的知识会有所不同，并且具有更验证的步骤并不一定会提供具有更全面知识的模型。我们将提供源代码和验证模型，以在https://github.com/d223302/albert-embryology上复制我们的结果。
While behaviors of pretrained language models (LMs) have been thoroughly examined, what happened during pretraining is rarely studied. We thus investigate the developmental process from a set of randomly initialized parameters to a totipotent language model, which we refer to as the embryology of a pretrained language model. Our results show that ALBERT learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. We also find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks' performance. These findings suggest that knowledge of a pretrained model varies during pretraining, and having more pretrain steps does not necessarily provide a model with more comprehensive knowledge. We will provide source codes and pretrained models to reproduce our results at https://github.com/d223302/albert-embryology.
