论文标题
参数有效的调整使得良好的分类头
Parameter-Efficient Tuning Makes a Good Classification Head
论文作者
论文摘要
近年来，审计的模型彻底改变了自然语言理解的范式（NLU），在该主链训练后，我们在该范围内附加了一个随机初始化的分类头，例如伯特和整个模型。由于预处理的骨干为改进做出了重大贡献，我们自然希望良好的分类头还可以使培训受益。但是，主链的最终输出，即分类头的输入，在填充过程中将发生很大变化，从而使通常的仅预处理（LP-ft）无效。在本文中，我们发现参数有效的调整使得一个良好的分类头，我们可以简单地替换随机初始化的头部以获得稳定的性能增益。我们的实验表明，与参数有效调整共同审议的分类头始终提高胶水和超粘合期9个任务的性能。
In recent years, pretrained models revolutionized the paradigm of natural language understanding (NLU), where we append a randomly initialized classification head after the pretrained backbone, e.g. BERT, and finetune the whole model. As the pretrained backbone makes a major contribution to the improvement, we naturally expect a good pretrained classification head can also benefit the training. However, the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning, making the usual head-only pretraining (LP-FT) ineffective. In this paper, we find that parameter-efficient tuning makes a good classification head, with which we can simply replace the randomly initialized heads for a stable performance gain. Our experiments demonstrate that the classification head jointly pretrained with parameter-efficient tuning consistently improves the performance on 9 tasks in GLUE and SuperGLUE.
