论文标题
元学习差异：准备大型语言模型以进行有效的适应
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation
论文作者
论文摘要
大型审慎的语言模型（PLM）通常是通过微调或提示来适应域或任务的。填充需要修改所有参数，并具有足够的数据以避免过度拟合，而提示不需要培训，很少有示例，而是限制性能。取而代之的是，我们通过学习学习一般和适应PLM之间的差异来准备数据和参数有效适应。通过我们提出的动态低级别重新聚合和学习的体系结构控制器，通过模型权重和子层结构来表达这种差异。关于对话完成的实验，低资源的抽象摘要以及多域语言建模显示了通过域名自适应预处理进行自适应时间和性能的改善。消融表明我们的任务自适应重新聚体化（TARP）和模型搜索（TAMS）组件分别改进了其他参数效率转移（如适配器和结构学习方法），例如学识渊博的稀疏方法。
Large pretrained language models (PLMs) are often domain- or task-adapted via fine-tuning or prompting. Finetuning requires modifying all of the parameters and having enough data to avoid overfitting while prompting requires no training and few examples but limits performance. Instead, we prepare PLMs for data- and parameter-efficient adaptation by learning to learn the difference between general and adapted PLMs. This difference is expressed in terms of model weights and sublayer structure through our proposed dynamic low-rank reparameterization and learned architecture controller. Experiments on few-shot dialogue completion, low-resource abstractive summarization, and multi-domain language modeling show improvements in adaptation time and performance over direct finetuning or preparation via domain-adaptive pretraining. Ablations show our task-adaptive reparameterization (TARP) and model search (TAMS) components individually improve on other parameter-efficient transfer like adapters and structure-learning methods like learned sparsification.
