论文标题
具有特殊令牌适应的参数有效调整
Parameter-Efficient Tuning with Special Token Adaptation
论文作者
论文摘要
参数有效的调整旨在仅在调整预审计模型以下游任务时仅更新一小部分参数。在这项工作中，我们介绍了意大利面，其中我们仅在基于变压器的模型中每个层的自我发项模块之前修改了特殊令牌表示（例如[sep]和[cls]）。意大利面可在自然语言中的全面填充方面达到可比的性能，理解包括文本分类的任务和NER，只有多达培训的总参数的0.029％。我们的工作不仅提供了一种简单但有效的参数调整方式，在为多个任务部署固定模型时，它具有广泛的实用应用，而且还展示了特殊令牌在预审前的语言模型中的关键作用。
Parameter-efficient tuning aims at updating only a small subset of parameters when adapting a pretrained model to downstream tasks. In this work, we introduce PASTA, in which we only modify the special token representations (e.g., [SEP] and [CLS] in BERT) before the self-attention module at each layer in Transformer-based models. PASTA achieves comparable performance to full finetuning in natural language understanding tasks including text classification and NER with up to only 0.029% of total parameters trained. Our work not only provides a simple yet effective way of parameter-efficient tuning, which has a wide range of practical applications when deploying finetuned models for multiple tasks, but also demonstrates the pivotal role of special tokens in pretrained language models
