论文标题
零击的跨语言转移中的正交语言和任务适配器
Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer
论文作者
论文摘要
适配器模块，可以进行有效的经过预告片的变压器进行有效微调的其他可训练参数，最近已用于多语言变压器的语言专业化，改善了下游零击的跨语言传输。在这项工作中，我们提出了正交语言和任务适配器（称为正装）进行跨语性转移。他们经过培训，可以编码与已经存储在预告片的变压器参数中的知识相互互补的语言和特定于任务的信息。我们的零击跨语言转移实验，涉及三个任务（POS-TAGGIG，NER，NLI）和一组10种不同的语言，1）指向正载器在跨语言转移中的有用性，尤其是对于最复杂的NLI任务，但也针对最佳适配器配置，高度取决于任务和目标语言和目标语言。我们希望我们的工作能够激发对验证变压器对语言和特定于任务的微调的正交限制的有用性的更广泛研究。
Adapter modules, additional trainable parameters that enable efficient fine-tuning of pretrained transformers, have recently been used for language specialization of multilingual transformers, improving downstream zero-shot cross-lingual transfer. In this work, we propose orthogonal language and task adapters (dubbed orthoadapters) for cross-lingual transfer. They are trained to encode language- and task-specific information that is complementary (i.e., orthogonal) to the knowledge already stored in the pretrained transformer's parameters. Our zero-shot cross-lingual transfer experiments, involving three tasks (POS-tagging, NER, NLI) and a set of 10 diverse languages, 1) point to the usefulness of orthoadapters in cross-lingual transfer, especially for the most complex NLI task, but also 2) indicate that the optimal adapter configuration highly depends on the task and the target language. We hope that our work will motivate a wider investigation of usefulness of orthogonality constraints in language- and task-specific fine-tuning of pretrained transformers.
