论文标题
微调预训练的语言模型，用于几次意图检测：有监督的预训练和各向同性化
Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization
论文作者
论文摘要
训练一个良好的意图分类器为一个只有少数注释的良好意图分类器培训良好的意图分类器。最近的研究表明，以有监督的方式，以少量标记的公共基准标记的话语进行微调的预训练的语言模型非常有帮助。但是，我们发现受监督的训练会产生各向异性特征空间，这可能会抑制语义表示的表达能力。受各向同性化的最新研究的启发，我们建议通过将各向同性的特征空间规范化，以改善监督的预训练。我们分别根据对比度学习和相关矩阵提出了两个正则化，并通过广泛的实验证明了它们的有效性。我们的主要发现是，有望通过各向同性化对监督的预训练进行正规化，以进一步提高少数射击意图检测的性能。可以在https://github.com/fanolabs/isointentbert-main上找到源代码。
It is challenging to train a good intent classifier for a task-oriented dialogue system with only a few annotations. Recent studies have shown that fine-tuning pre-trained language models with a small amount of labeled utterances from public benchmarks in a supervised manner is extremely helpful. However, we find that supervised pre-training yields an anisotropic feature space, which may suppress the expressive power of the semantic representations. Inspired by recent research in isotropization, we propose to improve supervised pre-training by regularizing the feature space towards isotropy. We propose two regularizers based on contrastive learning and correlation matrix respectively, and demonstrate their effectiveness through extensive experiments. Our main finding is that it is promising to regularize supervised pre-training with isotropization to further improve the performance of few-shot intent detection. The source code can be found at https://github.com/fanolabs/isoIntentBert-main.
