论文标题
PADA：修剪辅助领域适应自我监督的语音表示
PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations
论文作者
论文摘要
虽然自我监督的语音表示学习（SSL）模型执行了各种下游任务，但已经观察到这些模型过于拟合未标记数据来源的域。为了减轻此问题，我们提出了PADA（修剪辅助域的适应性），并从大量室外（OOD）数据中预先训练的模型中零重量。直观地，这有助于为目标域ASR填充腾出空间。可以通过各种修剪策略来识别冗余权重，这些策略已作为本工作的一部分进行了详细讨论。具体而言，我们研究了最近发现的任务不合时宜的和任务感知的修剪对PADA的效果，并根据后者提出了一个新的修剪范式，我们称之为跨域任务意识到的修剪（CD-TAW）。 CD-TAW从良好的OOD模型中获得了初始修剪面膜，这使其与本文讨论的其余修剪策略完全不同。当在没有语言模型（LM）解码的2小时子集中进行微调时，我们提出的CD-TAW方法比基线相对相对改善了20.6％。此外，我们进行了详细的分析，以突出我们提出的方法的关键设计选择。
While self-supervised speech representation learning (SSL) models serve a variety of downstream tasks, these models have been observed to overfit to the domain from which the unlabelled data originates. To alleviate this issue, we propose PADA (Pruning Assisted Domain Adaptation) and zero out redundant weights from models pre-trained on large amounts of out-of-domain (OOD) data. Intuitively, this helps to make space for the target-domain ASR finetuning. The redundant weights can be identified through various pruning strategies which have been discussed in detail as a part of this work. Specifically, we investigate the effect of the recently discovered Task-Agnostic and Task-Aware pruning on PADA and propose a new pruning paradigm based on the latter, which we call Cross-Domain Task-Aware Pruning (CD-TAW). CD-TAW obtains the initial pruning mask from a well fine-tuned OOD model, which makes it starkly different from the rest of the pruning strategies discussed in the paper. Our proposed CD-TAW methodology achieves up to 20.6% relative WER improvement over our baseline when fine-tuned on a 2-hour subset of Switchboard data without language model (LM) decoding. Furthermore, we conduct a detailed analysis to highlight the key design choices of our proposed method.
