论文标题
通过私人联邦学习培训大型唱歌的神经语言模型
Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices
论文作者
论文摘要
联合学习（FL）是一种使用跨设备分布的数据训练模型的技术。差异隐私（DP）为敏感数据提供了正式的隐私保证。我们的目标是在使用FL和DP保护隐私的同时，在计算受限的设备上训练大型神经网络语言模型（NNLM）。但是，随着模型大小的增长，引入模型的DP噪声通常会增加，从而阻止收敛。我们提出了部分嵌入更新（PEU），这是一种新颖的技术，可通过降低有效载荷大小来降低噪声。此外，我们采用低级适应（LORA）和噪声对比估计（NCE）来减少计算受限设备上大型模型的记忆需求。这种技术的结合使得可以在保留准确性和隐私的同时训练大型胶卷语言模型。
Federated Learning (FL) is a technique to train models using data distributed across devices. Differential Privacy (DP) provides a formal privacy guarantee for sensitive data. Our goal is to train a large neural network language model (NNLM) on compute-constrained devices while preserving privacy using FL and DP. However, the DP-noise introduced to the model increases as the model size grows, which often prevents convergence. We propose Partial Embedding Updates (PEU), a novel technique to decrease noise by decreasing payload size. Furthermore, we adopt Low Rank Adaptation (LoRA) and Noise Contrastive Estimation (NCE) to reduce the memory demands of large models on compute-constrained devices. This combination of techniques makes it possible to train large-vocabulary language models while preserving accuracy and privacy.
