论文标题
在语言融合的ASR中检测意想不到的记忆
Detecting Unintended Memorization in Language-Model-Fused ASR
论文作者
论文摘要
端到端（E2E）模型通常通过浅融合伴随语言模型（LMS），以提高其整体质量以及对稀有单词的认可。同时，几项先前的作品表明，LMS容易在训练数据中无意中记住稀有或独特的序列。在这项工作中，我们设计了一个框架，用于检测LM培训数据中随机文本序列的记忆（我们称为Canaries），当一个人只能访问LM融合语音识别器，而不是直接访问LM。在与变压器LM融合的生产级构象体RNN-T E2E模型上，我们表明可以从300m示例的LM训练数据中检测单一疾病的金丝雀的记忆。我们还激发了保护隐私的动机，我们还表明，通过示例梯度倾斜的LM培训而不会损害整体质量，这种记忆将大大降低。
End-to-end (E2E) models are often being accompanied by language models (LMs) via shallow fusion for boosting their overall quality as well as recognition of rare words. At the same time, several prior works show that LMs are susceptible to unintentionally memorizing rare or unique sequences in the training data. In this work, we design a framework for detecting memorization of random textual sequences (which we call canaries) in the LM training data when one has only black-box (query) access to LM-fused speech recognizer, as opposed to direct access to the LM. On a production-grade Conformer RNN-T E2E model fused with a Transformer LM, we show that detecting memorization of singly-occurring canaries from the LM training data of 300M examples is possible. Motivated to protect privacy, we also show that such memorization gets significantly reduced by per-example gradient-clipped LM training without compromising overall quality.
