论文标题
Secemplip：将语音与预训练的视觉和语言模型相结合
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
论文作者
论文摘要
数据驱动的语音处理模型通常在大量文本监督下表现良好，但是收集抄录的语音数据成本高昂。因此，我们提出了SpeechClip，这是一个新颖的框架，通过图像桥接语音和文本，以增强语音模型而无需转录。我们利用最先进的预训练的休伯特和剪辑，通过配对的图像和口语标题对齐，并以最小的微调对齐。 SpeechClip在图像语音检索上的先验优于先前的最新时间，并且在没有转录的直接监督的情况下执行零击的语音文本检索。此外，SecemClip可以直接从语音中检索语义上相关的关键字。
Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.
