论文标题
量身定制的多式模式学习，用于多标签情感识别
Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition
论文作者
论文摘要
多模式多标签情感识别（MMER）旨在从异质视觉，音频和文本方式中识别各种人类情绪。先前的方法主要集中于将多种方式投射到一个共同的潜在空间中，并为所有标签学习相同的表示，这忽略了每种模式的多样性，并且从不同的角度捕获了每个标签的更丰富的语义信息。此外，尚未完全利用相关的方式和标签的关系。在本文中，我们提出了多种模式学习的多式模式学习（量身定制），旨在完善多模式表示并增强每个标签的判别能力。具体而言，我们设计了一个对抗性多模式改进模块，以充分探索不同方式之间的共同点并增强每种模式的多样性。为了进一步利用标签模式依赖性，我们设计了一个类似BERT的跨模式编码器，以粒度下降方式逐渐融合私人和常见的模态表示，以及标签引导的解码器，以适应为每个标签的标签代表生成量身定制的表示，并使用标签语义的指导。此外，我们在对齐和非对齐设置的基准MMER数据集CMU-MOSEI上进行实验，这证明了量身定制者优于最先进的。代码可在https://github.com/kniter1/tailor上找到。
Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various human emotions from heterogeneous visual, audio and text modalities. Previous methods mainly focus on projecting multiple modalities into a common latent space and learning an identical representation for all labels, which neglects the diversity of each modality and fails to capture richer semantic information for each label from different perspectives. Besides, associated relationships of modalities and labels have not been fully exploited. In this paper, we propose versaTile multi-modAl learning for multI-labeL emOtion Recognition (TAILOR), aiming to refine multi-modal representations and enhance discriminative capacity of each label. Specifically, we design an adversarial multi-modal refinement module to sufficiently explore the commonality among different modalities and strengthen the diversity of each modality. To further exploit label-modal dependence, we devise a BERT-like cross-modal encoder to gradually fuse private and common modality representations in a granularity descent way, as well as a label-guided decoder to adaptively generate a tailored representation for each label with the guidance of label semantics. In addition, we conduct experiments on the benchmark MMER dataset CMU-MOSEI in both aligned and unaligned settings, which demonstrate the superiority of TAILOR over the state-of-the-arts. Code is available at https://github.com/kniter1/TAILOR.
