论文标题
每个域的背后都有一个转变：适应扭曲感知的视觉变压器以进行全景语义分段
Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation
论文作者
论文摘要
在本文中，我们解决了全景语义分割，该分段由于两个关键的挑战而探索了：（1）全景上的图像扭曲和对象变形； （2）在360°图像中缺乏语义注释。 To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels).其次，我们通过伪标签的整流化增强了相互原型适应性（MPA）策略，以进行无监督的域自适应全景分割。第三，除了针孔到型 - 帕尼他的（PIN2PAN）适应外，我们还创建了一个具有9,080个全景图像的新数据集（Synpass），在360°图像中促进合成对真实（Syn2real）适应方案。进行了广泛的实验，这些实验涵盖室内和室外场景，并且使用PIN2PAN和SYN2REAL方案进行了研究。 Trans4Pass+在四个域自适应的全景语义分段基准上实现了最先进的性能。代码可从https://github.com/jamycheung/trans4pass获得。
In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360° imagery. To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels). Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (Pin2Pan) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (Syn2Real) adaptation scheme in 360° imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with Pin2Pan and Syn2Real regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.
