论文标题
使用聚集场
Low to High Dimensional Modality Hallucination using Aggregated Fields of View
论文作者
论文摘要
真实的机器人系统系统处理来自多种模式的数据，尤其是用于导航和识别等任务。由于传感器的故障或不良环境等因素，当一种或多种方式变得无法访问时，这些系统的性能会大大降低。在这里，我们认为模态幻觉是确保稳定方式可用性并减少不利后果的一种有效方法。虽然对模式的数据具有更丰富的信息，例如RGB至深度进行了广泛的研究，但我们研究了在机器人技术和自主系统中使用有趣的用例中更具挑战性的低到高模态幻觉。我们提出了一种新颖的幻觉体系结构，该体系结构从当地社区的多个视野中汇总了信息，以从现有方式中恢复丢失的信息。该过程是通过捕获数据模式之间的非线性映射来实现的，而学习的映射用于帮助现有模式以减轻涉及模态损失的不良情况下对系统的风险。我们还对UWRGBD和NYUD数据集进行了广泛的分类和分割实验，并证明了幻觉会减轻模态损失的负面影响。实施和模型：https：//github.com/kausic94/hallucination
Real-world robotics systems deal with data from a multitude of modalities, especially for tasks such as navigation and recognition. The performance of those systems can drastically degrade when one or more modalities become inaccessible, due to factors such as sensors' malfunctions or adverse environments. Here, we argue modality hallucination as one effective way to ensure consistent modality availability and thereby reduce unfavorable consequences. While hallucinating data from a modality with richer information, e.g., RGB to depth, has been researched extensively, we investigate the more challenging low-to-high modality hallucination with interesting use cases in robotics and autonomous systems. We present a novel hallucination architecture that aggregates information from multiple fields of view of the local neighborhood to recover the lost information from the extant modality. The process is implemented by capturing a non-linear mapping between the data modalities and the learned mapping is used to aid the extant modality to mitigate the risk posed to the system in the adverse scenarios which involve modality loss. We also conduct extensive classification and segmentation experiments on UWRGBD and NYUD datasets and demonstrate that hallucination allays the negative effects of the modality loss. Implementation and models: https://github.com/kausic94/Hallucination
