论文标题
通过耳朵播放：通过视听模仿学习在阻塞中学习技能
Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning
论文作者
论文摘要
人类能够完成一系列具有挑战性的操纵任务，这些任务需要在视觉，触摸和声音等方式上共同推理。此外，许多这样的任务是部分观察的。例如，从背包中取出笔记本将导致视觉遮挡，并需要在音频或触觉信息的历史上进行推理。虽然在机器人上捕获强大的触觉感应可能是昂贵的，但机器人握把附近或麦克风是获取接触事件的音频反馈的一种便宜，简便的方法，在没有视力的情况下，这可能是令人惊讶的有价值的数据源。由于声音减轻视觉阻塞的可能性，我们旨在从视觉和音频输入中学习一组具有挑战性的部分观察到的操纵任务。我们提出的系统通过使用人类提供的干预措施将离线模仿学习与脱机模仿学习和在线填充相结合，从而学习了这些任务。在一组模拟任务中，我们发现我们的系统从使用音频中受益，并且通过使用在线干预措施，我们能够将离线模仿学习的成功率提高约20％。最后，我们发现我们的系统可以在Franka Emika Panda机器人上完成一系列具有挑战性的，部分观察到的任务，例如从袋子中提取钥匙，成功率70％，比不使用音频的政策高50％。
Humans are capable of completing a range of challenging manipulation tasks that require reasoning jointly over modalities such as vision, touch, and sound. Moreover, many such tasks are partially-observed; for example, taking a notebook out of a backpack will lead to visual occlusion and require reasoning over the history of audio or tactile information. While robust tactile sensing can be costly to capture on robots, microphones near or on a robot's gripper are a cheap and easy way to acquire audio feedback of contact events, which can be a surprisingly valuable data source for perception in the absence of vision. Motivated by the potential for sound to mitigate visual occlusion, we aim to learn a set of challenging partially-observed manipulation tasks from visual and audio inputs. Our proposed system learns these tasks by combining offline imitation learning from a modest number of tele-operated demonstrations and online finetuning using human provided interventions. In a set of simulated tasks, we find that our system benefits from using audio, and that by using online interventions we are able to improve the success rate of offline imitation learning by ~20%. Finally, we find that our system can complete a set of challenging, partially-observed tasks on a Franka Emika Panda robot, like extracting keys from a bag, with a 70% success rate, 50% higher than a policy that does not use audio.
