论文标题
使用无线电信号的家庭日常生活字幕
In-Home Daily-Life Captioning Using Radio Signals
论文作者
论文摘要
本文的目的是为日常生活标题 - 即，对人们的活动以及与房屋中对象进行互动的文本描述。解决此问题需要除传统视频字幕之外的新方法，因为大多数人会担心在整个家中部署相机。我们介绍了RF-Diare，这是一种新的模型，用于通过使用房屋的Floormap分析房屋中隐私的无线电信号来为日常生活进行字幕。 RF-Diary can further observe and caption people's life through walls and occlusions and in dark settings.在设计RF-Diare时，我们利用无线电信号捕获人们的3D动力学的能力，并使用FloorMap来帮助模型学习人们与对象的互动。我们还使用多模式功能对齐训练方案，该方案利用现有的基于视频的字幕数据集来提高基于无线电的字幕模型的性能。 Extensive experimental results demonstrate that RF-Diary generates accurate captions under visible conditions.它还在黑暗或阻塞的设置中维持其良好的性能，基于视频的字幕方法无法生成有意义的字幕。 For more information, please visit our project webpage: http://rf-diary.csail.mit.edu
This paper aims to caption daily life --i.e., to create a textual description of people's activities and interactions with objects in their homes. Addressing this problem requires novel methods beyond traditional video captioning, as most people would have privacy concerns about deploying cameras throughout their homes. We introduce RF-Diary, a new model for captioning daily life by analyzing the privacy-preserving radio signal in the home with the home's floormap. RF-Diary can further observe and caption people's life through walls and occlusions and in dark settings. In designing RF-Diary, we exploit the ability of radio signals to capture people's 3D dynamics, and use the floormap to help the model learn people's interactions with objects. We also use a multi-modal feature alignment training scheme that leverages existing video-based captioning datasets to improve the performance of our radio-based captioning model. Extensive experimental results demonstrate that RF-Diary generates accurate captions under visible conditions. It also sustains its good performance in dark or occluded settings, where video-based captioning approaches fail to generate meaningful captions. For more information, please visit our project webpage: http://rf-diary.csail.mit.edu
