论文标题
改进了音频表图像同步的重复和跳跃的处理
Improved Handling of Repeats and Jumps in Audio-Sheet Image Synchronization
论文作者
论文摘要
本文研究了录制录制和原始乐谱图像的视频后自动生成钢琴得分的问题。尽管以前的作品专注于已清理和预处理数据的合成乐谱音乐，但我们专注于开发一个可以应对IMSLP的原始，未经处理的乐谱PDF的混乱的系统。我们研究了现有系统如何应对真正的扫描的乐谱，填充页面和无关的作品或动作以及由于跳跃和重复而引起的不连续性。我们发现，系统性能的重要瓶颈正在处理跳跃并正确重复。特别是，我们发现先前提出的跳跃DTW算法在跳跃位置未知的先验位置时并不能稳健。我们提出了一种称为层次dtw的新型对齐算法，即使跳跃位置尚不清楚，也可以处理跳跃和重复。它首先在每个表音乐线上的功能级别上进行对齐，然后在细分级别进行第二个对齐。通过在细分级别上操作，它可以编码有关特定跳跃可能性的知识。通过对IMSLP的未经处理的乐谱PDF进行了精心控制的实验，我们表明，在处理各种类型的跳跃时，Hierarachical DTW显着胜过DTW。
This paper studies the problem of automatically generating piano score following videos given an audio recording and raw sheet music images. Whereas previous works focus on synthetic sheet music where the data has been cleaned and preprocessed, we instead focus on developing a system that can cope with the messiness of raw, unprocessed sheet music PDFs from IMSLP. We investigate how well existing systems cope with real scanned sheet music, filler pages and unrelated pieces or movements, and discontinuities due to jumps and repeats. We find that a significant bottleneck in system performance is handling jumps and repeats correctly. In particular, we find that a previously proposed Jump DTW algorithm does not perform robustly when jump locations are unknown a priori. We propose a novel alignment algorithm called Hierarchical DTW that can handle jumps and repeats even when jump locations are not known. It first performs alignment at the feature level on each sheet music line, and then performs a second alignment at the segment level. By operating at the segment level, it is able to encode domain knowledge about how likely a particular jump is. Through carefully controlled experiments on unprocessed sheet music PDFs from IMSLP, we show that Hierarachical DTW significantly outperforms Jump DTW in handling various types of jumps.
