论文标题
零级拓扑见解对迭代幅度修剪
Zeroth-Order Topological Insights into Iterative Magnitude Pruning
论文作者
论文摘要
现代的神经网络是著名的，但也高度多余和可压缩。深度学习文献中存在许多修剪策略，这些策略产生了超过90％的稀疏子网络，这些子网络训练有素，密集的架构，同时仍保持其原始精度。不过，在这些方法中，由于其概念上的简单性，易于实施和功效 - 迭代幅度修剪（IMP）在实践中占主导地位，并且实际上是在修剪社区中击败的基线。但是，关于为什么像IMP这样的简单方法完全有限的理论解释是很少而有限的。在这项工作中，我们利用持久同源性的概念来了解IMP的运作，并表明它本质地鼓励保留那些在受过训练的网络中保留拓扑信息的权重。随后，我们还提供了有关在完美保留其零订单拓扑特征的同时可以修剪多少不同网络的界限，并为IMP的修改版本提供了相同的操作。
Modern-day neural networks are famously large, yet also highly redundant and compressible; there exist numerous pruning strategies in the deep learning literature that yield over 90% sparser sub-networks of fully-trained, dense architectures while still maintaining their original accuracies. Amongst these many methods though -- thanks to its conceptual simplicity, ease of implementation, and efficacy -- Iterative Magnitude Pruning (IMP) dominates in practice and is the de facto baseline to beat in the pruning community. However, theoretical explanations as to why a simplistic method such as IMP works at all are few and limited. In this work, we leverage the notion of persistent homology to gain insights into the workings of IMP and show that it inherently encourages retention of those weights which preserve topological information in a trained network. Subsequently, we also provide bounds on how much different networks can be pruned while perfectly preserving their zeroth order topological features, and present a modified version of IMP to do the same.
