论文标题
通过神经切线转移找到可训练的稀疏网络
Finding trainable sparse networks through Neural Tangent Transfer
论文作者
论文摘要
深度神经网络已经急剧改变了机器学习，但是它们的记忆和能量需求是巨大的。相比之下，实际生物神经网络的要求相当谦虚，这可能是这种紧缩政策的一个功能是它们的稀疏连接性。在深度学习中，在特定任务上表现良好的可训练的稀疏网络通常使用依赖标签的修剪标准构建。在本文中，我们介绍了神经切线转移，这种方法以无标签的方式找到了可训练的稀疏网络。具体而言，我们发现稀疏网络的训练动力学以神经切线内核为特征，模仿了功能空间中密集网络的训练动力学。最后，我们在几个标准分类任务上评估了我们的标签 - 敏捷方法，并表明所得的稀疏网络在更快地收敛的同时达到了更高的分类性能。
Deep neural networks have dramatically transformed machine learning, but their memory and energy demands are substantial. The requirements of real biological neural networks are rather modest in comparison, and one feature that might underlie this austerity is their sparse connectivity. In deep learning, trainable sparse networks that perform well on a specific task are usually constructed using label-dependent pruning criteria. In this article, we introduce Neural Tangent Transfer, a method that instead finds trainable sparse networks in a label-free manner. Specifically, we find sparse networks whose training dynamics, as characterized by the neural tangent kernel, mimic those of dense networks in function space. Finally, we evaluate our label-agnostic approach on several standard classification tasks and show that the resulting sparse networks achieve higher classification performance while converging faster.
