论文标题
神经网络深度与目标功能的位置之间的相互作用
Interplay between depth of neural networks and locality of target functions
论文作者
论文摘要
人们已经认识到，在各种机器学习任务中，大量过度参数化的深神经网络（DNN）表现出令人惊讶的良好概括性能。尽管已经从不同的角度（例如近似理论和统计学习理论）进行了研究，但现有理论并不能充分解释过度参数化的DNN的经验成功。在这项工作中，我们报告了目标函数的深度和位置之间的显着相互作用。我们介绍了$ k $ - 本地和$ k $ - 全球功能，发现深度有益于学习本地功能，但对学习全球功能有害。这种相互作用未被神经切线内核捕获，该神经切线内核描述了懒惰学习方面的无限宽神经网络。
It has been recognized that heavily overparameterized deep neural networks (DNNs) exhibit surprisingly good generalization performance in various machine-learning tasks. Although benefits of depth have been investigated from different perspectives such as the approximation theory and the statistical learning theory, existing theories do not adequately explain the empirical success of overparameterized DNNs. In this work, we report a remarkable interplay between depth and locality of a target function. We introduce $k$-local and $k$-global functions, and find that depth is beneficial for learning local functions but detrimental to learning global functions. This interplay is not properly captured by the neural tangent kernel, which describes an infinitely wide neural network within the lazy learning regime.
