论文标题
高原现象通过神经网络学习中的数据依赖性---统计机械分析
Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysis
论文作者
论文摘要
高原现象在学习过程中停止损失价值降低，已经报告了各种研究人员。该现象在1990年代进行了积极检查，发现是由于神经网络模型的基本层次结构。然后，这种现象被认为是不可避免的。但是，这种现象很少发生在最近的深度学习的背景下。理论与现实之间存在差距。在本文中，使用统计机械制定，我们阐明了高原现象与所学数据的统计特性之间的关系。结果表明，协方差的数据较小且分散的特征值往往会使高原现象不显眼。
The plateau phenomenon, wherein the loss value stops decreasing during the process of learning, has been reported by various researchers. The phenomenon is actively inspected in the 1990s and found to be due to the fundamental hierarchical structure of neural network models. Then the phenomenon has been thought as inevitable. However, the phenomenon seldom occurs in the context of recent deep learning. There is a gap between theory and reality. In this paper, using statistical mechanical formulation, we clarified the relationship between the plateau phenomenon and the statistical property of the data learned. It is shown that the data whose covariance has small and dispersed eigenvalues tend to make the plateau phenomenon inconspicuous.
