论文标题
公共神经网络中的插值，外推和局部概括
Interpolation, extrapolation, and local generalization in common neural networks
论文作者
论文摘要
有悠久的作品历史表明，神经网络在训练场上很难推断。 Balesteriero等人最近的一项研究。 （2021）挑战以下观点：将插值定义为训练集凸壳的状态，他们表明，由于数据的高维度，该测试集在输入或神经空间中都不能在此凸船上大部分躺在该凸船上，从而唤起了众所周知的维度诅咒。然后假定神经网络必须在外推性模式下起作用。我们在这里研究典型神经网络最后一层隐藏层的神经活动。使用自动编码器来揭示神经活动的固有空间，我们表明该空间实际上是低维的，并且模型越好，该内在空间的维度越低。在这个空间中，测试集的大多数样本实际上位于训练集的凸面上：在凸船体的定义下，模型因此在插值方面起作用。此外，我们表明属于凸船体似乎不是相关标准。实际上，与训练集的邻近度的不同度量与性能准确性更好。因此，典型的神经网络似乎确实在插值方面起作用。良好的概括性能与神经网络在这种制度中运作良好的能力有关。
There has been a long history of works showing that neural networks have hard time extrapolating beyond the training set. A recent study by Balestriero et al. (2021) challenges this view: defining interpolation as the state of belonging to the convex hull of the training set, they show that the test set, either in input or neural space, cannot lie for the most part in this convex hull, due to the high dimensionality of the data, invoking the well known curse of dimensionality. Neural networks are then assumed to necessarily work in extrapolative mode. We here study the neural activities of the last hidden layer of typical neural networks. Using an autoencoder to uncover the intrinsic space underlying the neural activities, we show that this space is actually low-dimensional, and that the better the model, the lower the dimensionality of this intrinsic space. In this space, most samples of the test set actually lie in the convex hull of the training set: under the convex hull definition, the models thus happen to work in interpolation regime. Moreover, we show that belonging to the convex hull does not seem to be the relevant criteria. Different measures of proximity to the training set are actually better related to performance accuracy. Thus, typical neural networks do seem to operate in interpolation regime. Good generalization performances are linked to the ability of a neural network to operate well in such a regime.
