论文标题
通过对抗模型扰动使神经网络正规化
Regularizing Neural Networks via Adversarial Model Perturbation
论文作者
论文摘要
有效的正规化技术在深度学习方面非常需要减轻过度拟合和改善的概括。这项工作提出了一种新的正规化计划，基于以下理解：经验风险的平坦局部最小值会导致模型更好地概括。该方案被称为对抗模型扰动（AMP），在这种情况下，它不是直接最大程度地降低经验风险，而是通过SGD最小化了另一种“ AMP损失”。具体而言，通过在参数空间中的每个点上应用“最坏”规范的扰动，从经验风险中获得了AMP损失。与大多数现有的正规化方案相比，AMP具有强大的理论理由，因此可以在理论上显示最小化AMP损失，以偏爱经验风险的平坦局部最小值。关于各种现代深层体系结构的广泛实验，将放大器作为正规化计划中的新技术建立。我们的代码可从https://github.com/hiyouga/amp-regularizer获得。
Effective regularization techniques are highly desired in deep learning for alleviating overfitting and improving generalization. This work proposes a new regularization scheme, based on the understanding that the flat local minima of the empirical risk cause the model to generalize better. This scheme is referred to as adversarial model perturbation (AMP), where instead of directly minimizing the empirical risk, an alternative "AMP loss" is minimized via SGD. Specifically, the AMP loss is obtained from the empirical risk by applying the "worst" norm-bounded perturbation on each point in the parameter space. Comparing with most existing regularization schemes, AMP has strong theoretical justifications, in that minimizing the AMP loss can be shown theoretically to favour flat local minima of the empirical risk. Extensive experiments on various modern deep architectures establish AMP as a new state of the art among regularization schemes. Our code is available at https://github.com/hiyouga/AMP-Regularizer.
