论文标题
反馈梯度下降：有效且稳定的优化，与DNN的正交性具有正交性
Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs
论文作者
论文摘要
与正交性的优化在训练深神网络（DNN）中有用。为了对DNN施加正交性，计算效率和稳定性都很重要。但是，利用Riemannian优化或硬约束的现有方法只能确保稳定性，而使用软约束的方法只能提高效率。据我们所知，在本文中，我们提出了一种新的方法，称为“反馈梯度下降（FGD）”，这是同时显示出高效率和稳定性的第一项工作。 FGD基于在Stiefel歧管的切线束上的连续时间动力系统的简单但必不可少的Euler离散化诱导正交性。特别是，受称为反馈集成商的数值集成方法的启发，我们提议首次将其实例化在Stiefel歧管的切线束上。在广泛的图像分类实验中，FGD在准确性，效率和稳定性方面全面超过了现有的最新方法。
The optimization with orthogonality has been shown useful in training deep neural networks (DNNs). To impose orthogonality on DNNs, both computational efficiency and stability are important. However, existing methods utilizing Riemannian optimization or hard constraints can only ensure stability while those using soft constraints can only improve efficiency. In this paper, we propose a novel method, named Feedback Gradient Descent (FGD), to our knowledge, the first work showing high efficiency and stability simultaneously. FGD induces orthogonality based on the simple yet indispensable Euler discretization of a continuous-time dynamical system on the tangent bundle of the Stiefel manifold. In particular, inspired by a numerical integration method on manifolds called Feedback Integrators, we propose to instantiate it on the tangent bundle of the Stiefel manifold for the first time. In the extensive image classification experiments, FGD comprehensively outperforms the existing state-of-the-art methods in terms of accuracy, efficiency, and stability.
