论文标题
随机梯度方法与非平滑非凸优化的动量的收敛
Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization
论文作者
论文摘要
具有动量的随机梯度方法在许多流行的机器学习库中广泛用于应用程序和优化子例程的核心。但是，除了凸或光滑的问题以外的问题以外的问题尚未获得它们的样本复杂性。本文确定了随机亚级别方法的收敛速率，其动量术语为多类的非平滑型，非凸和受限的优化问题。我们的关键创新是构建特殊的Lyapunov函数，可以在无需调整动量参数的情况下就可以实现验证的复杂性。对于平滑的问题，我们将已知的复杂性扩展到受约束的情况下，并证明如何在较弱的假设下分析与最先进的情况。数值结果证实了我们的理论发展。
Stochastic gradient methods with momentum are widely used in applications and at the core of optimization subroutines in many popular machine learning libraries. However, their sample complexities have not been obtained for problems beyond those that are convex or smooth. This paper establishes the convergence rate of a stochastic subgradient method with a momentum term of Polyak type for a broad class of non-smooth, non-convex, and constrained optimization problems. Our key innovation is the construction of a special Lyapunov function for which the proven complexity can be achieved without any tuning of the momentum parameter. For smooth problems, we extend the known complexity bound to the constrained case and demonstrate how the unconstrained case can be analyzed under weaker assumptions than the state-of-the-art. Numerical results confirm our theoretical developments.
