论文标题
riemannian非凸优化的差异，并适应了批处理大小
Variance reduction for Riemannian non-convex optimization with batch size adaptation
论文作者
论文摘要
降低方差技术在加速梯度下降和随机梯度下降方面很受欢迎，以在欧几里得空间和riemannian歧管上定义的优化问题。在本文中，我们进一步改进了非凸riemannian优化的现有差异方法，包括具有批处理大小适应的R-SVRG和R-SRG/R-Spider。我们表明，该策略可以实现较低的总复杂性，以优化有限和在线设置下的一般非凸和梯度主导的功能。结果，我们还为R-SVRG提供了更简单的收敛分析，并在有限和设置下改善了R-SRG的复杂性界限。具体而言，我们证明R-SRG具有与R-Spider相同的近乎理想的复杂性，而无需较小的步长。对各种任务的经验实验证明了提出的自适应批量大小方案的有效性。
Variance reduction techniques are popular in accelerating gradient descent and stochastic gradient descent for optimization problems defined on both Euclidean space and Riemannian manifold. In this paper, we further improve on existing variance reduction methods for non-convex Riemannian optimization, including R-SVRG and R-SRG/R-SPIDER with batch size adaptation. We show that this strategy can achieve lower total complexities for optimizing both general non-convex and gradient dominated functions under both finite-sum and online settings. As a result, we also provide simpler convergence analysis for R-SVRG and improve complexity bounds for R-SRG under finite-sum setting. Specifically, we prove that R-SRG achieves the same near-optimal complexity as R-SPIDER without requiring a small step size. Empirical experiments on a variety of tasks demonstrate effectiveness of proposed adaptive batch size scheme.
