论文标题
加权汇总的随机梯度下降用于平行深度学习
Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning
论文作者
论文摘要
本文研究了随机优化问题，重点是为深度学习任务开发可扩展的并行算法。我们的解决方案涉及对神经网络模型中随机优化的目标函数的改革，以及一种新型的平行策略，即加权聚合随机梯度下降（WASGD）。经过对新目标函数特征的理论分析，WASGD根据本地工人的表现引入了分散的加权聚合方案。没有任何中心变量，新方法将自动评估当地工人的重要性，并根据他们的贡献接受。此外，通过（1）考虑设计的样品顺序和（2）应用更高级的权重评估功能，我们已经开发了该方法的增强版本WASGD+。为了验证新方法，我们将我们的方案基准针对几种流行的算法，包括最先进的技术（例如，弹性平均SGD）在培训深层神经网络以进行分类任务中。已经在四个经典数据集上进行了全面的实验，包括CIFAR-100，CIFAR-10，Fashion-Mnist和MNIST。随后的结果表明，WASGD方案在加速深度建筑训练方面具有优越性。更好的是，增强版本WASGD+已被证明是其基本版本的重大改进。
This paper investigates the stochastic optimization problem with a focus on developing scalable parallel algorithms for deep learning tasks. Our solution involves a reformation of the objective function for stochastic optimization in neural network models, along with a novel parallel strategy, coined weighted aggregating stochastic gradient descent (WASGD). Following a theoretical analysis on the characteristics of the new objective function, WASGD introduces a decentralized weighted aggregating scheme based on the performance of local workers. Without any center variable, the new method automatically assesses the importance of local workers and accepts them according to their contributions. Furthermore, we have developed an enhanced version of the method, WASGD+, by (1) considering a designed sample order and (2) applying a more advanced weight evaluating function. To validate the new method, we benchmark our schemes against several popular algorithms including the state-of-the-art techniques (e.g., elastic averaging SGD) in training deep neural networks for classification tasks. Comprehensive experiments have been conducted on four classic datasets, including the CIFAR-100, CIFAR-10, Fashion-MNIST, and MNIST. The subsequent results suggest the superiority of the WASGD scheme in accelerating the training of deep architecture. Better still, the enhanced version, WASGD+, has been shown to be a significant improvement over its basic version.
