论文标题
RSO：一种基于梯度的基于梯度的基于训练深神经网络的方法
RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks
论文作者
论文摘要
我们提出了RSO（随机搜索优化），这是一种基于训练深神经网络的基于梯度的Markov Chain Monte Carlo搜索方法。为此，RSO在深层神经网络中的重量增加了扰动，并测试它是否减少了迷你批次的损失。如果这减少了损失，则重量会更新，否则将保留现有的重量。令人惊讶的是，我们发现每次重复几次此过程足以训练深层神经网络。与使用SGD的反向传播相比，RSO的重量更新数量较小。 RSO可以在每个步骤中进行积极的体重更新，因为没有学习率的概念。单个图层的重量更新步骤也与损失的幅度相结合。 RSO对MNIST和CIFAR-10数据集的分类任务进行了评估，其深层神经网络为6至10层，其精度分别达到99.1％和81.8％。我们还发现，在更新权重仅5次之后，该算法在MNIST上获得了98％的分类精度。
We propose RSO (random search optimization), a gradient free Markov Chain Monte Carlo search based approach for training deep neural networks. To this end, RSO adds a perturbation to a weight in a deep neural network and tests if it reduces the loss on a mini-batch. If this reduces the loss, the weight is updated, otherwise the existing weight is retained. Surprisingly, we find that repeating this process a few times for each weight is sufficient to train a deep neural network. The number of weight updates for RSO is an order of magnitude lesser when compared to backpropagation with SGD. RSO can make aggressive weight updates in each step as there is no concept of learning rate. The weight update step for individual layers is also not coupled with the magnitude of the loss. RSO is evaluated on classification tasks on MNIST and CIFAR-10 datasets with deep neural networks of 6 to 10 layers where it achieves an accuracy of 99.1% and 81.8% respectively. We also find that after updating the weights just 5 times, the algorithm obtains a classification accuracy of 98% on MNIST.
