论文标题
研究激活弛豫算法的可伸缩性和生物学合理性
Investigating the Scalability and Biological Plausibility of the Activation Relaxation Algorithm
论文作者
论文摘要
最近提出的激活松弛（AR）算法提供了一种简单且可靠的方法，用于仅使用本地学习规则近似误差算法的反向传播。与竞争方案不同，它会收敛到确切的反向传播梯度，并且仅利用单一类型的计算单元和一个向后放松阶段。我们先前已经表明，通过（i）引入一组可学习的后置重量集，可以进一步简化算法，并使其在生物学上变得更加合理，从而克服了权重 - 传播问题，并且（ii）避免在每个神经元上计算非线性衍生物。但是，到目前为止，这些简化的功效仅在简单的多层 - 佩心（MLP）网络上进行了测试。在这里，我们表明这些简化仍然使用更复杂的CNN体系结构和具有挑战性的数据集维护性能，事实证明，这些数据集很难扩展到其他生物学上的方案。我们还调查了原始AR算法的另一种生物学上令人难以置信的假设 - 冷冻馈电通途 - 而无需损害性能而放松。
The recently proposed Activation Relaxation (AR) algorithm provides a simple and robust approach for approximating the backpropagation of error algorithm using only local learning rules. Unlike competing schemes, it converges to the exact backpropagation gradients, and utilises only a single type of computational unit and a single backwards relaxation phase. We have previously shown that the algorithm can be further simplified and made more biologically plausible by (i) introducing a learnable set of backwards weights, which overcomes the weight-transport problem, and (ii) avoiding the computation of nonlinear derivatives at each neuron. However, tthe efficacy of these simplifications has, so far, only been tested on simple multi-layer-perceptron (MLP) networks. Here, we show that these simplifications still maintain performance using more complex CNN architectures and challenging datasets, which have proven difficult for other biologically-plausible schemes to scale to. We also investigate whether another biologically implausible assumption of the original AR algorithm -- the frozen feedforward pass -- can be relaxed without damaging performance.
