论文标题
带随机二进制重量的二进制自动编码器
Binary autoencoder with random binary weights
论文作者
论文摘要
在此介绍了具有二进制激活的自动编码器的分析$ \ {0，1 \} $和二进制$ \ {0，1 \} $随机权重。这种设置将该模型放在不同领域的交集：神经科学，信息理论，稀疏编码和机器学习。结果表明，隐藏层的稀疏激活自然出现，以保留层之间的信息。此外，只要通过改变神经元的阈值，就可以在任何输入中获得零重建误差。该模型保留了在隐藏层的隐藏层中输入的相似性，这对于密集的隐藏层激活而言是最大的。通过分析层之间的相互信息，表明稀疏和密集表示之间的差异与内存损失的权衡有关。该模型类似于果蝇的嗅觉感知系统，所提出的理论结果为理解更复杂的神经网络提供了有用的见解。
Here is presented an analysis of an autoencoder with binary activations $\{0, 1\}$ and binary $\{0, 1\}$ random weights. Such set up puts this model at the intersection of different fields: neuroscience, information theory, sparse coding, and machine learning. It is shown that the sparse activation of the hidden layer arises naturally in order to preserve information between layers. Furthermore, with a large enough hidden layer, it is possible to get zero reconstruction error for any input just by varying the thresholds of neurons. The model preserves the similarity of inputs at the hidden layer that is maximal for the dense hidden layer activation. By analyzing the mutual information between layers it is shown that the difference between sparse and dense representations is related to a memory-computation trade-off. The model is similar to an olfactory perception system of a fruit fly, and the presented theoretical results give useful insights toward understanding more complex neural networks.
