论文标题
正规化线性自动编码器恢复主组件，最终
Regularized linear autoencoders recover the principal components, eventually
论文作者
论文摘要
近年来，我们对学习输入输出与神经网的了解迅速改善，但是即使在简单的线性自动编码器（LAES）的情况下，对于基础表示的融合也知之甚少。我们表明，当经过适当的正则化训练时，Laes可以直接学习最佳表示形式 - 有序的，轴对准的主组件。我们分析了两个这样的正则化方案：非均匀$ \ ell_2 $正则化和嵌套掉落的确定性变体[Rippel等，ICML'2014]。尽管这两种正则化方案都融合到最佳表示形式，但我们表明，由于不良条件的恶化，这种收敛性很慢，而潜在维度会加剧。我们表明，学习最佳表示的效率低下并不是不可避免的 - 我们对梯度下降更新进行了简单的修改，从而大大加快了融合的经验。
Our understanding of learning input-output relationships with neural nets has improved rapidly in recent years, but little is known about the convergence of the underlying representations, even in the simple case of linear autoencoders (LAEs). We show that when trained with proper regularization, LAEs can directly learn the optimal representation -- ordered, axis-aligned principal components. We analyze two such regularization schemes: non-uniform $\ell_2$ regularization and a deterministic variant of nested dropout [Rippel et al, ICML' 2014]. Though both regularization schemes converge to the optimal representation, we show that this convergence is slow due to ill-conditioning that worsens with increasing latent dimension. We show that the inefficiency of learning the optimal representation is not inevitable -- we present a simple modification to the gradient descent update that greatly speeds up convergence empirically.
