论文标题
Cholesky因子的二阶随机梯度更新在Stein的引理中高斯变异近似
Second order stochastic gradient update for Cholesky factor in Gaussian variational approximation from Stein's Lemma
论文作者
论文摘要
在随机变化推断中，对多元高斯的使用重新质化技巧可引起协方差矩阵的平均值和chel液因子的有效更新，这取决于对数关节模型密度的一阶导数。在本文中，我们表明，Cholesky因子的替代无偏梯度估计值取决于对数关节模型密度的二阶导数，可以使用Stein的引理得出。这导致了Cholesky因子的二阶随机梯度更新，该更新能够改善收敛性，因为它的差异低于一阶更新（几乎可以忽略不计）。我们还得出了精度矩阵的cholesky因子的二阶更新，当精密矩阵具有稀疏的结构，反映了真实后验分布中有条件独立性时，这很有用。我们的结果也可用于获得Cholesky因子的二阶自然梯度更新，与基于欧几里得梯度的更新相比，这更强大。
In stochastic variational inference, use of the reparametrization trick for the multivariate Gaussian gives rise to efficient updates for the mean and Cholesky factor of the covariance matrix, which depend on the first order derivative of the log joint model density. In this article, we show that an alternative unbiased gradient estimate for the Cholesky factor which depends on the second order derivative of the log joint model density can be derived using Stein's Lemma. This leads to a second order stochastic gradient update for the Cholesky factor which is able to improve convergence, as it has variance lower than the first order update (almost negligible) when close to the mode. We also derive second order update for the Cholesky factor of the precision matrix, which is useful when the precision matrix has a sparse structure reflecting conditional independence in the true posterior distribution. Our results can be used to obtain second order natural gradient updates for the Cholesky factor as well, which are more robust compared to updates based on Euclidean gradients.
