论文标题
通过Pac-Bayes的冷次寄生
Cold Posteriors through PAC-Bayes
论文作者
论文摘要
我们通过Pac-Bayes概括界的镜头研究了冷后效应。我们认为，在非反应环境中，当训练样品的数量相对较小时，应考虑到冷后效应的讨论，即近似贝叶斯推论并不能容易地提供对样本外数据的性能的保证。取而代之的是，通过泛化结合更好地描述了样本外误差。在这种情况下，我们探讨了各种推理与PAC-Bayes目标的ELBO目标之间的联系。我们注意到，虽然Elbo和Pac-Bayes目标相似，但后一个目标自然包含温度参数$λ$，不限于$λ= 1 $。对于回归和分类任务，在各向同性拉普拉斯与后部的近似值的情况下，我们展示了这种对温度参数的PAC-bayesian解释如何捕获冷后效应。
We investigate the cold posterior effect through the lens of PAC-Bayes generalization bounds. We argue that in the non-asymptotic setting, when the number of training samples is (relatively) small, discussions of the cold posterior effect should take into account that approximate Bayesian inference does not readily provide guarantees of performance on out-of-sample data. Instead, out-of-sample error is better described through a generalization bound. In this context, we explore the connections between the ELBO objective from variational inference and the PAC-Bayes objectives. We note that, while the ELBO and PAC-Bayes objectives are similar, the latter objectives naturally contain a temperature parameter $λ$ which is not restricted to be $λ=1$. For both regression and classification tasks, in the case of isotropic Laplace approximations to the posterior, we show how this PAC-Bayesian interpretation of the temperature parameter captures the cold posterior effect.
