论文标题
在虚构噪声模型中学习和推断
Learning and Inference in Imaginary Noise Models
论文作者
论文摘要
受经验贝叶斯学习平滑密度的最新发展的启发，我们研究了使用解码器的变量自动编码器，该解码器是针对随机变量$ y = x+n（0，σ^2 i_d）$量身定制的。在解码器的噪声模型隐式执行平滑的情况下，出现了平滑的变异推理的概念。 “隐式”，因为在训练过程中，编码器只能看到干净的样品。这是虚构噪声模型的概念，其中噪声模型决定了变分的下限$ \ Mathcal {l}（σ）$的功能形式，但是在学习过程中从未见过嘈杂的数据。该型号称为$σ$ -VAE。我们证明，通过简单的$β$ -VAE扩展：$ \ Mathcal {l}（σ_2）（σ_2）\ equiv \ Mathcal {l}（σ_1，β）$，其中$β= =σ_2=σ_2^2^2/σ_1^2 $。我们证明了指数家族的拉普拉斯分布的结果类似。从经验上，我们向学习模型报告了一个有趣的权力定律$ \ MATHCAL {d} _ {\ rm kl} \simσ^{ - ν{ - ν{ - ν} $，我们研究了$σ$ -VAE中的推断，以了解$σ$ -VAE。这些实验是在MNIST上进行的，我们表明，尽管在训练过程中没有看到任何东西，但该模型可以对极度嘈杂的样本进行合理的推论。在这个政权中，香草vae彻底崩溃了。我们在此处的发现上以假设（XYZ假设）的方式完成。
Inspired by recent developments in learning smoothed densities with empirical Bayes, we study variational autoencoders with a decoder that is tailored for the random variable $Y=X+N(0,σ^2 I_d)$. A notion of smoothed variational inference emerges where the smoothing is implicitly enforced by the noise model of the decoder; "implicit", since during training the encoder only sees clean samples. This is the concept of imaginary noise model, where the noise model dictates the functional form of the variational lower bound $\mathcal{L}(σ)$, but the noisy data are never seen during learning. The model is named $σ$-VAE. We prove that all $σ$-VAEs are equivalent to each other via a simple $β$-VAE expansion: $\mathcal{L}(σ_2) \equiv \mathcal{L}(σ_1,β)$, where $β=σ_2^2/σ_1^2$. We prove a similar result for the Laplace distribution in exponential families. Empirically, we report an intriguing power law $\mathcal{D}_{\rm KL} \sim σ^{-ν}$ for the learned models and we study the inference in the $σ$-VAE for unseen noisy data. The experiments were performed on MNIST, where we show that quite remarkably the model can make reasonable inferences on extremely noisy samples even though it has not seen any during training. The vanilla VAE completely breaks down in this regime. We finish with a hypothesis (the XYZ hypothesis) on the findings here.
