论文标题
神经ODES的随机性：一项实证研究
Stochasticity in Neural ODEs: An Empirical Study
论文作者
论文摘要
神经网络的随机正规化（例如辍学）是深度学习中的广泛技术，可以更好地泛化。尽管成功，但连续时间模型（例如神经普通微分方程（ODE））通常依赖于完全确定性的进料前进操作。这项工作提供了关于几种图像分类任务（CIFAR-10，CIFAR-100，Tinyimagenet）的随机正规化神经ode的经验研究。在随机微分方程（SDE）的形式上，我们证明了神经SDE能够胜过其确定性的对应物。此外，我们表明训练期间的数据增加可改善同一模型的确定性和随机版本的性能。但是，数据增强获得的改进完全消除了随机正规化的经验收益，从而改变了神经ODE和神经SDE的性能可忽略不计。
Stochastic regularization of neural networks (e.g. dropout) is a wide-spread technique in deep learning that allows for better generalization. Despite its success, continuous-time models, such as neural ordinary differential equation (ODE), usually rely on a completely deterministic feed-forward operation. This work provides an empirical study of stochastically regularized neural ODE on several image-classification tasks (CIFAR-10, CIFAR-100, TinyImageNet). Building upon the formalism of stochastic differential equations (SDEs), we demonstrate that neural SDE is able to outperform its deterministic counterpart. Further, we show that data augmentation during the training improves the performance of both deterministic and stochastic versions of the same model. However, the improvements obtained by the data augmentation completely eliminate the empirical gains of the stochastic regularization, making the difference in the performance of neural ODE and neural SDE negligible.
