论文标题
随机搜索超参数调整：预期改进估计和相应的下限
Random Search Hyper-Parameter Tuning: Expected Improvement Estimation and the Corresponding Lower Bound
论文作者
论文摘要
高参数调整是改善神经网络性能的常见技术。大多数用于超参数搜索的技术都涉及一个迭代过程，该过程在每次迭代中都会重新训练模型。但是，每次其他搜索迭代的预期准确性提高，仍然未知。计算预期的改进可以帮助创建停止规则，以进行高参数调整，并可以更明智地分配项目的计算预算。在本文中，我们通过额外的超参数搜索迭代提高了预期准确性提高的经验估计。我们的结果适用于基于随机搜索\ cite {bergstra2012random}的任何超参数调整方法，并从固定分布中采样超参数。我们以$ o \ left（\ sqrt {\ frac {\ log k} {k}}} {k}} \ right）$ o \ left的错误限制了我们的估计。其中$ k $是当前的迭代次数。据我们所知，这是从额外的超参数搜索迭代中获得预期增益的第一个束缚。最后，我们证明了预期准确性的最佳估计仍然将具有$ \ frac {1} {k} $的错误。
Hyperparameter tuning is a common technique for improving the performance of neural networks. Most techniques for hyperparameter search involve an iterated process where the model is retrained at every iteration. However, the expected accuracy improvement from every additional search iteration, is still unknown. Calculating the expected improvement can help create stopping rules for hyperparameter tuning and allow for a wiser allocation of a project's computational budget. In this paper, we establish an empirical estimate for the expected accuracy improvement from an additional iteration of hyperparameter search. Our results hold for any hyperparameter tuning method which is based on random search \cite{bergstra2012random} and samples hyperparameters from a fixed distribution. We bound our estimate with an error of $O\left(\sqrt{\frac{\log k}{k}}\right)$ w.h.p. where $k$ is the current number of iterations. To the best of our knowledge this is the first bound on the expected gain from an additional iteration of hyperparameter search. Finally, we demonstrate that the optimal estimate for the expected accuracy will still have an error of $\frac{1}{k}$.
