论文标题
通过结构化稀疏恢复在神经网络中优化高参数
Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
论文作者
论文摘要
在本文中，我们通过稀疏恢复方法的镜头研究了神经网络自动设计的两个重要问题 - 高参数优化（HPO）和神经结构搜索（NAS）。在本文的第一部分中，我们建立了HPO与结构化稀疏恢复之间的新联系。特别是，我们表明，超参数空间的特殊编码可以实现自然的组疗法恢复公式，当与超频带（一种多军匪徒策略）结合时，它会改善现有的超参数优化方法。图像数据集（例如CIFAR-10）的实验结果证实了我们方法的好处。在本文的第二部分中，我们建立了NAS与结构化稀疏恢复之间的联系。在NAS中的``单镜头''方法的基础上，我们提出了一种新颖的算法，我们通过将一声方法与学习低度稀疏稀疏布尔值多项式的技术合并为CONAS。我们提供有关验证误差测量数量的理论分析。最后，我们在几个数据集上验证了我们的方法，并发现迄今未报告的新颖体系结构，与现有的NAS方法相比，竞争（或更好）的竞争性（或更好）既可以产生性能和搜索时间。
In this paper, we study two important problems in the automated design of neural networks -- Hyper-parameter Optimization (HPO), and Neural Architecture Search (NAS) -- through the lens of sparse recovery methods. In the first part of this paper, we establish a novel connection between HPO and structured sparse recovery. In particular, we show that a special encoding of the hyperparameter space enables a natural group-sparse recovery formulation, which when coupled with HyperBand (a multi-armed bandit strategy), leads to improvement over existing hyperparameter optimization methods. Experimental results on image datasets such as CIFAR-10 confirm the benefits of our approach. In the second part of this paper, we establish a connection between NAS and structured sparse recovery. Building upon ``one-shot'' approaches in NAS, we propose a novel algorithm that we call CoNAS by merging ideas from one-shot approaches with a techniques for learning low-degree sparse Boolean polynomials. We provide theoretical analysis on the number of validation error measurements. Finally, we validate our approach on several datasets and discover novel architectures hitherto unreported, achieving competitive (or better) results in both performance and search time compared to the existing NAS approaches.
