论文标题
计算高效的深度学习：算法趋势和机会
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities
论文作者
论文摘要
尽管近年来深度学习取得了长足的进步，但训练神经网络的经济和环境成本爆炸变得不可持续。为了解决这个问题，关于 *算法有效的深度学习 *已经进行了大量研究，该研究旨在降低培训成本，而不是在硬件或实施水平上，而是通过改变培训计划的语义的变化。在本文中，我们介绍了该领域研究的结构化和全面的概述。首先，我们将 *算法加速 *问题进行正式化，然后使用算法有效培训的基本构建块来开发分类法。我们的分类学突出了看似不同的方法的共同点，并揭示了当前的研究差距。接下来，我们提出评估最佳实践，以实现对加速技术的全面，公平和可靠的比较。为了进一步帮助研究和应用，我们讨论了培训管道中的常见瓶颈（通过实验说明），并为他们提供分类减轻策略。最后，我们重点介绍了一些未解决的研究挑战，并提出了有希望的未来方向。
Although deep learning has made great progress in recent years, the exploding economic and environmental costs of training neural networks are becoming unsustainable. To address this problem, there has been a great deal of research on *algorithmically-efficient deep learning*, which seeks to reduce training costs not at the hardware or implementation level, but through changes in the semantics of the training program. In this paper, we present a structured and comprehensive overview of the research in this field. First, we formalize the *algorithmic speedup* problem, then we use fundamental building blocks of algorithmically efficient training to develop a taxonomy. Our taxonomy highlights commonalities of seemingly disparate methods and reveals current research gaps. Next, we present evaluation best practices to enable comprehensive, fair, and reliable comparisons of speedup techniques. To further aid research and applications, we discuss common bottlenecks in the training pipeline (illustrated via experiments) and offer taxonomic mitigation strategies for them. Finally, we highlight some unsolved research challenges and present promising future directions.
