论文标题
灵活的数据集蒸馏：学习标签而不是图像
Flexible Dataset Distillation: Learn Labels Instead of Images
论文作者
论文摘要
我们研究数据集蒸馏的问题 - 创建一组能够训练良好模型的合成示例。特别是，我们研究了标签蒸馏的问题 - 为一小部分真实图像创建合成标签，并表明它比以前基于图像的数据集蒸馏方法更有效。从方法上讲，我们引入了一种更健壮和灵活的元学习算法，用于蒸馏，以及基于凸优化层的有效的一阶策略。使用我们的新算法蒸馏标签可改善基于图像的蒸馏的结果。更重要的是，它从与现成的优化器和各种神经体系结构的兼容性方面可以明确提高蒸馏数据集的灵活性。有趣的是，也可以在数据集中应用标签蒸馏，例如，仅通过对合成标记的英语字母进行培训来学习日本角色识别。
We study the problem of dataset distillation - creating a small set of synthetic examples capable of training a good model. In particular, we study the problem of label distillation - creating synthetic labels for a small set of real images, and show it to be more effective than the prior image-based approach to dataset distillation. Methodologically, we introduce a more robust and flexible meta-learning algorithm for distillation, as well as an effective first-order strategy based on convex optimization layers. Distilling labels with our new algorithm leads to improved results over prior image-based distillation. More importantly, it leads to clear improvements in flexibility of the distilled dataset in terms of compatibility with off-the-shelf optimizers and diverse neural architectures. Interestingly, label distillation can also be applied across datasets, for example enabling learning Japanese character recognition by training only on synthetically labeled English letters.
