论文标题
典型可学习任务空间的图片
A picture of the space of typical learnable tasks
论文作者
论文摘要
我们开发信息几何技术，以了解深层网络在使用受监管的，元，半监督和对比的学习中对不同任务进行培训时所学的表示。我们阐明了与任务空间结构相关的以下现象：（1）使用不同的表示学习方法在不同任务上训练的概率模型的歧管实际上是低维度的； （2）对一项任务的监督学习也会导致令人惊讶的进步，即使在看似不同的任务上也会取得令人惊讶的进步；如果培训任务具有不同的课程，则其他任务的进展更大。 （3）我们分析指示的任务空间的结构与WordNet系统发育树的一部分一致； （4）情节元学习算法和监督的学习在训练过程中遍历不同的轨迹，但它们最终适合相似的模型； （5）对比和半监督的学习方法，类似于监督学习的轨迹。我们使用从CIFAR-10和Imagenet数据集构建的分类任务来研究这些现象。
We develop information geometric techniques to understand the representations learned by deep networks when they are trained on different tasks using supervised, meta-, semi-supervised and contrastive learning. We shed light on the following phenomena that relate to the structure of the space of tasks: (1) the manifold of probabilistic models trained on different tasks using different representation learning methods is effectively low-dimensional; (2) supervised learning on one task results in a surprising amount of progress even on seemingly dissimilar tasks; progress on other tasks is larger if the training task has diverse classes; (3) the structure of the space of tasks indicated by our analysis is consistent with parts of the Wordnet phylogenetic tree; (4) episodic meta-learning algorithms and supervised learning traverse different trajectories during training but they fit similar models eventually; (5) contrastive and semi-supervised learning methods traverse trajectories similar to those of supervised learning. We use classification tasks constructed from the CIFAR-10 and Imagenet datasets to study these phenomena.
