论文标题
抽象的视觉推理与Tangram形状
Abstract Visual Reasoning with Tangram Shapes
论文作者
论文摘要
我们介绍了千克，这是一种研究人类和机器中抽象视觉推理的资源。利用坦格拉姆难题的历史作为认知科学中的刺激，我们构建了一个丰富的注释数据集，具有> 1k不同的刺激，比先前的资源更大，更多样化。它在视觉上和语言上都更丰富，超越了整个形状描述，包括分割图和部分标签。我们使用此资源来评估最近多模式模型的抽象视觉推理能力。我们观察到，预训练的权重表现出有限的抽象推理，这可以通过微调大大改善。我们还观察到，明确描述零件的人类和模型的抽象推理，尤其是在共同编码语言和视觉输入时。 Kilogram可在https://lil.nlp.cornell.edu/kilogram上找到。
We introduce KiloGram, a resource for studying abstract visual reasoning in humans and machines. Drawing on the history of tangram puzzles as stimuli in cognitive science, we build a richly annotated dataset that, with >1k distinct stimuli, is orders of magnitude larger and more diverse than prior resources. It is both visually and linguistically richer, moving beyond whole shape descriptions to include segmentation maps and part labels. We use this resource to evaluate the abstract visual reasoning capacities of recent multi-modal models. We observe that pre-trained weights demonstrate limited abstract reasoning, which dramatically improves with fine-tuning. We also observe that explicitly describing parts aids abstract reasoning for both humans and models, especially when jointly encoding the linguistic and visual inputs. KiloGram is available at https://lil.nlp.cornell.edu/kilogram .
