论文标题
Crossre：用于关系提取的跨域数据集
CrossRE: A Cross-Domain Dataset for Relation Extraction
论文作者
论文摘要
关系提取（RE）吸引了越来越多的关注，但是当前的RE评估仅限于内域评估设置。关于挑战但现实的分布评估设置的重新系统票价的票据知之甚少。为了解决这一差距，我们提出了Crossre，这是RE的一种新的，可用的跨域基准，该基准包括六个不同的文本域，包括多标签注释。另一个创新是，我们发布了在注释期间收集的元数据，以包括困难实例的解释和旗帜。我们提供了一种经验评估，该评估具有用于关系分类的最先进模型。当元数据使我们能够对最先进的模型发明新的启示，我们对困难案例的影响并找到模型与人类注释之间的相关性进行了全面的分析。总体而言，我们的实证研究凸显了跨域RE的困难。我们发布数据集，以刺激这一方向的更多研究。
Relation Extraction (RE) has attracted increasing attention, but current RE evaluation is limited to in-domain evaluation setups. Little is known on how well a RE system fares in challenging, but realistic out-of-distribution evaluation setups. To address this gap, we propose CrossRE, a new, freely-available cross-domain benchmark for RE, which comprises six distinct text domains and includes multi-label annotations. An additional innovation is that we release meta-data collected during annotation, to include explanations and flags of difficult instances. We provide an empirical evaluation with a state-of-the-art model for relation classification. As the meta-data enables us to shed new light on the state-of-the-art model, we provide a comprehensive analysis on the impact of difficult cases and find correlations between model and human annotations. Overall, our empirical investigation highlights the difficulty of cross-domain RE. We release our dataset, to spur more research in this direction.
