论文标题
寻求信息质量检查中的挑战：无法回答的问题和段落检索
Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval
论文作者
论文摘要
最近预处理的语言模型“解决”了许多阅读理解基准，其中有问题的访问证据文档。但是，包含寻求信息查询的数据集在询问书写后提供证据文件仍然具有挑战性。我们分析了为什么回答信息寻求查询更具挑战性，以及它们在自然问题和TYDI QA中出现普遍的未解决方案的地方。我们的对照实验建议了两个净空 - 段落的选择和答案性预测，即配对证据文件是否包含查询的答案。当提供有金段并知道何时弃权回答时，现有模型很容易胜过人类注释者。但是，预测答案本身仍然具有挑战性。我们在六种语言上手动注释了800个无法回答的示例，这使它们具有挑战性。有了这些新数据，我们进行了每个类别的答复性预测，在当前数据集收集以及任务公式中揭示了问题。我们的研究共同指出了用于寻求信息的问题回答的未来研究的途径，无论是用于数据集创建还是模型开发。
Recent pretrained language models "solved" many reading comprehension benchmarks, where questions are written with access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA. Our controlled experiments suggest two headrooms -- paragraph selection and answerability prediction, i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, existing models easily outperform a human annotator. However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per-category answerability prediction, revealing issues in the current dataset collection as well as task formulation. Together, our study points to avenues for future research in information-seeking question answering, both for dataset creation and model development.
