论文标题
谁评论审稿人？多层陪审团问题
Who Reviews The Reviewers? A Multi-Level Jury Problem
论文作者
论文摘要
我们考虑使用一组独立审阅者（专家）的建议来确定二进制基础真相的问题，他们以某些独立的概率（能力）表达了对基础真理的猜测。在这种情况下，当所有审阅者都有能力（大于一半的能力）时，Condorcet陪审团告诉我们，添加更多的审阅者提高了整体准确性，如果所有能力已知，那么审阅者的最佳权重。但是，在实际环境中，审阅者可能是嘈杂的或无能的，即能力以下的能力，专家的数量可能很少，因此渐近的陪审员定理无关紧要。在这种情况下，我们探索任命一个或多个椅子（法官），他们确定每个审阅者的重量重量，从而创建多个级别。但是，这些椅子可能无法正确识别他们监督的审阅者的能力，因此无法计算最佳权重。当一组椅子能够优化审稿人并根据代理的能力分布，我们给出条件，从而给出有关何时更好地拥有更多椅子或更多审稿人的结果。通过数值模拟，我们表明在某些情况下最好拥有更多的椅子，但是在许多情况下，最好有更多的审阅者。
We consider the problem of determining a binary ground truth using advice from a group of independent reviewers (experts) who express their guess about a ground truth correctly with some independent probability (competence). In this setting, when all reviewers are competent (competence greater than one-half), the Condorcet Jury Theorem tells us that adding more reviewers increases the overall accuracy, and if all competences are known, then there exists an optimal weighting of the reviewers. However, in practical settings, reviewers may be noisy or incompetent, i.e., competence below half, and the number of experts may be small, so the asymptotic Condorcet Jury Theorem is not practically relevant. In such cases we explore appointing one or more chairs (judges) who determine the weight of each reviewer for aggregation, creating multiple levels. However, these chairs may be unable to correctly identify the competence of the reviewers they oversee, and therefore unable to compute the optimal weighting. We give conditions when a set of chairs is able to weight the reviewers optimally, and depending on the competence distribution of the agents, give results about when it is better to have more chairs or more reviewers. Through numerical simulations we show that in some cases it is better to have more chairs, but in many cases it is better to have more reviewers.
