论文标题
关于基准数据集和模拟在方法比较研究中的作用
On the role of benchmarking data sets and simulations in method comparison studies
论文作者
论文摘要
方法比较对于为应用研究人员提供建议和指导至关重要，这些研究人员通常必须从多种可用的方法中进行选择。尽管文献中存在许多比较，但这些比较通常不是中立的，而是一种新颖的方法。除了选择设计和对发现的适当报告外，有关这种方法比较研究的基础数据还有不同的方法。大多数关于统计方法论的手稿都依赖于模拟研究，并提供了一个现实世界中的数据集，以激励和说明所研究方法的示例。相比之下，在监督学习的背景下，通常使用所谓的基准测试数据集评估方法，即作为社区中黄金标准的现实世界数据。另一方面，在这种情况下，模拟研究不太常见。本文的目的是调查这些方法之间的差异和相似性，讨论它们的优势和缺点，并最终开发出新的方法来评估挑选两全其美的方法。为此，我们借鉴了不同背景的想法，例如混合方法研究和临床方案评估。
Method comparisons are essential to provide recommendations and guidance for applied researchers, who often have to choose from a plethora of available approaches. While many comparisons exist in the literature, these are often not neutral but favour a novel method. Apart from the choice of design and a proper reporting of the findings, there are different approaches concerning the underlying data for such method comparison studies. Most manuscripts on statistical methodology rely on simulation studies and provide a single real-world data set as an example to motivate and illustrate the methodology investigated. In the context of supervised learning, in contrast, methods are often evaluated using so-called benchmarking data sets, i.e. real-world data that serve as gold standard in the community. Simulation studies, on the other hand, are much less common in this context. The aim of this paper is to investigate differences and similarities between these approaches, to discuss their advantages and disadvantages and ultimately to develop new approaches to the evaluation of methods picking the best of both worlds. To this aim, we borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.
