论文标题
高度可扩展云建模的原位数据分析
In-situ data analytics for highly scalable cloud modelling on Cray machines
论文作者
论文摘要
MONC是用于研究大气流，湍流和云微物理学的高度可扩展的建模工具。典型的模拟产生了大量的原始数据，然后必须分析这些数据才能进行科学研究。出于性能和可伸缩性原因，该分析和随后的磁盘撰写应在数据上进行原位，因为它是生成的，但是在进行分析时不希望暂停计算。 在本文中，我们介绍了MONC的分析方法，其中一个节点的核心在计算和数据分析之间共享。通过异步将其数据发送到分析核心，计算内核可以连续运行，而无需暂停数据编写或分析。我们描述了我们的IO服务器框架和分析工作流程，这是高度异步的，以及针对这种方法提出的挑战的解决方案以及某些常见配置选择的性能影响。这项工作的结果是一种高度可扩展的分析方法，我们在Cray XC30的最多32768个计算核心上说明，在启用MONC中的数据分析时，对运行时的性能影响很小，并且还研究了我们在KNL上的方法和适用性。
MONC is a highly scalable modelling tool for the investigation of atmospheric flows, turbulence and cloud microphysics. Typical simulations produce very large amounts of raw data which must then be analysed for scientific investigation. For performance and scalability reasons this analysis and subsequent writing to disk should be performed in-situ on the data as it is generated however one does not wish to pause the computation whilst analysis is carried out. In this paper we present the analytics approach of MONC, where cores of a node are shared between computation and data analytics. By asynchronously sending their data to an analytics core, the computational cores can run continuously without having to pause for data writing or analysis. We describe our IO server framework and analytics workflow, which is highly asynchronous, along with solutions to challenges that this approach raises and the performance implications of some common configuration choices. The result of this work is a highly scalable analytics approach and we illustrate on up to 32768 computational cores of a Cray XC30 that there is minimal performance impact on the runtime when enabling data analytics in MONC and also investigate the performance and suitability of our approach on the KNL.
