论文标题

Vera C. Rubin天文台数据管家和管道执行系统

The Vera C. Rubin Observatory Data Butler and Pipeline Execution System

论文作者

Jenness, Tim, Bosch, James F., Lust, Nate B., Pease, Nathan M., Gower, Michelle, Kowalik, Mikolaj, Dubois-Felsmann, Gregory P., Mueller, Fritz, Schellart, Pim

论文摘要

鲁宾天文台的数据巴特勒旨在允许数据文件位置和文件格式从编写科学管道算法的人们中抽象出来。巴特勒与工作流程图构建器一起工作,允许从算法任务构建管道。这些管道可以使用对象存储和多节点群集在大规模执行,也可以使用本地文件系统在笔记本电脑上执行。现在,在鲁宾建设和早期操作期间,巴特勒和管道系统正在日常使用。

The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations.

扫码加入交流群

加入微信交流群

微信交流群二维码

扫码加入学术交流群,获取更多资源