论文标题
大规模构建图：联合会找到洗牌
Building Graphs at a Large Scale: Union Find Shuffle
论文作者
论文摘要
使用分布式计算框架的大规模图处理在行业中变得普遍有效。在这项工作中，我们提出了一种用于构建连接组件的高度可扩展性和可配置的分布式算法，称为Union Find shuffle（ufs）具有路径压缩。算法的规模和复杂性是最初分区数据的分区数量以及连接组件的大小的函数。我们讨论了与类似方法相比的复杂性和基准。我们还介绍了我们生产系统的当前基准，该基准在一年多以前部署的算法上运行的商品云基础设施上运行，扩展到约750亿个节点和600亿个链接（以及增长）。我们强调了我们的算法的关键方面，即使存在偏斜的数据，该算法也可以实现无缝缩放和性能，每个数据的大小为100亿个节点。
Large scale graph processing using distributed computing frameworks is becoming pervasive and efficient in the industry. In this work, we present a highly scalable and configurable distributed algorithm for building connected components, called Union Find Shuffle (UFS) with Path Compression. The scale and complexity of the algorithm are a function of the number of partitions into which the data is initially partitioned, and the size of the connected components. We discuss the complexity and the benchmarks compared to similar approaches. We also present current benchmarks of our production system, running on commodity out-of-the-box cloud Hadoop infrastructure, where the algorithm was deployed over a year ago, scaled to around 75 Billion nodes and 60 Billions linkages (and growing). We highlight the key aspects of our algorithm which enable seamless scaling and performance even in the presence of skewed data with large connected components in the size of 10 Billion nodes each.
