论文标题
联邦学习中的部分模型：绩效保证和福利
Partial Model Averaging in Federated Learning: Performance Guarantees and Benefits
论文作者
论文摘要
具有定期模型平均（FedAvg）的局部随机梯度下降（SGD）是联合学习中的基础算法。该算法独立于多名工人运行SGD，并定期在所有工人中平均该模型。但是，当本地SGD与许多工人一起运行时，定期平均会导致在工人中导致重大模型差异，从而使全球损失趋于缓慢地融合。尽管最近的高级优化方法解决了该问题的重点是非IID设置，但由于基本的定期模型平均，仍然存在模型差异问题。我们提出了一个平均框架的部分模型，以减轻联合学习中的模型差异问题。部分平均鼓励本地模型在参数空间上保持靠近，并使其能够更有效地最大程度地减少全球损失。鉴于固定数量的迭代和大量工人（128），部分平均实现的验证精度高达2.2％。
Local Stochastic Gradient Descent (SGD) with periodic model averaging (FedAvg) is a foundational algorithm in Federated Learning. The algorithm independently runs SGD on multiple workers and periodically averages the model across all the workers. When local SGD runs with many workers, however, the periodic averaging causes a significant model discrepancy across the workers making the global loss converge slowly. While recent advanced optimization methods tackle the issue focused on non-IID settings, there still exists the model discrepancy issue due to the underlying periodic model averaging. We propose a partial model averaging framework that mitigates the model discrepancy issue in Federated Learning. The partial averaging encourages the local models to stay close to each other on parameter space, and it enables to more effectively minimize the global loss. Given a fixed number of iterations and a large number of workers (128), the partial averaging achieves up to 2.2% higher validation accuracy than the periodic full averaging.
