论文标题
通过规模解锁高准确性差异化私有图像分类
Unlocking High-Accuracy Differentially Private Image Classification through Scale
论文作者
论文摘要
差异隐私（DP）提供了正式的隐私保证，以防止对手可以访问机器学习模型，从而提取有关单个培训点的信息。最受欢迎的DP训练方法是差异私有随机梯度下降（DP-SGD），它通过在训练过程中注入噪声来实现这种保护。然而，以前的工作发现，DP-SGD通常会导致标准图像分类基准的性能显着下降。此外，一些作者假设DP-SGD在大型模型上固有地表现差，因为保留隐私所需的噪声规范与模型维度成正比。相比之下，我们证明了过度参数化模型上的DP-SGD可以比以前想象的要好得多。将仔细的超参数调整与简单技术结合起来，以确保信号传播并提高收敛速率，我们获得了新的SOTA，而没有（8，10^{ - 5}）的CIFAR-10中的81.4％的额外数据（8，10^{-5}） - DP使用40层的宽液压NET，而不是先前的SOTA，从而提高了71.7％的先前SOTA。微调预先训练的NFNET-F3时，我们在Imavelet（0.5，8*10^{ - 7}）下实现了83.8％的TOP-1准确性。此外，我们还达到了（8，8 \ cdot 10^{ - 7}）-DP下的86.7％TOP-1精度，该任务的当前非私人SOTA仅比当前的非私人SOTA低4.3％。我们认为，我们的结果是缩小私人图像分类和非私有图像分类之间准确性差距的重要一步。
Differential Privacy (DP) provides a formal privacy guarantee preventing adversaries with access to a machine learning model from extracting information about individual training points. Differentially Private Stochastic Gradient Descent (DP-SGD), the most popular DP training method for deep learning, realizes this protection by injecting noise during training. However previous works have found that DP-SGD often leads to a significant degradation in performance on standard image classification benchmarks. Furthermore, some authors have postulated that DP-SGD inherently performs poorly on large models, since the norm of the noise required to preserve privacy is proportional to the model dimension. In contrast, we demonstrate that DP-SGD on over-parameterized models can perform significantly better than previously thought. Combining careful hyper-parameter tuning with simple techniques to ensure signal propagation and improve the convergence rate, we obtain a new SOTA without extra data on CIFAR-10 of 81.4% under (8, 10^{-5})-DP using a 40-layer Wide-ResNet, improving over the previous SOTA of 71.7%. When fine-tuning a pre-trained NFNet-F3, we achieve a remarkable 83.8% top-1 accuracy on ImageNet under (0.5, 8*10^{-7})-DP. Additionally, we also achieve 86.7% top-1 accuracy under (8, 8 \cdot 10^{-7})-DP, which is just 4.3% below the current non-private SOTA for this task. We believe our results are a significant step towards closing the accuracy gap between private and non-private image classification.
