论文标题
FGSM对抗训练的技巧袋
Bag of Tricks for FGSM Adversarial Training
论文作者
论文摘要
通过快速梯度符号方法（FGSM）生成的样品（也称为FGSM-AT）生成的样品是一种计算上简单的方法来训练强大的网络。但是，在训练过程中，在ARXIV：2001.03994 [CS.LG]中确定了一种不稳定的“灾难性过度拟合”模式，在单个训练步骤中，可靠的精度突然降至零。现有方法使用梯度正规化器或随机初始化技巧来减轻此问题，而它们要么承担高计算成本或导致较低的鲁棒精度。在这项工作中，我们提供了第一项研究，该研究从三个角度彻底研究了技巧的集合：数据初始化，网络结构和优化，以克服FGSM-AT中的灾难性过度拟合。 令人惊讶的是，我们发现简单的技巧，即a）掩盖部分像素（即使没有随机性），b）设置较大的卷积步幅和平滑的激活功能，或c）正规化第一卷积层的重量，可以有效地解决过度拟合问题。对一系列网络体系结构的广泛结果验证了每个提出的技巧的有效性，还研究了技巧的组合。例如，在CIFAR-10上接受了PREACTRESNET-18培训，我们的方法对PGD-50攻击者的准确性为49.8％，针对AutoAttack的攻击者的精度为46.4％，这表明Pure FGSM-AT能够启用鲁棒的学习者。代码和模型可在https://github.com/ucsc-vlaa/bag-of-tricks-for-for-fgsm-at上公开获得。
Adversarial training (AT) with samples generated by Fast Gradient Sign Method (FGSM), also known as FGSM-AT, is a computationally simple method to train robust networks. However, during its training procedure, an unstable mode of "catastrophic overfitting" has been identified in arXiv:2001.03994 [cs.LG], where the robust accuracy abruptly drops to zero within a single training step. Existing methods use gradient regularizers or random initialization tricks to attenuate this issue, whereas they either take high computational cost or lead to lower robust accuracy. In this work, we provide the first study, which thoroughly examines a collection of tricks from three perspectives: Data Initialization, Network Structure, and Optimization, to overcome the catastrophic overfitting in FGSM-AT. Surprisingly, we find that simple tricks, i.e., a) masking partial pixels (even without randomness), b) setting a large convolution stride and smooth activation functions, or c) regularizing the weights of the first convolutional layer, can effectively tackle the overfitting issue. Extensive results on a range of network architectures validate the effectiveness of each proposed trick, and the combinations of tricks are also investigated. For example, trained with PreActResNet-18 on CIFAR-10, our method attains 49.8% accuracy against PGD-50 attacker and 46.4% accuracy against AutoAttack, demonstrating that pure FGSM-AT is capable of enabling robust learners. The code and models are publicly available at https://github.com/UCSC-VLAA/Bag-of-Tricks-for-FGSM-AT.
