RETHINKING THE EFFECT OF DATA AUGMENTATION IN ADVERSARIAL CONTRASTIVE LEARNING

Abstract

Recent works have shown that self-supervised learning can achieve remarkable robustness when integrated with adversarial training (AT). However, the robustness gap between supervised AT (sup-AT) and self-supervised AT (self-AT) remains significant. Motivated by this observation, we revisit existing self-AT methods and discover an inherent dilemma that affects self-AT robustness: either strong or weak data augmentations are harmful to self-AT, and a medium strength is insufficient to bridge the gap. To resolve this dilemma, we propose a simple remedy named DYNACL (Dynamic Adversarial Contrastive Learning). In particular, we propose an augmentation schedule that gradually anneals from a strong augmentation to a weak one to benefit from both extreme cases. Besides, we adopt a fast post-processing stage for adapting it to downstream tasks. Through extensive experiments, we show that DYNACL can improve state-of-the-art self-AT robustness by 8.84% under Auto-Attack on the CIFAR-10 dataset, and can even outperform vanilla supervised adversarial training for the first time.

1. INTRODUCTION

Learning low-dimensional representations of inputs without supervision is one of the ultimate goals of machine learning. As a promising approach, self-supervised learning is rapidly closing the performance gap with respect to supervised learning (He et al., 2016; Chen et al., 2020b) in downstream tasks. However, for whatever supervised and self-supervised learning models, adversarial vulnerability remains a widely-concerned security issue, i.e., natural inputs injected by small and human imperceptible adversarial perturbations can fool the deep neural networks (DNNs) into making wrong predictions (Goodfellow et al., 2014) . In supervised learning, the most effective approach to enhance adversarial robustness is adversarial training (sup-AT) that learns DNNs with adversarial examples (Madry et al., 2017; Wang et al., 2019; Zhang et al., 2019; Wang et al., 2020; Wang & Wang, 2022) . However, sup-AT requires groundtruth labels to craft adversarial examples. In self-supervised learning, recent works including RoCL (Kim et al., 2020) , ACL (Jiang et al., 2020), and AdvCL (Fan et al., 2021) explored some adversarial training counterparts (self-AT). However, despite obtaining a certain degree of robustness, there is still a very large performance gap between sup-AT and self-AT methods. As shown in Figure 1 (a), sup-AT obtains 46.2% robust accuracy while state-of-the-art self-AT method only gets 37.6% on CIFAR-10, and the gap is > 8%. As a reference, in standard training (ST) using clean examples, the gap in classification accuracy between sup-ST and self-ST is much smaller (lower than 1% on CIFAR-10, see da Costa et al. (2022) ). This phenomenon leads to the following question: What is the key factor that prevents self-AT from obtaining comparable robustness to sup-AT? To answer this question, we need to examine the real difference between sup-AT and self-AT. As they share the same minimax training scheme, the difference mainly lies in the learning objective. 0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Different from sup-AT relying on labels, self-AT methods (RoCL, ACL, and AdvCL) mostly adopt contrastive learning objectives, which instead rely on matching representations under (strong) data augmentations to learn meaningful features. As noted by recent theoretical understandings (HaoChen et al., 2021; Wang et al., 2022) , the core mechanism of contrastive learning is that it leverages strong augmentations to create support overlap between intra-class samples, and as a result, the alignment between augmented positive samples could implicitly cluster intra-class samples together. Weak augmentations, instead, are not capable of generating enough overlap, and thus lead to severe performance degradation (Figure 1 (b) red line). In the meanwhile, we also find that strong data augmentations can be very harmful for adversarial robustness (Figure 1 (b) blue line). This reveals a critical dilemma of self-AT, that (default) strong augmentations, although useful for standard accuracy, also severely hurt adversarial robustness. To understand why, we draw some augmented samples and observe that self-AT augmentations induce significant semantic shifts to original images (Figure 1(c) ). This suggests that strong augmentations create a very large distribution shift between training and test data, and the local ℓ p robustness on training examples becomes less transferable to test data. Notably, some augmented samples from different classes can even become inseparable (the bird and the cat samples in red squares), and thus largely distort the decision boundary. The necessity and harmfulness of data augmentations form a fundamental dilemma in self-AT. Therefore, as data augmentations play an essential role in self-AT, we need to strike a balance between utilizing strong augmentations for representation learning and avoiding large image distortions for good robustness. Although it seems impossible under the current static augmentation strategy, we notice that we can alleviate this dilemma by adopting a dynamic augmentation schedule. In particular, we could learn good representations with aggressive augmentations at the beginning stage, and gradually transfer the training robustness to the test data by adopting milder and milder augmentations at later stages. We name this method Dynamic Adversarial Contrastive Learning (DYNACL). In this way, DYNACL could benefit from both sides, gaining both representation power and robustness aligned with test distribution. Built upon DYNACL, we further design a fast postprocessing stage for bridging the difference between the pretraining and downstream tasks, dubbed DYNACL++. As a preview of the results, Figure 1 (a) shows that the proposed DYNACL and DYNACL++ bring a significant improvement over state-of-the-art self-AT methods, and achieve comparable, or even superior robustness, to vanilla sup-AT. Our main contributions are: • We reveal the reason behind the robustness gap between self-AT and sup-AT, i.e., the widely adopted aggressive data augmentations in self-supervised learning may bring the issues of training-test distribution shift and class inseparability. • We propose a dynamic augmentation strategy along the training process to balance the need for strong augmentations for representation and mild augmentations for robustness, called Dynamic Adversarial Contrastive Learning (DYNACL) with its variant DYNACL++. • Experiments show that our proposed methods improve both clean accuracy and robustness over existing self-AT methods by a large margin. Notably, DYNACL++ improves the robustness of ACL (Jiang et al., 2020) from 37.62% to 46.46% on CIFAR-10, which is even



Figure 1: Experiments on CIFAR-10: (a) Comparison of supervised AT (vanilla PGD-AT (Madry et al., 2017)) and five self-supervised AT methods: RoCL, ACL, AdvCL, our DYNACL and DYNACL++ with ResNet-18 backbone. (b) Performance of standard contrastive learning (Chen et al., 2020a) and adversarial contrastive learning (Jiang et al., 2020) using different augmentation strengths. (c) Illustrative examples of data augmentations adopted by sup-AT and self-AT. We can see that self-AT adopts much more aggressive augmentation than sup-AT.

availability

//github.com/

