IMPROVING ADVERSARIAL ROBUSTNESS BY CON-TRASTIVE GUIDED DIFFUSION PROCESS Anonymous

Abstract

Synthetic data generation has become an emerging tool to help improve the adversarial robustness in classification tasks since robust learning requires a significantly larger amount of training samples compared with standard classification tasks. Among various deep generative models, the diffusion model has been shown to produce high-quality synthetic images and has achieved good performance in improving the adversarial robustness. However, diffusion-type methods are typically slow in data generation as compared with other generative models. Although different acceleration techniques have been proposed recently, it is also of great importance to study how to improve the sample efficiency of generated data for the downstream task. In this paper, we first analyze the optimality condition of synthetic distribution for achieving non-trivial robust accuracy. We show that enhancing the distinguishability among the generated data is critical for improving adversarial robustness. Thus, we propose the Contrastive-Guided Diffusion Process (Contrastive-DP), which adopts the contrastive loss to guide the diffusion model in data generation. We verify our theoretical results using simulations and demonstrate the good performance of Contrastive-DP on image datasets.

1. INTRODUCTION

The success of most deep learning methods relies heavily on a massive amount of training data, which can be expensive to acquire in practice. For example, in autonomous driving (O'Kelly et al., 2018) and the medical diagnosis (Das et al., 2022) type applications, the number of rare scenes is usually very limited in real data. Moreover, it may be expensive to label the data in supervised learning. These challenges call for methods that can produce additional training data that satisfy two essential properties: (i) the additional data should help improve the downstream task performance; (ii) the additional data should be easy to generate. Synthetic data generation based on deep generative models has shown promising performance recently to tackle these challenges (Sehwag et al., 2022; Gowal et al., 2021; Das et al., 2022) . In synthetic data generation, one aims to learn a synthetic distribution (from which we generate synthetic data) that is close to the true date-generating distribution based on training data available, and most importantly, can help improve the downstream task performance. Synthetic data generation is highly related to generative models. Among various kinds of generative models, the score-based model and diffusion type models have gained much success in image generation recently (Song & Ermon, 2019; Song et al., 2021b; 2020; Song & Ermon, 2020; Sohl-Dickstein et al., 2015; Nichol & Dhariwal, 2021; Bao et al., 2022; Rombach et al., 2022) . As validated in image datasets, the prototype of diffusion models, the Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020) , and many variants can generate high-quality image data as compared with classical generative models such as GANs (Dhariwal & Nichol, 2021) . This paper mainly focuses on the adversarial robust classification of image data, which typically requires more training data than standard classification tasks. In Gowal et al. ( 2021), 100M highquality synthetic images are generated by DDPM and achieve the state-of-the-art performance on adversarial robustness on the CIFAR-10 dataset, which demonstrates the effectiveness of diffusion models in improving adversarial robustness. However, a major drawback of diffusion-type methods is the slow computational speed. More specifically, DDPM is usually 1000 times slower than GAN (Song et al., 2021a) and this drawback is more serious when generating a large number of samples, e.g., it takes more than 99 GPU daysfoot_0 for generating 100M image data according to Gowal et al. (2021) . Moreover, the computational costs will also increase dramatically when the resolution of images increases, which inspires a plentiful of works studying how to accelerate the diffusion models (Song et al., 2021a; Watson et al., 2022; Ma et al., 2022; Salimans & Ho, 2022; Bao et al., 2022; Cao et al., 2022; Yang et al., 2022) . In this paper, we aim to study the aforementioned problem from a different perspective -"how to generate effective synthetic data that are most helpful for the downstream task?". We analyze the optimal synthetic distribution for the downstream tasks to improve the sample efficiency of the generative model. We first study the theoretical insights for finding the optimal synthetic distributions for achieving adversarial robustness. Following the setting considered in Carmon et al. ( 2019), we introduce a family of synthetic distributions controlled by the distinguishability of the representation from different classes. Our theoretical results show that the more distinguishable the representation is for the synthetic data, the higher the classification accuracy we will get when training a model on such synthetic data sets. Motivated by the theoretical insights, we propose the Contrastive-Guided Diffusion Process (Contrastive-DP) for efficient synthetic data generation, incorporating the contrastive learning loss (van den Oord et al., 2018; Chuang et al., 2020; Robinson et al., 2021) into the diffusion process. We conduct comprehensive simulations and experiments on real image datasets to demonstrate the effectiveness of the proposed Contrastive-DP. The remainder of the paper is organized as follows. Section 2 presents the problem formulation and preliminaries on diffusion models. Section 3 contains the theoretical insights of optimal synthetic distribution under the Gaussian setting. Section 4 proposes a new type of data generation procedure that combines contrastive learning with diffusion models, as motivated by the theoretical insights obtained in Section 3. Finally, Section 5 conducts extensive numerical experiments to validate the good performance of the proposed generation method on simulation and image datasets.

2. PROBLEM FORMULATION AND PRELIMINARIES

We first give a brief overview of adversarial robust classification, which is our main focus, but the whole framework is widely applicable to other downstream tasks in general. Denote the feature space as X , the corresponding label space as Y, and the true (joint) data distribution as D = D X ×Y . Assume we have labeled training data D train := {(x i , y i )} n i=1 . We aim to learn a robust classifier f θ : X → Y, parameterized by a learnable θ, that can achieve minimum adversarial loss: min θ L adv (θ) := E (x,y)∼D max δ∈∆ ℓ(x + δ, y, θ) , where ℓ(x, y, θ) = 1{y ̸ = f θ (x)} is the 0-1 loss function, 1{•} is the indicator function, and ∆ = {δ : ∥δ∥ ∞ ≤ ϵ} is the adversarial set defined using ℓ ∞ -norm. Intuitively, the solution to (1) is a robust classifier that minimizes the worst-case loss within an ϵ-neighborhood of the input features. In the canonical form of adversarial training, we train the robust classifier f θ on the training set D train := {(x i , y i )} n i=1 by solving the following sample average approximation of (1): min θ L adv (θ) := 1 n n i=1 max δi∈∆ ℓ(x i + δ i , y i , θ). (2)

2.1. ADVERSARIAL TRAINING USING SYNTHETIC DATA

Synthetic data generation is one way to artificially increase the size of the training set by generating a sufficient amount of additional data, thus helping improve the learning algorithm's performance (Gowal et al., 2021) . The mainstream generation procedures can be categorized into two types: (i) generate the features (x) first and then assign pseudo labels to the generated features; (ii) or perform conditional generation conditioned on the desired label. Our analysis is mainly based on the former paradigm, which can be easily generalized to the conditional generation procedure, and our proposed algorithm is also flexible enough for both pipelines. Denote the distribution of the generated features as D X and the generated synthetic data as D syn := {( xi , ỹi )} ñ i=1 . Here the feature



Running on a RTX 4x2080Ti GPU cluster.

