IMPROVING ADVERSARIAL ROBUSTNESS BY CON-TRASTIVE GUIDED DIFFUSION PROCESS Anonymous

Abstract

Synthetic data generation has become an emerging tool to help improve the adversarial robustness in classification tasks since robust learning requires a significantly larger amount of training samples compared with standard classification tasks. Among various deep generative models, the diffusion model has been shown to produce high-quality synthetic images and has achieved good performance in improving the adversarial robustness. However, diffusion-type methods are typically slow in data generation as compared with other generative models. Although different acceleration techniques have been proposed recently, it is also of great importance to study how to improve the sample efficiency of generated data for the downstream task. In this paper, we first analyze the optimality condition of synthetic distribution for achieving non-trivial robust accuracy. We show that enhancing the distinguishability among the generated data is critical for improving adversarial robustness. Thus, we propose the Contrastive-Guided Diffusion Process (Contrastive-DP), which adopts the contrastive loss to guide the diffusion model in data generation. We verify our theoretical results using simulations and demonstrate the good performance of Contrastive-DP on image datasets.

1. INTRODUCTION

The success of most deep learning methods relies heavily on a massive amount of training data, which can be expensive to acquire in practice. For example, in autonomous driving (O'Kelly et al., 2018) and the medical diagnosis (Das et al., 2022) type applications, the number of rare scenes is usually very limited in real data. Moreover, it may be expensive to label the data in supervised learning. These challenges call for methods that can produce additional training data that satisfy two essential properties: (i) the additional data should help improve the downstream task performance; (ii) the additional data should be easy to generate. Synthetic data generation based on deep generative models has shown promising performance recently to tackle these challenges (Sehwag et al., 2022; Gowal et al., 2021; Das et al., 2022) . In synthetic data generation, one aims to learn a synthetic distribution (from which we generate synthetic data) that is close to the true date-generating distribution based on training data available, and most importantly, can help improve the downstream task performance. Synthetic data generation is highly related to generative models. Among various kinds of generative models, the score-based model and diffusion type models have gained much success in image generation recently (Song & Ermon, 2019; Song et al., 2021b; 2020; Song & Ermon, 2020; Sohl-Dickstein et al., 2015; Nichol & Dhariwal, 2021; Bao et al., 2022; Rombach et al., 2022) . As validated in image datasets, the prototype of diffusion models, the Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020) , and many variants can generate high-quality image data as compared with classical generative models such as GANs (Dhariwal & Nichol, 2021) . This paper mainly focuses on the adversarial robust classification of image data, which typically requires more training data than standard classification tasks. In Gowal et al. (2021) , 100M highquality synthetic images are generated by DDPM and achieve the state-of-the-art performance on adversarial robustness on the CIFAR-10 dataset, which demonstrates the effectiveness of diffusion models in improving adversarial robustness. However, a major drawback of diffusion-type methods is the slow computational speed. More specifically, DDPM is usually 1000 times slower than GAN (Song et al., 2021a) and this drawback is more serious when generating a large number of samples,

