CLASS BALANCING GAN WITH A CLASSIFIER IN THE LOOP Anonymous

Abstract

Generative Adversarial Networks (GANs) have swiftly evolved to imitate increasingly complex image distributions. However, majority of the developments focus on performance of GANs on balanced datasets. We find that the existing GANs and their training regimes which work well on balanced datasets fail to be effective in case of imbalanced (i.e. long-tailed) datasets. In this work we introduce a novel and theoretically motivated Class Balancing regularizer for training GANs. Our regularizer makes use of the knowledge from a pre-trained classifier to ensure balanced learning of all the classes in the dataset. This is achieved via modelling the effective class frequency based on the exponential forgetting observed in neural networks and encouraging the GAN to focus on underrepresented classes. We demonstrate the utility of our contribution in two diverse scenarios: (i) Learning representations for long-tailed distributions, where we achieve better performance than existing approaches, and (ii) Generation of Universal Adversarial Perturbations (UAPs) in the data-free scenario for the large scale datasets, where we bridge the gap between data-driven and data-free approaches for crafting UAPs.

1. INTRODUCTION

Image Generation witnessed unprecedented success in recent years following the invention of Generative Adversarial Networks (GANs) by Goodfellow et al. (2014) . GANs have improved significantly over time with the introduction of better architectures (Gulrajani et al., 2017; Radford et al., 2015) , formulation of superior objective functions (Jolicoeur-Martineau, 2018; Arjovsky et al., 2017) , and regularization techniques (Miyato et al., 2018) . An important breakthrough for GANs has been the ability to effectively use the information of class conditioning for synthesizing images (Mirza & Osindero, 2014; Miyato & Koyama, 2018) . Conditional GANs have been shown to scale to large datasets such as ImageNet (Deng et al., 2009) with 1000 classes (Miyato & Koyama, 2018) . One of the major issues with unconditional GANs has been their inability to produce balanced distributions over all the classes present in the dataset. This is seen as problem of missing modes in the generated distribution. A version of the missing modes problem, known as the 'covariate shift' problem was studied by Santurkar et al. (2018) . One possible reason may be the absence of knowledge about the class distribution P (Y |X)foot_0 of the generated samples during training. Conditional GANs on the other hand, do not suffer from this issue since the class label Y is supplied to the GAN during training. However, it has been recently found by Ravuri & Vinyals (2019) that despite being able to do well on metrics such as Inception Score (IS) (Salimans et al. (2016) ) and Frèchet Inception Distance (FID) (Heusel et al., 2017) , the samples generated from the state-of-the-art conditional GANs lack diversity in comparison to the underlying training datasets. Further, we observed that although conditional GANs work well in balanced case, they suffer performance degradation in the imbalanced case. In order to address these shortcomings, we propose an orthogonal method (with respect to label conditioning) to induce the information about the class distribution P (Y |X) of generated samples in the GAN framework using a pre-trained classifier. We achieve this by tracking the class distribution of samples produced by the GAN using a pre-trained classifier. The regularizer utilizes the class distribution to penalize excessive generation of samples from the majority classes, thus enforcing the GAN to generate samples from minority classes. Our regularizer involves a novel method of modelling the forgetting of samples by GANs, based on the exponential forgetting observed in neural networks (Kirkpatrick et al. (2017) ). We infer the implications of our regularizer by a theoretical bound and empirically verify the same. We conduct empirical analysis of the proposed class balancing regularizer in two diverse and challenging scenarios: (i) Training GANs for image generation on long-tailed datasets: Generally, even in the long-tailed distribution tasks, the test set is balanced despite the imbalance in the training set. This is because it is important to develop Machine Learning systems that generalize well across all the support regions of the data distribution, avoiding undesired over-fitting to the majority (or head) classes. Hence, it is pertinent to train GANs that can faithfully represent all classes. (ii) Transferring the knowledge from a learnt classifier (P (Y |X t )) to a GAN being trained on arbitrary prior distribution P (X p ): This is a specific situation where the samples from target distribution X t are unavailable. Instead, discriminative feature knowledge is indirectly available in the form of a trained classifier (P (Y |X t )). This is a perfect fit for crafting input-agnostic (Universal) adversarial perturbations in the data-free scenario. We show that the proposed regularizer can enable the generated samples to not only extract information about the target data with a trained classifier in the loop, but also represent its support to a greater extent. In summary, our contributions can be listed as follows: • We propose a 'class-balancing' regularizer that makes use of the statistics (P (Y |X)) of generated samples to promote uniformity while sampling from an unconditional GAN. The effect of our regularizer is depicted both theoretically (Section 3) and empirically (Section 4). • We show that our regularizer enables GANs to learn uniformly across classes even when the training distribution is long-tailed. We observe gains in FID and accuracy of a classifier trained on generated samples. • We also show that by combining a pre-trained classifier (i.e. P (Y |X t )) trained on a target dataset X t , with an arbitrary distribution P (X p ), our framework is capable of synthesizing novel samples related to the target dataset. We show that UAPs created on such novel samples generalize to real target data and hence lead to an effective data-free attack. This application is novel to our framework and cannot be realized by conditional GANs.

2. BACKGROUND

2.1 GENERATIVE ADVERSARIAL NETWORKS (GANS) Generative Adversarial Networks (GANs) are formulated as a two player game in which the discriminator D tries to classify images into two classes: real and fake. The generator G tries to generate images (transforming a noise vector z ∼ P z ) which fool the discriminator (D) into classifying them as real. The game can be formulated by the following objective: min G max D E x∼Pr [log(D(x))] + E z∼Pz [log(1 -D(G(z))] The exact optimization for training D is computationally prohibitive in large networks and the GAN is trained by alternative minimization using loss functions.  This unconditional GAN formulation does not have any class conditioning and produces different number of samples from different classes (Santurkar et al., 2018) . In other words, the distribution is not balanced (uniform) across different classes for the generated data.



Here Y represents labels and X represents data.



Multiple loss functions have been proposed for stabilizing the GAN training. In our work we use the relativistic loss function (Jolicoeur-Martineau, 2018) which is formulated as:L rel D = -E (x,z)∼(Pr,Pz) [log(σ(D(x) -D(G(z)))](2)L rel G = -E (x,z)∼(Pr,Pz) [log(σ(D(G(z)) -D(x))]

