MITIGATING MODE COLLAPSE BY SIDESTEPPING CATASTROPHIC FORGETTING

Abstract

Generative Adversarial Networks (GANs) are a class of generative models used for various applications, but they have been known to suffer from the mode collapse problem, in which some modes of the target distribution are ignored by the generator. Investigative study using a new data generation procedure indicates that the mode collapse of the generator is driven by the discriminator's inability to maintain classification accuracy on previously seen samples, a phenomenon called Catastrophic Forgetting in continual learning. Motivated by this observation, we introduce a novel training procedure that dynamically spawns additional discriminators to remember previous modes of generation. On several datasets, we show that our training scheme can be plugged-in to existing GAN frameworks to mitigate mode collapse and improve standard metrics for GAN evaluation.

1. INTRODUCTION

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are an extremely popular class of generative models that is not only used for text and image generation, but also in various fields of science and engineering, including biomedical imaging (Yi et al., 2019; Nie et al., 2018; Wolterink et al., 2017) , autonomous driving (Hoffman et al., 2018; Zhang et al., 2018), and robotics (Rao et al., 2020; Bousmalis et al., 2018) . However, GANs are widely known to be prone to mode collapse, which refers to a situation where the generator only samples a few modes of the real data, failing to faithfully capture other more complex or less frequent categories. While the mode collapse problem is often overlooked in text and image generation tasks, and even traded off for higher realism of individual samples (Karras et al., 2019; Brock et al., 2019) , dropping infrequent classes may cause serious problems in real-world problems, in which the infrequent classes represent important anomalies. For example, a collapsed GAN can produce racial/gender biased images (Menon et al., 2020) . Moreover, mode collapse causes instability in optimization, which can damage not only the diversity but also the realism of individual samples of the final results. As an example, we visualized the training progression of the vanilla GAN (Goodfellow et al., 2014) for a simple bimodal distribution in the top row of Figure 1 . At collapse, the discriminator conveniently assigns high realism to the region unoccupied by the generator, regardless of the true density of the real data. This produces a strong gradient for the generator to move its samples toward the dropped mode, swaying mode collapse to the opposite side. In particular, the discriminator loses its ability to detect fake samples it was previously able to, such as point X . The oscillation continues without convergence. From this observation, we hypothesize that the mode collapse problem in GAN training is closely related to Catastrophic Forgetting (McCloskey & Cohen, 1989; McClelland et al., 1995; Ratcliff, 1990) in continual learning. That is, since the distribution of the generated samples is not stationary, the discriminator forgets to classify the previously generated samples as fake, hindering convergence of the GAN minimax game. A promising line of works (Zhang et al., 2019b; Rajasegaran et al., 2019; Rusu et al., 2016; Fernando et al., 2017) tackle the problem in the supervised learning setting by instantiating multiple predictors, each of which takes charge in a particular subset of the whole distribution. Likewise, we also tackle the problem of mode collapse in GAN by tracking the severity of Catastrophic Forgetting by storing a few exemplar data during training, and dynamically spawning an additional discriminator if forgetting is detected, as shown in Figure 1 . The key idea is that the added discriminator is left intact unless the generator recovers from mode dropping of that sample, essentially sidestepping catastrophic forgetting.

