ON NOISE INJECTION IN GENERATIVE ADVERSARIAL NETWORKS

Abstract

Noise injection is an effective way of circumventing overfitting and enhancing generalization in machine learning, the rationale of which has been validated in deep learning as well. Recently, noise injection exhibits surprising effectiveness when generating high-fidelity images in Generative Adversarial Networks (e.g. StyleGAN). Despite its successful applications in GANs, the mechanism of its validity is still unclear. In this paper, we propose a geometric framework to theoretically analyze the role of noise injection in GANs. Based on Riemannian geometry, we successfully model the noise injection framework as fuzzy equivalence on geodesic normal coordinates. Guided by our theories, we find that existing methods are incomplete and a new strategy for noise injection is devised. Experiments on image generation and GAN inversion demonstrate the superiority of our method.

1. INTRODUCTION

Noise injection is usually applied as regularization to cope with overfitting or facilitate generalization in neural networks (Bishop, 1995; An, 1996) . The effectiveness of this simple technique has also been proved in various tasks in deep learning, such as learning deep architectures (Hinton et al., 2012; Srivastava et al., 2014; Noh et al., 2017) , defending adversarial attacks (He et al., 2019) , facilitating stability of differentiable architecture search with reinforcement learning (Liu et al., 2019; Chu et al., 2020) , and quantizing neural networks (Baskin et al., 2018) . In recent years, noise injectionfoot_0 has attracted more and more attention in the community of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014a) . Extensive research shows that it helps stabilize the training procedure (Arjovsky & Bottou, 2017; Jenni & Favaro, 2019) and generate images of high fidelity (Karras et al., 2019a; b; Brock et al., 2018) . In practice, Fig. 1 shows significant improvement in hair quality due to noise injection. Particularly, noise injection in StyleGAN (Karras et al., 2019a; b) has shown the amazing capability of helping generate sharp details in images, shedding new light on obtaining high-quality photo-realistic results using GANs. Therefore, studying the underlying principle of noise injection in GANs is an important theoretical work of understanding GAN algorithms. In this paper, we propose a theoretical framework to explain and improve the effectiveness of noise injection in GANs. Our framework is motivated from a geometric perspective and also combined with the results of optimal transportation problem in GANs (Lei et al., 2019a; b) . Our contributions are listed as follows: • We show that the existing GAN architectures, including Wasserstein GANs (Arjovsky et al., 2017) , may suffer from adversarial dimension trap, which severely penalizes the property of generator; • Based on our theory, we attempt to explain the properties that noise injection is applied in the related literatures; • Based on our theory, we propose a more proper form for noise injection in GANs, which can overcome the adversarial dimension trap. Experiments on the state-of-the-art GAN architecture, StyleGAN2 (Karras et al., 2019b) , demonstrate the superiority of our new method compared with original noise injection used in StyleGAN2.



It suffices to note that noise injection here is totally different from the research field of adversarial attacks raised inGoodfellow et al. (2014b).

