APPROXIMATED ANOMALOUS DIFFUSION: GAUSSIAN MIXTURE SCORE-BASED GENERATIVE MODELS

Abstract

Score-based generative models (SGMs) can generate high-quality samples via Langevin dynamics with a drift term and a diffusion term (Gaussian noise) iteratively calculated and added to a sample until convergence. In biological systems, it is observed that the neural population can conduct heavy-tailed Lévy dynamics for sampling-based probabilistic representation through neural fluctuations. Critically, unlike the existing sampling process of SGMs, heavy-tailed Lévy dynamics can produce both large jumps and small roaming to explore the sampling space, resulting in better sampling results than Langevin dynamics with a lacking of large jumps. Motivated by this contrast, we explore a new class of SGMs with the sampling based on the Lévy dynamics. However, exact numerical simulation of the Lévy dynamics is significantly more challenging and intractable. We hence propose an alternative solution by leveraging Gaussian mixture noises during training to mimic the desired large jumps and small roaming properties. Theoretically, GM-SGMs conduct a probabilistic graphical model used by empirical Bayes for sampling, expanding the maximum a posteriori (MAP) estimation applied by conventional SGMs. Expensive experiments on the challenging image generation tasks show that our GM-SGMs exhibit superior sampling quality over prior art SGMs across various sampling iterations.

1. INTRODUCTION

Score-based generative models (SGMs) (Song and Ermon, 2019; 2020; Song et al., 2021b; a; Dockhorn et al., 2022; Karras et al., 2022) have recently demonstrated tremendous performance in data synthesis, especially high-quality images, along with easier model optimization (Song and Ermon, 2019) , richer generative diversity (Xiao et al., 2022) , and solid theories (De Bortoli et al., 2021) . During optimization, SGMs learn to fit a score function by predicting the Gaussian noises added to a sample drawn from a target dataset. To generate a sample in the target distribution, the SGMs conduct Langevin dynamics constructed from the score function. This process reverses a Brownian motion starting from the dataset distribution with i.i.d. Gaussian increments. As a special case of Monte Carlo Markov Chain methods, the Langevin dynamics has been widely applied for constructing sampling-based algorithms (Rey-Bellet and Spiliopoulos, 2015). However, increasing evidence from experimental observation suggests that in biological systems, the neural population implements sampling-based probabilistic representations through a heavy-tailed Lévy dynamics (He, 2014; Donoghue et al., 2020; Townsend and Gong, 2018; Muller et al., 2018) , which instead reverses an anomalous diffusion process with heavy-tailed increments (Fig. 2 (Left)). The neural coding benefits from Lévy dynamics, since it can implement large jumps that facilitates the process to escape from local minimal and explore the sampling space more thoroughly (Ye and Zhu, 2018; Qi and Gong, 2022) . A natural question arises: can we apply the Lévy dynamics instead of the Langevin dynamics for better sampling performance of SGMs? Inspired by this insight, we explore a novel class of SGMs that reverse the anomalous diffusion for sampling. Nonetheless, exact numerical simulation of the Lévy dynamics (i.e., reversing the anomalous diffusion) is drastically more challenging and intractable, especially for high-dimensional data such as images. To tackle this challenge, we consider Brownian motion with Gaussian mixture as an approximation of the anomalous diffusion. To tackle this challenge, we train the SGMs with Gaussian mixture noises to enable both large jumps and small roaming during the sampling phase, 1

