GRADIENT-INFORMED QUALITY DIVERSITY FOR THE ILLUMINATION OF DISCRETE SPACES Anonymous authors Paper under double-blind review

Abstract

Quality Diversity (QD) algorithms have been proposed to search for a large collection of both diverse and high-performing solutions instead of a single set of local optima. While early QD algorithms view the objective and descriptor functions as black-box functions, novel tools have been introduced to use gradient information to accelerate the search and improve overall performance of those algorithms over continuous input spaces. However a broad range of applications involve discrete spaces, such as drug discovery or image generation. Exploring those spaces is challenging as they are combinatorially large and gradients cannot be used in the same manner as in continuous spaces. We introduce MAP-ELITES with a Gradient-Informed Discrete Emitter (ME-GIDE), which extends QD optimisation with differentiable functions over discrete search spaces. ME-GIDE leverages the gradient information of the objective and descriptor functions with respect to its discrete inputs to propose gradient-informed updates that guide the search towards a diverse set of high quality solutions. We evaluate our method on challenging benchmarks including protein design and discrete latent space illumination and find that our method outperforms state-of-the-art QD algorithms in all benchmarks.

1. INTRODUCTION

Quality-Diversity (QD) Optimization algorithms Chatzilygeroudis et al. (2021) ; Cully & Demiris (2017) have changed the classical paradigm of optimization: inspired by natural evolution, the essence of QD methods is to provide a large and diverse set of high-performing solutions rather than only the best one. This core idea showed great outcomes in different fields such as robotics Cully et al. (2015) where it allows to learn to control robots using diverse and efficient policies or latent space exploration Fontaine et al. (2021) of generative models to generate a diverse set of high quality images or video game levels. The main QD approaches are derivative-free optimizers, such as the MAP-ELITES algorithm Mouret & Clune (2015), a genetic algorithm that tries to find high quality solutions covering the space defined by a variety of user-defined features of interest. However, subsequent methods Nilsson & Cully (2021); Fontaine & Nikolaidis (2021) have shown that when the objective functions (quality) or the descriptor functions (diversity) are differentiable, using gradients of those functions can improve convergence both in terms of speed and performance compared to traditional QD algorithms. Those methods were applied to Reinforcement Learning problems such as robotics Pierrot et al. (2022a) or latent space illumination Fontaine & Nikolaidis (2021) of generative adversarial networks. Those applications focused on optimization over continuous variables. However many real-world applications are best framed using discrete features. For instance, generative architecture such as discrete VAEs Van Den Oord et al. (2017) ; Rolfe (2016) have shown capability to generate highquality images Razavi et al. (2019) and have been at the core of recent successful approaches for text-to-image generation Ramesh et al. (2021); Gafni et al. (2022) . A further natural application using discrete variables is protein design Huang et al. (2016) : proteins play a role in key functions in nature and protein design allows to create new drugs or biofuels. As proteins are sequences of 20 possible amino acids, designing them is a tremendously hard task as only a few of those possible sequences are plausible in the real world. Deep learning for biological sequences provided significant advances in several essential tasks for biological sequence, such AlphaFold for structure prediction Jumper et al. (2021a) or language models for amino acid likelihood prediction Rives et al.

