GRADIENT-INFORMED QUALITY DIVERSITY FOR THE ILLUMINATION OF DISCRETE SPACES Anonymous authors Paper under double-blind review

Abstract

Quality Diversity (QD) algorithms have been proposed to search for a large collection of both diverse and high-performing solutions instead of a single set of local optima. While early QD algorithms view the objective and descriptor functions as black-box functions, novel tools have been introduced to use gradient information to accelerate the search and improve overall performance of those algorithms over continuous input spaces. However a broad range of applications involve discrete spaces, such as drug discovery or image generation. Exploring those spaces is challenging as they are combinatorially large and gradients cannot be used in the same manner as in continuous spaces. We introduce MAP-ELITES with a Gradient-Informed Discrete Emitter (ME-GIDE), which extends QD optimisation with differentiable functions over discrete search spaces. ME-GIDE leverages the gradient information of the objective and descriptor functions with respect to its discrete inputs to propose gradient-informed updates that guide the search towards a diverse set of high quality solutions. We evaluate our method on challenging benchmarks including protein design and discrete latent space illumination and find that our method outperforms state-of-the-art QD algorithms in all benchmarks.

1. INTRODUCTION

Quality-Diversity (QD) Optimization algorithms Chatzilygeroudis et al. (2021) ; Cully & Demiris (2017) have changed the classical paradigm of optimization: inspired by natural evolution, the essence of QD methods is to provide a large and diverse set of high-performing solutions rather than only the best one. This core idea showed great outcomes in different fields such as robotics Cully et al. (2015) where it allows to learn to control robots using diverse and efficient policies or latent space exploration Fontaine et al. (2021) of generative models to generate a diverse set of high quality images or video game levels. The main QD approaches are derivative-free optimizers, such as the MAP-ELITES algorithm Mouret & Clune (2015) , a genetic algorithm that tries to find high quality solutions covering the space defined by a variety of user-defined features of interest. However, subsequent methods Nilsson & Cully (2021); Fontaine & Nikolaidis (2021) have shown that when the objective functions (quality) or the descriptor functions (diversity) are differentiable, using gradients of those functions can improve convergence both in terms of speed and performance compared to traditional QD algorithms. Those methods were applied to Reinforcement Learning problems such as robotics Pierrot et al. (2022a) or latent space illumination Fontaine & Nikolaidis (2021) of generative adversarial networks. Those applications focused on optimization over continuous variables. However many real-world applications are best framed using discrete features. At each iteration, an element is sampled in the repertoire. Gradients are computed over continuous fitness and descriptor functions with respect to their discrete inputs. Gradients are linearly combined to favour higher fitness and exploration of the descriptor space. Probabilities of mutation over the neighbours of the element are derived from this gradient information. Finally, a mutant is sampled according to those probabilities and inserted back in the repertoire. (2021) . These successes motivate the use of these tools as objective functions for protein design Anishchenko et al. (2021a) . In practice, objective functions are often written as differentiable functions of their inputs, enabling the use of gradient information of the continuous extension if the input are discrete Grathwohl et al. (2021) . We second this research trend and address gradient informed quality-diversity optimization over discrete inputs when objective and descriptors functions are differentiable. We make several contributions: (i) we introduce MAP-ELITES with a Gradient-Informed Discrete Emitter (ME-GIDE), a genetic algorithm where mutations are sampled thanks to a gradient informed distribution. (ii) We propose a way to select the hyperparameters of this algorithm by weighting the importance we give to this gradient information. (iii) We demonstrate the ability of our method to better illuminate discrete spaces on proteins and the discrete latent space of a generative model.

2. PROBLEM DEFINITION

In this work, we consider the problem of Quality Diversity Optimization over discrete spaces. The QD problem assumes an objective function f : X → R, where X is called search space, and d descriptors c i : X → R, or as a single descriptor function c : X → R d . We note S = c(X ) the descriptor space formed by the range of c. We only consider discrete spaces of dimension m with K categories such that X ⊂ {1, ..., K} m . QD algorithms of the family of the MAP-ELITES algorithm, discretize the descriptor space S via a tessellation method. Let T be the tessellation of S into M cells S i . The goal of QD methods is to find a set of solutions x i ∈ X so that each solution x i occupies a different cell S i in T and maximizes the objective function within that cell. The QD objective can thus be formalized as follows: max x∈X M i=1 f (x i ), where ∀i, c(x i ) ∈ S i . (1) We consider here that both the objective and descriptor functions are actually defined on real values R and restricted to the discrete inputs X . We also consider them to be first-order differentiable, hence for any input x ∈ X , we can compute gradients ∇ x f (x) and ∇ x c i (x).



For instance, generative architecture such as discrete VAEs Van Den Oord et al. (2017); Rolfe (2016) have shown capability to generate highquality images Razavi et al. (2019) and have been at the core of recent successful approaches for text-to-image generation Ramesh et al. (2021); Gafni et al. (2022). A further natural application using discrete variables is protein design Huang et al. (2016): proteins play a role in key functions in nature and protein design allows to create new drugs or biofuels. As proteins are sequences of 20 possible amino acids, designing them is a tremendously hard task as only a few of those possible sequences are plausible in the real world. Deep learning for biological sequences provided significant advances in several essential tasks for biological sequence, such AlphaFold for structure prediction Jumper et al. (2021a) or language models for amino acid likelihood prediction Rives et al.

Figure1: MAP-ELITES with Gradients Informed Discrete Emitter (ME-GIDE). At each iteration, an element is sampled in the repertoire. Gradients are computed over continuous fitness and descriptor functions with respect to their discrete inputs. Gradients are linearly combined to favour higher fitness and exploration of the descriptor space. Probabilities of mutation over the neighbours of the element are derived from this gradient information. Finally, a mutant is sampled according to those probabilities and inserted back in the repertoire.

