COVARIANCE MATRIX ADAPTATION MAP-ANNEALING

Abstract

Single-objective optimization algorithms search for the single highest-quality solution with respect to an objective. Quality diversity (QD) algorithms, such as Covariance Matrix Adaptation MAP-Elites (CMA-ME), search for a collection of solutions that are both high-quality with respect to an objective and diverse with respect to specified measure functions. However, CMA-ME suffers from three major limitations highlighted by the QD community: prematurely abandoning the objective in favor of exploration, struggling to explore flat objectives, and having poor performance for low-resolution archives. We propose a new quality diversity algorithm, Covariance Matrix Adaptation MAP-Annealing (CMA-MAE), that addresses all three limitations. We provide theoretical justifications for the new algorithm with respect to each limitation. Our theory informs our experiments, which support the theory and show that CMA-MAE achieves state-of-the-art performance.

1. INTRODUCTION

Consider an example problem of searching for celebrity faces in the latent space of a generative model. As a single-objective optimization problem, we specify an objective f that targets a celebrity such as Tom Cruise. A single-objective optimizer, such as CMA-ES (Hansen, 2016) , will converge to a single solution of high objective value, an image that looks like Tom Cruise as much as possible. However, this objective has ambiguity. How old was Tom Cruise in the photo? Did we want the person in the image to have short or long hair? By instead framing the problem as a quality diversity optimization problem, we additionally specify a measure function m 1 that quantifies age and a measure function m 2 that quantifies hair length. A quality diversity algorithm (Pugh et al., 2015; Chatzilygeroudis et al., 2021) , such as CMA-ME (Fontaine et al., 2020) , can then optimize for a collection of images that are diverse with respect to age and hair length, but all look like Tom Cruise. While previous work (Fontaine et al., 2020; 2021a; b; Earle et al., 2021) has shown that CMA-ME solves such QD problems efficiently, three important limitations of the algorithm have been discovered. First, on difficult to optimize objectives, variants of CMA-ME will abandon the objective too soon (Tjanaka et al., 2022) , and instead favor exploring the measure space, the vector space defined by the measure function outputs. Second, the CMA-ME algorithm struggles to explore flat objective functions (Paolo et al., 2021) . Third, CMA-ME works well on high-resolution archives, but struggles to explore low-resolution archives (Cully, 2021; Fontaine & Nikolaidis, 2021a) . We note that the chosen archive resolution affects the performance of all current QD algorithms. We propose a new algorithm, CMA-MAE, that addresses these three limitations. To address the first limitation, we derive an algorithm that smoothly blends between CMA-ES and CMA-ME. First, consider how CMA-ES and CMA-ME differ. At each step CMA-ES's objective ranking maximizes the objective function f by approximating the natural gradient of f at the current solution point (Akimoto et al., 2010) . In contrast, CMA-ME's improvement ranking moves in the direction of the natural gradient of f -f A at the current solution point, where f A is a discount function equal to the objective of the best solution so far that has the same measure values as the current solution point. The function f -f A quantifies the gap between a candidate solution and the best solution so far at the candidate solution's position in measure space. Our key insight is to anneal the function f A by a learning rate α. We observe that when α = 0, then our discount function f A never increases and our algorithm behaves like CMA-ES. However, when

