C-LEARNING: HORIZON-AWARE CUMULATIVE ACCESSIBILITY ESTIMATION

Abstract

Multi-goal reaching is an important problem in reinforcement learning needed to achieve algorithmic generalization. Despite recent advances in this field, current algorithms suffer from three major challenges: high sample complexity, learning only a single way of reaching the goals, and difficulties in solving complex motion planning tasks. In order to address these limitations, we introduce the concept of cumulative accessibility functions, which measure the reachability of a goal from a given state within a specified horizon. We show that these functions obey a recurrence relation, which enables learning from offline interactions. We also prove that optimal cumulative accessibility functions are monotonic in the planning horizon. Additionally, our method can trade off speed and reliability in goal-reaching by suggesting multiple paths to a single goal depending on the provided horizon. We evaluate our approach on a set of multi-goal discrete and continuous control tasks. We show that our method outperforms state-of-the-art goal-reaching algorithms in success rate, sample complexity, and path optimality.

1. INTRODUCTION

Multi-goal reinforcement learning tackles the challenging problem of reaching multiple goals, and as a result, is an ideal framework for real-world agents that solve a diverse set of tasks. Despite progress in this field (Kaelbling, 1993; Schaul et al., 2015; Andrychowicz et al., 2017; Ghosh et al., 2019) , current algorithms suffer from a set of limitations: an inability to find multiple paths to a goal, high sample complexity, and poor results in complex motion planning tasks. In this paper we propose C-learning, a method which addresses all of these shortcomings. Many multi-goal reinforcement learning algorithms are limited by learning only a single policy π(a|s, g) over actions a to reach goal g from state s. There is an unexplored trade-off between reaching the goal reliably and reaching it quickly. We illustrate this shortcoming in Figure 1a , which represents an environment where an agent must reach a goal on the opposite side of some predator. Shorter paths can reach the goal faster at the cost of a higher probability of being eaten. Existing algorithms do not allow a dynamic choice of whether to act safely or quickly at test time. The second limitation is sample complexity. Despite significant improvements (Andrychowicz et al., 2017; Ghosh et al., 2019) , multi-goal reaching still requires a very large amount of environment interactions for effective learning. We argue that the optimal Q-function must be learned to high accuracy for the agent to achieve reasonable performance, and this leads to sample inefficiency. The same drawback of optimal Q-functions often causes agents to learn sub-optimal ways of reaching the intended goal. This issue is particularly true for motion planning tasks (Qureshi et al., 2020) , where current algorithms struggle. We propose to address these limitations by learning horizon-aware policies π(a|s, g, h), which should be followed to reach goal g from state s in at most h steps. The introduction of a time horizon h naturally allows us to tune the speed/reliability trade-off, as an agent wishing to reach the goal faster should select a policy with a suitably small h value. To learn these policies, we introduce the optimal cumulative accessibility function C * (s, a, g, h). This is a generalization of the state-action value function and corresponds to the probability of reaching goal g from state s after at most h steps if action a is taken, and the agent acts optimally thereafter. Intuitively it is similar to the optimal Q-function, but Q-functions rarely correspond to probabilities, whereas the C * -function does so by construction. We derive Bellman backup update rules for C * , which allow it to be learned via minimization of unbiased estimates of the cross-entropy loss -this is in contrast to Q-learning, which optimizes biased estimates of the squared error. Policies π(a|s, g, h) can then be recovered from the C * function. We call our method cumulative accessibility estimation, or Clearning. Pong et al. (2018) proposed TDMs, a method involving horizon-aware policies. We point out that their method is roughly related to a non-cumulative version of ours with a different loss that does not enable the speed/reliability trade-off and is ill-suited for sparse rewards. We include a detailed discussion of TDMs in section 4. One might expect that adding an extra dimension to the learning task, namely h, would increase the difficulty -as C * effectively contains the information of several optimal Q-functions for different discount factors. However, we argue that C * does not need to be learned to the same degree of accuracy as the optimal Q-function for the agent to solve the task. As a result, learning C * is more efficient, and converges in fewer environmental interactions. This property, combined with our proposed goal sampling technique and replay buffer used during training, provides empirical improvements over Q-function based methods. In addition to these advantages, learning C * is itself useful, containing information that the horizonaware policies do not. It estimates whether a goal g is reachable from the current state s within h steps. In contrast, π(a|s, g, h) simply returns some action, even for unreachable goals. We show that C * can be used to determine reachability with examples in a nonholonomic environment. Summary of contributions: (i) introducing C-functions and cumulative accessibility estimation for both discrete and continuous action spaces; (ii) highlighting the importance of the speed vs reliability trade-off in finite horizon reinforcement learning; (iii) introducing a novel replay buffer specially tailored for learning C * which builds on HER (Andrychowicz et al., 2017) ; and (iv) empirically showing the effectiveness of our method for goal-reaching as compared to existing alternatives, particularly in the context of complex motion planning tasks.

2. BACKGROUND AND RELATED WORK

Let us extend the Markov Decision Process (MDP) formalism (Sutton et al., 1998) for goal-reaching. We consider a set of actions A, a state space S, and a goal set G. We assume access to a goal checking function G : S × G → {0, 1} such that G(s, g) = 1 if and only if state s achieves goal g. For example, achieving the goal could mean exactly reaching a certain state, in which case G = S and



Figure1: (a) A continuous spectrum of paths allow the mouse to reach its goal faster, at an increased risk of disturbing the cat and being eaten. (b) Q * (with γ = 0.99) needs to be learned more accurately than C * to act optimally. The goal g can be reached in h * = 5 steps from s, so that Q * (s, g, a * ) = 0.99 5 and Q * (s, g, a-) = 0.99 7 ; while C * (s, a * , g, h * ) = 1 and C * (s, a-, g, h * ) = 0.

availability

visualizations can be found at https://sites.google.com/view/ learning-cae/.

