A CMDP-WITHIN-ONLINE FRAMEWORK FOR META-SAFE REINFORCEMENT LEARNING

Abstract

Meta-reinforcement learning has widely been used as a learning-to-learn framework to solve unseen tasks with limited experience. However, the aspect of constraint violations has not been adequately addressed in the existing works, making their application restricted in real-world settings. In this paper, we study the problem of meta-safe reinforcement learning (Meta-SRL) through the CMDP-within-online framework to establish the first provable guarantees in this important setting. We obtain task-averaged regret bounds for the reward maximization (optimality gap) and constraint violations using gradient-based meta-learning and show that the taskaveraged optimality gap and constraint satisfaction improve with task-similarity in a static environment or task-relatedness in a dynamic environment. Several technical challenges arise when making this framework practical. To this end, we propose a meta-algorithm that performs inexact online learning on the upper bounds of within-task optimality gap and constraint violations estimated by offpolicy stationary distribution corrections. Furthermore, we enable the learning rates to be adapted for every task and extend our approach to settings with a competing dynamically changing oracle. Finally, experiments are conducted to demonstrate the effectiveness of our approach.

1. INTRODUCTION

The field of meta-reinforcement learning (meta-RL) has recently evolved as one of the promising directions that enables reinforcement learning (RL) agents to learn quickly in dynamically changing environments (Finn et al., 2017; Mitchell et al., 2021; Zintgraf et al., 2021) . Many real-world applications, nevertheless, have safety constraints that should rarely be violated, which existing works do not fully address. Safe RL problems are often modeled as constrained Markov decision processes (CMDPs), where the agent aims to maximize the value function while satisfying given constraints on the trajectory (Altman, 1999). However, unlike meta-learning, CMDP algorithms are not designed to generalize efficiently over unseen tasks (Paternain et al., 2022; Ding et al., 2021a; Ding & Lavaei, 2022) In this paper, we study how meta-learning can be principally designed to help safe RL algorithms adapt quickly while satisfying safety constraints. There are several unique challenges involved in meta-learning for the CMDP settings. First, multiple losses are incurred at each time step, i.e., reward and constraints, which are typically nonconvex and coupled through dynamics. Hence, adapting existing theories developed for stylized settings such as online convex optimization (Hazan et al., 2016) is not straightforward. Second, it is unrealistic to assume the computation of a globally optimal policy for CMDPs (unlike online learning (Hazan et al., 2016) ). Thus, classical online learning algorithms that assume exact or unbiased estimator of the loss

