MIND THE GAP: OFFLINE POLICY OPTIMIZATION FOR IMPERFECT REWARDS

Abstract

Reward function is essential in reinforcement learning (RL), serving as the guiding signal to incentivize agents to solve given tasks, however, is also notoriously difficult to design. In many cases, only imperfect rewards are available, which inflicts substantial performance loss for RL agents. In this study, we propose a unified offline policy optimization approach, RGM (Reward Gap Minimization), which can smartly handle diverse types of imperfect rewards. RGM is formulated as a bi-level optimization problem: the upper layer optimizes a reward correction term that performs visitation distribution matching w.r.t. some expert data; the lower layer solves a pessimistic RL problem with the corrected rewards. By exploiting the duality of the lower layer, we derive a tractable algorithm that enables sampled-based learning without any online interactions. Comprehensive experiments demonstrate that RGM achieves superior performance to existing methods under diverse settings of imperfect rewards. Further, RGM can effectively correct wrong or inconsistent rewards against expert preference and retrieve useful information from biased rewards.

1. INTRODUCTION

Reward plays an imperative role in every reinforcement learning (RL) problem. It encodes the desired task behaviors, serving as a guiding signal to incentivize agents to learn and solve a given task. As widely recognized in RL studies, a desirable reward function should not only define the task the agent learns to solve, but also offers the "bread crumbs" that allow the agent to efficiently learn to solve the task (Abel et al., 2021; Singh et al., 2009; Sorg, 2011) . However, due to task complexity and human cognitive biases (Hadfield-Menell et al., 2017) , accurately describing a complex task using numerical rewards is often difficult or impossible (Abel et al., 2021; Li et al., 2019) . In most practical settings, the rewards are typically "imperfect" and hard to be fixed through reward tuning when online interactions are costly or dangerous (Zhan et al., 2022) . Such imperfect rewards are widespread in real-world applications and can appear in forms such as partially correct rewards, sparse rewards, mismatched rewards from other tasks, and completely incorrect rewards (see Figure 1 for an intuitive illustration ). These rewards either fail to incentivize agents to learn correct behaviors or cannot provide effective signals to speed up the learning process. Consequently, it is of great importance and practical value to devise a versatile method that can perform robust offline policy optimization under diverse settings of imperfect rewards. Reward shaping (Ng et al., 1999) is the most common approach to tackling imperfect rewards, but it requires tremendous human efforts and numerous online evaluations. Another possible avenue is imitation learning (IL) (Pomerleau, 1988; Kostrikov et al., 2019) or offline inverse reinforcement learning methods (IRL) (Jarboui & Perchet, 2021) , by directly imitating or deriving new rewards from expert behaviors. However, these methods heavily depend on the quantity and quality of expert demonstrations and offline datasets, which are often beyond reach in practice. Another key challenge is how to precisely measure the discrepancy between the given reward in the data and the true reward

availability

//github.com/Facebear

