TEMPORAL DIFFERENCE UNCERTAINTIES AS A SIGNAL FOR EXPLORATION Anonymous authors Paper under double-blind review

Abstract

An effective approach to exploration in reinforcement learning is to rely on an agent's uncertainty over the optimal policy, which can yield near-optimal exploration strategies in tabular settings. However, in non-tabular settings that involve function approximators, obtaining accurate uncertainty estimates is almost as challenging as the exploration problem itself. In this paper, we highlight that value estimates are easily biased and temporally inconsistent. In light of this, we propose a novel method for estimating uncertainty over the value function that relies on inducing a distribution over temporal difference errors. This exploration signal controls for state-action transitions so as to isolate uncertainty in value that is due to uncertainty over the agent's parameters. Because our measure of uncertainty conditions on state-action transitions, we cannot act on this measure directly. Instead, we incorporate it as an intrinsic reward and treat exploration as a separate learning problem, induced by the agent's temporal difference uncertainties. We introduce a distinct exploration policy that learns to collect data with high estimated uncertainty, which gives rise to a "curriculum" that smoothly changes throughout learning and vanishes in the limit of perfect value estimates. We evaluate our method on hardexploration tasks, including Deep Sea and Atari 2600 environments and find that our proposed form of exploration facilitates efficient exploration.

1. INTRODUCTION

Striking the right balance between exploration and exploitation is fundamental to the reinforcement learning problem. A common approach is to derive exploration from the policy being learned. Dithering strategies, such as -greedy exploration, render a reward-maximising policy stochastic around its reward maximising behaviour (Williams & Peng, 1991) . Other methods encourage higher entropy in the policy (Ziebart et al., 2008) , introduce an intrinsic reward (Singh et al., 2005) , or drive exploration by sampling from the agent's belief over the MDP (Strens, 2000) . While greedy or entropy-maximising policies cannot facilitate temporally extended exploration (Osband et al., 2013; 2016a) , the efficacy of intrinsic rewards depends crucially on how they relate to the extrinsic reward that comes from the environment (Burda et al., 2018a) . Typically, intrinsic rewards for exploration provide a bonus for visiting novel states (e.g Bellemare et al., 2016) or visiting states where the agent cannot predict future transitions (e.g Pathak et al., 2017; Burda et al., 2018a) . Such approaches can facilitate learning an optimal policy, but they can also fail entirely in large environments as they prioritise novelty over rewards (Burda et al., 2018b) . Methods based on the agent's uncertainty over the optimal policy explicitly trade off exploration and exploitation (Kearns & Singh, 2002) . Posterior Sampling for Reinforcement Learning (PSRL; Strens, 2000; Osband et al., 2013 ) is one such approach, which models a distribution over Markov Decision Processes (MDPs). While PSRL is near-optimal in tabular settings (Osband et al., 2013; 2016b) , it cannot be easily scaled to complex problems that require function approximators. Prior work has attempted to overcome this by instead directly estimating the agent's uncertainty over the policy's value function (Osband et al., 2016a; Moerland et al., 2017; Osband et al., 2019; O'Donoghue et al., 2018; Janz et al., 2019) . While these approaches can scale posterior sampling to complex problems and nonlinear function approximators, estimating uncertainty over value functions introduces issues that can cause a bias in the posterior distribution (Janz et al., 2019) . In response to these challenges, we introduce Temporal Difference Uncertainties (TDU), which derives an intrinsic reward from the agent's uncertainty over the value function. Concretely, TDU relies on the Bootstrapped DQN (Osband et al., 2016a) and separates exploration and reward-maximising behaviour into two separate policies that bootstrap from a shared replay buffer. This separation allows us to derive an exploration signal for the exploratory policy from estimates of uncertainty of the reward-maximising policy. Thus, TDU encourages exploration to collect data with high model uncertainty over reward-maximising behaviour, which is made possible by treating exploration as a separate learning problem. In contrast to prior works that directly estimate value function uncertainty, we estimate uncertainty over temporal difference (TD) errors. By conditioning on observed stateaction transitions, TDU controls for environment uncertainty and provides an exploration signal only insofar as there is model uncertainty. We demonstrate that TDU can facilitate efficient exploration in challenging exploration problems such as Deep Sea and Montezuma's Revenge.

2. ESTIMATING VALUE FUNCTION UNCERTAINTY IS HARD

We begin by highlighting that estimating uncertainty over the value function can suffer from bias that is very hard to overcome with typical approaches (see also Janz et al., 2019) . Our analysis shows that biased estimates arise because uncertainty estimates require an integration over unknown future state visitations. This requires tremendous model capacity and is in general infeasible. Our results show that we cannot escape a bias in general, but we can take steps to mitigate it by conditioning on an observed trajectory. Doing so removes some uncertainty over future state-visitations and we show in Section 3 that it can result in a substantially smaller bias. We consider a Markov Decision Process (S, A, P, R, γ) for some given state space (S), action space (A), transition dynamics (P), reward function (R) and discount factor (γ). For a given (deterministic) policy π : S → A, the action value function is defined as the expected cumulative reward under the policy starting from state s with action a: Q π (s, a) := E π ∞ t=0 γ t r t+1 s 0 = s, a 0 = a = E r∼R(s,a) s ∼P(s,a) [r + γQ π (s , π(s ))] , where t index time and the expectation E π is with respect to realised rewards r sampled under the policy π; the right-hand side characterises Q recursively under the Bellman equation. The actionvalue function Q π is estimated under a function approximator Q θ parameterised by θ. Uncertainty over Q π is expressed by placing a distribution over the parameters of the function approximator, p(θ). We overload notation slightly and write p(θ) to denote the probability density function p θ over a random variable θ. Further, we denote by θ ∼ p(θ) a random sample θ from the distribution defined by p θ . Methods that rely on posterior sampling under function approximators assume that the induced distribution, p(Q θ ), is an accurate estimate of the agent's uncertainty over its value function, p(Q π ), so that sampling Q θ ∼ p(Q θ ) is approximately equivalent to sampling from Q π ∼ p(Q π ). For this to hold, the moments of p(Q θ ) at each state-action pair (s, a) must correspond to the expected moments in future states. In particular, moments of p(Q π ) must satisfy a Bellman Equation akin to Eq. 1 (O'Donoghue et al., 2018) . We focus on the mean (E) and variance (V): E θ [Q θ (s, a)] = E θ [E r,s [r + γQ θ (s , π(s ))]] , V θ [Q θ (s, a)] = V θ [E r,s [r + γQ θ (s , π(s ))]]. (3) If E θ [Q θ ] and V θ [Q θ ] fail to satisfy these conditions, the estimates of E[Q π ] and V[Q π ] are biased, causing a bias in exploration under posterior sampling from p(Q θ ). Formally, the agent's uncertainty over p(Q) implies uncertainty over the MDP (Strens, 2000) . Given a belief over the MDP, i.e., a distribution p(M ), we can associate each M ∼ p(M ) with a distinct value function Q M π . Lemma 1 below shows that, for p(θ) to be interpreted as representing some p(M ) by push-forward to p(Q θ ), the induced moments must match under the Bellman Equation. 



If E θ [Q θ ] and V θ [Q θ ]fail to satisfy Eqs. 2 and 3, respectively, they are biased estimators of E M Q M π and V M Q M π for any choice of p(M ).

