ONLINE REINFORCEMENT LEARNING VIA POSTERIOR SAMPLING OF POLICY Anonymous authors Paper under double-blind review

Abstract

We propose a Reward-Weighted Posterior Sampling of Policy (RWPSP) algorithm to tackle the classic trade-off problem between exploration and exploitation under finite Markov decision processes (MDPs). The Thompson sampling method so far has only considered posterior sampling over transition probabilities, which is hard to gain the globally sub-optimal rewards. RWPSP runs posterior sampling over stationary policy distributions instead of transition probabilities, and meanwhile keeps transition probabilities updated. Particularly, we leverage both relevant count functions and reward-weighting to online update the policy posterior, aiming to balance between local and long-term policy distributions for a globally near-optimal game value. Theoretically, we establish a bound of Õ(Γ T /S 2 ) 1 on the total regret in time horizon T with Γ/S 2 < D SA satisfied in general, where S and A represents the sizes of state and action spaces, respectively, D the diameter. This matches the best regret bound thus far for MDPs. Experimental results corroborate our theoretical results and show the advantage of our algorithm over baselines in terms of efficiency.

1. INTRODUCTION

Online reinforcement learning (Wei et al., 2017) addresses the problem of learning and planning in real-time sequential decision making systems with the interacting environment partially observed or fully observed. The decision maker tries to maximize the cumulative reward during the interaction with the environment, which however inevitably leads to the trade-off between exploration and exploitation. Many attempts have been made to mitigate such dilemma by improving underlying regret bounds (Zhang et al., 2020b) (Ménard et al., 2021) (Zhang et al., 2021b) (Zhang et al., 2022 )(Agrawal et al., 2021) . Trade-off between exploration and exploitation has been studied extensively in various scenarios. The goal of exploration is to find as much information as possible of the environment, while the exploitation process aims to maximize the long-term total reward based on the exploited part of the environment. To handle the trade-off problem, one popular way is to use the naive exploration method such as adaptive ϵ-greedy exploration (Tokic, 2010) . The method adjusts the exploration parameter adaptively, depending on the temporal-difference (TD) error observed from the value function. Optimistic initialisation methods have also been studied in factored MDPs (Szita & Lörincz, 2009; Brafman & Tennenholtz, 2003) . They encourage systematic exploration in the early stage. Another common way is to use the optimism in the face of uncertainty (OFU) principle (Lai & Robbins, 1985) , where the agent constructs confidence sets to search for the optimistic parameters associated with the maximum reward. Thompson sampling, as an OFU-based approach, was originally presented for stochastic bandit scenarios (Thompson, 1933) . It has been applied in various MDPs contexts (Osband et al., 2013; Agrawal & Goyal, 2012) since it can achieve tighter bounds (Ding et al., 2021; Oh & Iyengar, 2019; Moradipari et al., 2019) and better compatibility with other structures in both theory and practice (Chapelle & Li, 2011; Zhang et al., 2021a; Agrawal & Goyal, 2013) . It has also achieved great performance on contextual bandit problems(Agrawal & Jia, 2017)(Osband & Van Roy, 2017) (Osband et al., 2019) .The general optimistic algorithms require to solve all MDPs lying within the confident sets, while Thompson sampling-based algorithms only need to solve the sampled MDPs to achieve similar results (Russo & Van Roy, 2014) . Thompson sampling offers speedup on one hand, and results in biased estimates of the transition matrix on the other hand. This paper addresses the trade-off problem between exploration and exploitation in finite MDPs. We propose a reward-weighted posterior sampling of policy (RWPSP) algorithm that samples posterior policy distributions rather than posterior transition distributions, which optimizes the long-term policy probability distribution. While updating posterior policy distribution, we use the count functions of the state-action pairs to capture the importance of each sampled episode. This way, we manage to optimize the policy distribution in time horizon T and achieve the total regret bound of Õ(Γ • We propose a reward-weighted posterior sampling of policy (RWPSP) algorithm that strikes a balance between the posterior projection of the long-term policy and the local policy. √ T /S 2 ) with Γ/S 2 < D √ SA, • RWPSP is the first posterior sampling method that samples posterior policy distributions while Bayesian updating transition probabilities. It achieves a regret bound of Õ( Γ √ T S 2 ), where Γ/S 2 < D

√

SA. We show that the total regret bound is less than the state-of-the-art, i.e., D SAT , to the best of our knowledge. • We conduct experimental studies to verify our theoretical results and demonstrate that our RWPSP algorithm outperforms other online learning methods in complex MDP environments.

2. RELATED WORK

Regret Bound Analysis In the finite-horizon setting, most of the Thompson Sampling-based algorithms follow a model-based approach (Abbasi Yadkori et al., 2013; Xu & Tewari, 2020; Auer et al., 2008; Fruit et al., 2020; Dong et al., 2020; Agarwal et al., 2020) , as model-based reinforcement learning methods are required to approximate the optimal transition matrix of a MDP. In Xu & Tewari (2020), non-episodic factored Markov decision processes are sampled using extreme transition dynamics which encourages visiting new states in order to minimize regret. Although various approaches had been used to minimize the regret bound, current methods still minimize the regret bound by updating the transition matrix. A good comparison can be found in Zhang et al. (2021c); Wei et al. (2020) among existing Thompson sampling based methods. In contrast to existing works with a focus on posterior sampling over transition matrices, our work only considers posterior sampling over policy distributions in a finite-horizon MDP. The transition probabilities will be updated based on the real trajectory. On the other hand, while existing model-free methods have not yet achieved the state-of-art regret bound (Jin et al., 2018; Strehl et al., 2006) , some of them improved the total regret bound (Zhang et al., 2020a) . Intrinsic Reward Shaping Intrinsic reward shaping was first introduced in 1999 (Ng et al., 1999), which is a generic idea to guide the policy iteration with intrinsic reward. Count-based methods are then proposed to reach nearly state-of-the-art performance on a high-dimensional environment (Tang et al., 2017) . Intrinsic reward is also used in Du et al. (2019) to compute a distinct proxy critic for the agent to guide the update of its individual policy. In order to shape the reward during the policy iteration, we adopt the reward-weighted update to verify the intrinsic reward. Count functions of states and/or actions are usually used in the exploration process of an agent to help build the intrinsic reward (Tang et al., 2017; Bellemare et al., 2016; Burda et al., 2018) . In our algorithm, we consider the count function as the posterior projection of the intrinsic reward, and then use the generated reward to update the posterior distribution. The previous methods mainly focus on the instantaneous rewards generated from the exploration process, while our method uses a reward-weighted count function to generate long-term rewards which can guide the policy towards a globally optimal value.



The symbol Õ hides logarithmic factors.



where S and A represent the sizes of the state and action spaces respectively. D is the diameter of the finite MDP. In addition, we propose a new Bayesian method to update transition probabilities which also achieves a state-of-art regret bound. In comparison, existing model-based methods like Upper Confidence Stochastic Game Algorithm(UCSG) achieve a regret bound of Õ 3 √ DS 2 AT 2 on stochastic MDPs (Wei et al., 2017), while model-free methods like optimistic Q-learning achieve a regret bound of Õ T 2/3 under infinite-horizon average reward MDPs (Wei et al., 2020). To summarize, this work makes the following contributions:

