IMPACT OF REPRESENTATION LEARNING IN LINEAR BANDITS

Abstract

We study how representation learning can improve the efficiency of bandit problems. We study the setting where we play T linear bandits with dimension d concurrently, and these T bandit tasks share a common k( d) dimensional linear representation. For the finite-action setting, we present a new algorithm which achieves O(T

√

kN + √ dkN T ) regret, where N is the number of rounds we play for each bandit. When T is sufficiently large, our algorithm significantly outperforms the naive algorithm (playing T bandits independently) that achieves O(T √ dN ) regret. We also provide an Ω(T √ kN + √ dkN T ) regret lower bound, showing that our algorithm is minimax-optimal up to poly-logarithmic factors. Furthermore, we extend our algorithm to the infinite-action setting and obtain a corresponding regret bound which demonstrates the benefit of representation learning in certain regimes. We also present experiments on synthetic and realworld data to illustrate our theoretical findings and demonstrate the effectiveness of our proposed algorithms.

1. INTRODUCTION

This paper investigates the benefit of using representation learning for sequential decision-making problems. Representation learning learns a joint low-dimensional embedding (feature extractor) from different but related tasks and then uses a simple function (often a linear one) on top of the embedding (Baxter, 2000; Caruana, 1997; Li et al., 2010) The mechanism behind is that since the tasks are related, we can extract the common information more efficiently than treating each task independently. Empirically, representation learning has become a popular approach for improving sample efficiency across various machine learning tasks (Bengio et al., 2013) . In particular, recently, representation learning has become increasingly more popular in sequential decision-making problems (Teh et al., 2017; Taylor & Stone, 2009; Lazaric & Restelli, 2011; Rusu et al., 2015; Liu et al., 2016; Parisotto et al., 2015; Higgins et al., 2017; Hessel et al., 2019; Arora et al., 2020; D'Eramo et al., 2020) . For example, many sequential decision-making tasks share the same environment but have different reward functions. Thus a natural approach is to learn a succinct representation that describes the environment and then make decisions for different tasks on top of the learned representation. While representation learning is already widely applied in sequential decision-making problems empirically, its theoretical foundation is still limited. One important problem remains open: When does representation learning provably improve efficiency of sequential decision-making problems? We take a step to characterize the benefit of representation learning in sequential decision-making problems. We tackle the above problem in the linear bandits setting, one of the most fundamental

