ON THE CRITICAL ROLE OF CONVENTIONS IN ADAPTIVE HUMAN-AI COLLABORATION

Abstract

Humans can quickly adapt to new partners in collaborative tasks (e.g. playing basketball), because they understand which fundamental skills of the task (e.g. how to dribble, how to shoot) carry over across new partners. Humans can also quickly adapt to similar tasks with the same partners by carrying over conventions that they have developed (e.g. raising hand signals pass the ball), without learning to coordinate from scratch. To collaborate seamlessly with humans, AI agents should adapt quickly to new partners and new tasks as well. However, current approaches have not attempted to distinguish between the complexities intrinsic to a task and the conventions used by a partner, and more generally there has been little focus on leveraging conventions for adapting to new settings. In this work, we propose a learning framework that teases apart rule-dependent representation from convention-dependent representation in a principled way. We show that, under some assumptions, our rule-dependent representation is a sufficient statistic of the distribution over best-response strategies across partners. Using this separation of representations, our agents are able to adapt quickly to new partners, and to coordinate with old partners on new tasks in a zero-shot manner. We experimentally validate our approach on three collaborative tasks varying in complexity: a contextual multi-armed bandit, a block placing task, and the card game Hanabi.

1. INTRODUCTION

Humans collaborate well together in complex tasks by adapting to each other through repeated interactions. What emerges from these repeated interactions is shared knowledge about the interaction history. We intuitively refer to this shared knowledge as conventions. Convention formation helps explain why teammates collaborate better than groups of strangers, and why friends develop lingo incomprehensible to outsiders. The notion of conventions has been studied in language (Clark & Wilkes-Gibbs, 1986; Clark, 1996; Hawkins et al., 2017; Khani et al., 2018) and also alluded to in more general multiagent collaborative tasks (Boutilier, 1996; Stone et al., 2010; Foerster et al., 2019; Carroll et al., 2019; Lerer & Peysakhovich, 2019; Hu et al., 2020) . For example, Foerster et al. ( 2019) trained agents to play the card game Hanabi, and noted the emergent convention that "hinting for red or yellow indicates that the newest card of the other player is playable". One established characterization of a convention that is commonly used (Boutilier, 1996; Hawkins et al., 2017; Lerer & Peysakhovich, 2019 ) is an arbitrary solution to a recurring coordination problem (Lewis, 1969) . A convention is thus one of many possible solutions that a group of partners happens to converge to. This is in contrast to problem solutions that are enforced by the rule constraints, and would have arisen no matter how the partners collaborated and what behavior they converged to. Success in a collaborative task typically involves learning both types of knowledge, which we will refer to as convention-dependent and rule-dependent behavior. The distinction between these two types of behavior has been studied extensively in the linguistic literature (Franke et al.; Brochhagen, 2020) . In this work, we focus on the less-explored setting of implicit communication, and provide a concrete approach to learn representations for these two different types of behavior. In the context of multi-agent or human-AI collaboration, we would like our AI agents to adapt quickly to new partners. AI agents should be able to flexibly adjust their partner-specific convention- dependent behavior while reusing the same rule-dependent behavior to simplify the learning problem -just like how humans quickly adapt when playing basketball with a new friend, without the need to re-learn the rules of the game. We would also like our AI agents to coordinate well on similar tasks when paired with the same partners -just like how humans can coordinate better when playing a new sport but with an old friend. Although many existing works similarly recognize conventions as an important factor of successful collaboration, they do not focus on separating conventions from rules of the task. This means that adapting to a new partner can be as difficult as learning a new task altogether. For example, existing techniques that emphasize modeling the partner's policy, such as theory of mind (Simon, 1995; Baker et al., 2017; Brooks & Szafir, 2019) or multi-agent reinforcement learning (Foerster et al., 2018) , attempt to model everything about the agent's belief of the partner's state and policies. Such belief modeling approaches very quickly become computationally intractable, as opposed to solely focusing on the relevant conventions developed with a partner. To address the challenges above, we propose a framework that explicitly separates conventiondependent representations and rule-dependent representations through repeated interactions with multiple partners. After many rounds of solving a task (e.g. playing basketball) with different partners, an AI agent can learn to distinguish between conventions formed with a specific partner (e.g. pointing down signals bounce pass) and intrinsic complexities of the task (e.g. dribbling). This enables us to leverage the representations separately for fast adaptation to new interactive settings. In the rest of this paper, we formalize the problem setting, and describe the underlying model of partners and tasks. Next, we present our framework for learning separations between rule and convention representations. We show that, under some conditions, our rule representation learns a sufficient statistic of the distribution over best-response strategies. We then run a study on humanhuman interactions to test if our hypothesis -that partners can carry over the same conventions across tasks -indeed holds for human subjects. Finally, we show the merits of our method on 3 collaborative tasks: contextual bandit, block placing, and a small scale version of Hanabi (Bard et al., 2020) .

2. RELATED WORK

Convention formation has been studied under the form of iterated reference games (Hawkins et al., 2017; 2019) , and language emergence (Mordatch & Abbeel, 2018; Lazaridou et al., 2017) . In these works, partners learn to reference objects more efficiently by forming conventions through repeated interactions. But in these tasks, there is little intrinsic task difficulty beyond breaking symmetries. In multi-agent reinforcement learning, techniques such as self-play, cross-training, or opponent learning (Foerster et al., 2018) have been used to observe emergent behavior in complex, physical settings (Nikolaidis & Shah, 2013; Liu et al., 2019; Baker et al., 2020) . In addition, convention formation has been shown in tasks like Hanabi (Foerster et al., 2019; Hu & Foerster, 2019 ), Overcooked (Carroll et al., 2019; Wang et al., 2020) , and negotiation tasks (Cao et al., 2018) , where agents learn both how to solve a task and how to coordinate with a partner. But, these works qualitatively analyze emergent conventions through post-hoc inspection of the learned policies, and do not learn representations



Figure 1: Partners form conventions in a collaborative task through repeated interactions. An AI agent can adapt quickly to conventions with new partners by reusing a shared rule representation g t and learning a new partner-specific convention representation g p . Certain collaborative tasks, such as friendly Rock-Paper-Scissors, are more convention-dependent than others, such as 4-player chess.

