ROBUST MULTI-AGENT REINFORCEMENT LEARNING AGAINST ADVERSARIES ON OBSERVATION

Abstract

With the broad applications of deep learning, such as image classification, it is becoming increasingly essential to tackle the vulnerability of neural networks when facing adversarial attacks, which have been widely studied recently. In the cooperative multi-agent reinforcement learning field, which has also shown potential in real-life domains, little work focuses on the problem of adversarial attacks. However, adversarial attacks on observations that can undermine the coordination among agents are likely to occur in actual deployment. This paper proposes a training framework that progressively generates adversarial attacks on agents' observations to help agents learn a robust cooperative policy. One attacker makes decisions on a hybrid action space that it first chooses an agent to attack and then outputs the perturbation vector. The victim policy is then trained against the attackers. Experimental results show that our generated adversarial attacks are diverse enough to improve the agents' robustness against possible disturbances.

1. INTRODUCTION

The target of reinforcement learning (RL) is to learn the policies in complex environments to get long-term rewards. The technique of MARL is introduced to adapt RL algorithms to multi-agent systems. In recent years, Multi-Agent Reinforcement Learning (MARL) has attracted widespread attention (Du & Ding, 2021) and has been applied in numerous domains, including sensor networks (Zhang & Lesser, 2011) , autonomous vehicle teams (Zhou et al., 2020) , and traffic signal control (Du et al., 2021) . However, neural networks are proven to be vulnerable to adversarial perturbations (Huang et al., 2017) , and some small perturbations may cause the deep RL policy to fail. Therefore, it is of great significance to train a robust policy to help deploy current RL algorithms to real-life applications. In single-agent reinforcement learning, some research studies enhance policy robustness by using adversarial learning and achieve good results. Pinto et al. ( 2017) propose a method that jointly trains a pair of agents, including a protagonist and an adversary, and the protagonist learns to fulfill the original task goals while being robust to the disruptions generated by its adversary. Pattanaik et al. (2018) show that deep RL can be fooled easily and train an RL agent under naive attacks to improve its robustness. Zhang et al. (2021) propose a framework of alternating training with learned adversaries, which trains an adversary online with the agent using a policy gradient following the optimal adversarial attack framework. However, such studies are rare in cooperative MARL, and current works mainly focus on the setting where teammates may betray or agents' actions may be maliciously modified (Li et al., 2019; Phan et al., 2021; 2020; Hu & Zhang, 2022) . However, in real-life applications of cooperative MARL, the most vulnerable parts of the agents are the sensors that can be disturbed by noise or jamming attacks. Agents are closely related to each other when cooperating to accomplish tasks, and even a small perturbation on one agent's observation from the sensors can make it deflect from coordination and cause the whole multi-agent system to fail. Therefore, how to design an algorithm that can obtain a policy that is robust on observations in cooperative MARL is noteworthy. This paper proposes a robust MARL training framework for observation perturbations, RObust Multi-agent reinforcement learning against Adversaries on Observation (ROMAO). Our contributions can be summarized as follows:

