INTERACTIVE VISUALIZATION FOR DEBUGGING RL

Abstract

Visualization tools for supervised learning (SL) allow users to interpret, introspect, and gain an intuition for the successes and failures of their models. While reinforcement learning (RL) practitioners ask many of the same questions while debugging agent policies, existing tools aren't a great fit for the RL setting as these tools address challenges typically found in the SL regime. Whereas SL involves a static dataset, RL often entails collecting new data in challenging environments with partial observability, stochasticity, and non-stationary data distributions. This necessitates the creation of alternate visual interfaces to help us better understand agent policies trained using RL. In this work, we design and implement an interactive visualization tool for debugging and interpreting RL. Our system 1 identifies and addresses important aspects missing from existing tools such as (1) visualizing alternate state representations (different from those seen by the agent) that researchers could use while debugging RL policies; (2) interactive interfaces tailored to metadata stored while training RL agents (3) a conducive workflow designed around RL policy debugging. We provide an example workflow of how this system could be used, along with ideas for future extensions.

1. INTRODUCTION

Machine learning systems have made impressive advances due to their ability to learn high dimensional models from large amounts of data (LeCun et al., 2015) . However, high dimensional models are hard to understand and trust (Doshi-Velez & Kim, 2017) . Many tools exist for addressing this challenge in the supervised learning setting, which find usage in tracking metrics (Abadi et al., 2015; Satyanarayan et al., 2017) , generating graphs of model internals (Wongsuphasawat et al., 2018), and visualizing embeddings (van der Maaten & Hinton, 2008) . However, there is no corresponding set of tools for the reinforcement learning setting. At first glance, it appears we may repurpose existing tools for this task. However, we quickly run into limitations, that arise due to the intent with which these tools were designed. Reinforcement learning (RL) is a more interactive science (Neftci & Averbeck, 2019) compared to supervised learning, due to a stronger feedback loop between the researcher and the agent. Whereas supervised learning involves a static dataset, RL often entails collecting new data. To fully understand an RL algorithm, we must understand the effect it has on the data collected. Note that in supervised learning, the learned model has no effect on a fixed dataset. Visualization systems are important for overcoming these challenges. At their core visualization systems, consist of two components: representation and interaction. Representation is concerned with how data is mapped to a representation and then rendered. Interaction is concerned with the dialog between the user and the system as the user explores the data to uncover insights (Yi et al., 2007) . Though appearing to be disparate, these two processes have a symbiotic influence on each other. The tools we use for representation affect how we interact with the system, and our interaction affects the representations that we create. Thus, while designing visualization systems, it is important to think about the application domain from which the data originates, in this case, reinforcement learning. Using existing tools we can plot descriptive metrics such as cumulative reward, TD-error, and action values, to name a few. However, it is harder to pose and easily answer questions such as: -How does the agent state-visitation distribution change as training progresses? -What effect do noteworthy, influential states have on the policy? However, we argue that there exists a large space of unexplored interfaces that could help aid the process of debugging RL algorithms and trained policies. We explore one such solution that is designed around the spatial and temporal aspects of training RL agents. This approach might help increase understanding, interpretability, and thereby serve as a complement to tools in the existing ecosystem. -Are there repetitive patterns across space and time that result in the observed agent behavior? These are far from an exhaustive list of questions that a researcher may pose while training agent policies, but are chosen to illustrate the limitations created by our current set of tools that prevent us from being able to easily answer such questions. This paper describes our attempt at constructing Vizarelfoot_1 , an interactive visualization system to help interpret RL algorithms, debug RL policies, and help RL researchers pose and answer questions of this nature. Towards these goals, we identify features that an interactive system for interpretable reinforcement learning should encapsulate and build a prototype of these ideas. We complement this by providing a walkthrough example of how this system could fit into the RL debugging workflow and be used in a real scenario to debug a policy.

2. RELATED WORK

As we have argued in the introduction, existing visualization tools for machine learning primarily focus on the supervised learning setting. However, the process of designing and debugging RL algorithms requires a different set of tools, that can complement the strengths and overcome the weaknesses of offerings in the current ecosystem. In the rest of this section, we highlight aspects of prior work upon which our system builds. To the best of our knowledge, there do not exist visualization systems built for interpretable reinforcement learning that effectively address the broader goals we have identified. There exists prior work, aspects of which are relevant to features which the current system encapsulates, that we now detail. Visual Interpretability Related work for increasing understanding in machine learning models using visual explanations includes: feature visualization in neural networks (Olah et al., 2017; Simonyan et al., 2014; Zeiler & Fergus, 2013) , visual analysis tools for variants of machine learning models (Strobelt et al., 2017; Kahng et al., 2017; Kapoor et al., 2010; Krause et al., 2016; Yosinski et al.) , treating existing methods as composable building blocks for user interfaces (Olah et al., 2018) , and visualization techniques for increasing explainability in reinforcement learning (Rupprecht et al., 2020; Atrey et al., 2020; McGregor et al., 2015) Explaining agent behavior There exists related work that tries to explain agent behavior. Amir & Amir summarize agent behavior by displaying important trajectories. van der Waa et al. ( 2018) introduce a method to provide contrastive explanations between user derived and agent learned policies. Huang et al. (2017) show maximally informative examples to guide the user towards understanding



An interactive (anonymized) demo of the system can be found at https://vizarel-demo. herokuapp.com Vizarel is a portmanteau of visualization + reinforcement learning.



Figure 1: Tool Comparison Contrasting between a representative tool for debugging RL in the existing ecosystem (L), and Vizarel (R), highlights the difference in design intent between both systems. The former was designed for the supervised learning setting, and has shown promise for use in reinforcement learning.However, we argue that there exists a large space of unexplored interfaces that could help aid the process of debugging RL algorithms and trained policies. We explore one such solution that is designed around the spatial and temporal aspects of training RL agents. This approach might help increase understanding, interpretability, and thereby serve as a complement to tools in the existing ecosystem.

