LEARNING HETEROGENEOUS INTERACTION STRENGTHS BY TRAJECTORY PREDICTION WITH GRAPH NEURAL NETWORK

Abstract

Dynamical systems with interacting agents are universal in nature, commonly modeled by a graph of relationships between their constituents. Recently, various works have been presented to tackle the problem of inferring those relationships from the system trajectories via deep neural networks, but most of the studies assume binary or discrete types of interactions for simplicity. In the real world, the interaction kernels often involve continuous interaction strengths, which cannot be accurately approximated by discrete relations. In this work, we propose the relational attentive inference network (RAIN) to infer continuously weighted interaction graphs without any ground-truth interaction strengths. Our model employs a novel pairwise attention (PA) mechanism to refine the trajectory representations and a graph transformer to extract heterogeneous interaction weights for each pair of agents. We show that our RAIN model with the PA mechanism accurately infers continuous interaction strengths for simulated physical systems in an unsupervised manner. Further, RAIN with PA successfully predicts trajectories from motion capture data with an interpretable interaction graph, demonstrating the virtue of modeling unknown dynamics with continuous weights.

1. INTRODUCTION

Dynamical systems with interactions provide a fundamental model for a myriad of academic fields, yet finding out the form and strength of interactions remains an open problem due to its inherent degeneracy and complexity. Although it is crucial to identify the interaction graph of a complex system for understanding its dynamics, disentangling individual interactions from trajectory data without any ground-truth labels is a notoriously hard inverse problem. Further, if the interactions are heterogeneous and coupled with continuous strength constants, the interaction graph is called weighted and the inference became much harder with increased level of degeneracies. In this work, we assume the dynamical system with N objects (or agents), and their (discretized) trajectories x 1 , x 2 , . . . , x N from timestep t = 0 to T are given. If the system has an interaction kernel Q(x i , x j ) and the dynamics are governed by a form of ẋi = j =i k ij Q(x i , x j ) with some variable k ij , which is prevalent in nature and physical system, we call k ij as an interaction strength between the object i and j. With proper normalization, we can always regard 0 ≤ k ij ≤ 1. In general, k ij may have continuous values and forms a weighted interaction graph, which can be expressed in the form of a connectivity matrix K; a conventional adjacency matrix with continuous-valued entries of k ij . Hence, the problem is inferring continuous adjacency matrix K from trajectories x alone. In the current work, we propose a neural network called Relational Attentive Inference Network (RAIN) to address the problem of inferring weighted interaction graphs from multivariate trajectory data in an unsupervised manner. RAIN infers the interaction strength between two agents from previous trajectories by learning the attentive weight while simultaneously learning the unknown dynamics of the system and thus is able to precisely predict the future trajectories. Our model employs the attention mechanism twice: once for the construction of pairwise trajectory embedding 2017), RAIN aims to infer the absolute interaction strength that governs the system dynamics by employing attention module with multilayer perceptron (MLP) and sigmoid activation. By comparing the inferred interaction strengths of simulated physical systems with ground-truth values that are not provided at the training stage, we verify that RAIN is capable of inferring both system dynamics and weighted interaction graphs solely from multivariate data. We further show that RAIN outperforms discrete baselines on real-world motion capture data, representing a system in which we cannot be certain whether a continuous form of interaction strengths even exists. In this way, we demonstrate that the rich flexibility and expressibility of the continuous modeling of interaction strengths are crucial for the accurate prediction of the future dynamics of an unknown empirical system.

2. RELATED STUDIES

There has been a long history and substantial amount of work on both inferring the network topology and the nonlinear correlation between interacting constituents from data Casadiego et al. ( 2017 The key strength of these approaches comes from the fact that a neural network enables relatively free-form modeling of the system. One influential work in this direction, neural relational inference (NRI) Kipf et al. (2018) , explicitly infers edges by predicting the future trajectories of the given system. But previous studies for extracting interaction graphs with neural networks Kipf et al. (2018); Webb et al. (2019) ; Graber & Schwing (2020) mainly focused on inferring edges with discrete edge types, which means that they are incapable of distinguishing the edges of different interaction strengths with the same type. Considering the common occurrence of such heterogeneous interaction strengths throughout diverse systems, the assumption of discrete edge types severely limits the expressibility of the model.



Figure 1: Overview of system formulation and RAIN architecture. RAIN encodes each agent's trajectory with an LSTM encoder and applies pairwise attention (PA) to the hidden states for constructing a pair of embeddings for each agent pair. Then the graph attentive module extracts the interaction strength from a pair of embeddings in the form of an attention weight with an MLP, A θ . The decoder module finally predicts the future trajectories of each agent with an LSTM decoder, but here, each prediction can only employ the weighted information from other agents. This restriction on information induces the attention weights in the learning process to properly reflect the strengths of the connections.

); Ching & Tam (2017); Chang et al. (2018); Shi et al. (2020); Ha & Jeong (2021); Fujii et al. (2021), along with the development of various measures to capture the relation between constituents (e.g., Pearson correlations, mutual information, transfer entropy, Granger causality, and variants thereof Schreiber (2000)). Many of these inferences focus on specific systems with the necessity for a model prior, such as domain knowledge of the agent characteristics, proper basis construction, and detailed assumptions on the system dynamics. Recently, by phenomenal advances in machine learning, adopting a neural network as a key component of the interaction inference has gained attention from researchers Veličković et al. (2017); Kipf et al. (2018); Zhang et al. (2019).

