FUNCTIONAL RELATION FIELD: A MODEL-AGNOSTIC FRAMEWORK FOR MULTIVARIATE TIME SERIES FORE-CASTING

Abstract

In multivariate time series forecasting, the most popular strategy for modeling the relationship between multiple time series is the construction of graph, where each time series is represented as a node and related nodes are connected by edges, i.e. spatial-temporal graph neural networks. The graph structure is either given apriori or learned based the similarity between nodes. However, the relationship between multiple time series is typically complicated, for instance, the sum of outflows from upstream nodes may be equal to the inflows of downstream nodes. Such relations widely exist in many real-world multivariate time series forecasting scenarios, yet are far from well studied. In these cases, graph might only be a crude description on the dependency between nodes. To this end, we explore a new framework to model the inter-node relationship in a more precise way based our proposed inductive bias for graphs, Functional Relation Field, where a group of functions parameterized by neural networks are learned to characterize the dependency between multiple time series. These learned functions are versatile: they can then be used to discover the underlying graph structure by identifying the most relevant neighbors of the target node; and on the other hand, the learned functions will form a "field" where the nodes in the backbone prediction networks are enforced to satisfy the constraints defined by these functions. The experiment is conducted on one toy dataset to show our approach can well recover the true constraint relationship between nodes. And two real-world MiniApp calling traffic and road networks datasets are also considered with various different backbone networks. Results show that the prediction error can be reduced remarkably with the aid of the proposed functional relation field framework.

1. INTRODUCTION

Multivariate time series forecasting has surged recently due to its strong expressiveness of the spatio-temporal dependence among the data and its enormous popularity in vast application areas, such as the prediction of urban traffic, computer network flow, cloud micro-services calling flow, and rigid body motion, to name a few (Li et al., 2018; Yu et al., 2018; Bai et al., 2020; Yan et al., 2018; Liu et al., 2020) . The most popular and straightforward strategy for modeling the relationship between multiple time series is the introduction of graph, where each time series is represented as a node and related nodes are connected by edges. This particular inductive bias for multivariate time series prediction results in the so called spatial-temporal graph neural networks (Yu et al., 2018) . The graph structure is either given apriori (e.g. in traffic flow prediction, each road as a node has connected roads forming the graph.) or learned based the similarity between nodes (Yu et al., 2019; Bai et al., 2020; Shang et al., 2021) . However, in practice, the relationship between multiple time series is typically complicated. For instance, there often exist constraints among the nodes, ranging from the equality between the inflow and the outflow for a node in a traffic network to the geometric constraints of the rigid body motion. Such relations widely exist in many real-world multivariate time series forecasting scenarios, yet are far from well studied. In these cases, graph might not be sufficient for characterizing the dependency between nodes. As a remedy, in this work, we explore a new framework to model the inter-node relationship in a more precise manner than graph, Functional Relation Field (FRF), where a group of functions parameterized by neural networks are learned to characterize the dependency between multiple time series explicitly. These learned functions are versatile: first they can then be used to discover the underlying graph structure by identifying the most relevant neighbors of the target node; and on the other hand, the learned functions will form a "field" where the nodes in the backbone prediction networks are further enforced to satisfy the constraints defined by these functions. As illustrated in Fig. 1 , the left panel shows the traditional graph neural networks assuming similar time series have edge connections, while our framework on the right panel models the dependency between nodes through a functional relationship, e.g. a linear form to enforce the constraints between the flows of target and dependent nodes. In our framework, we mainly solve the following two issues: (i) How to learn the functional field? We need to select the dependent nodes that have a relationship with the target node, and express the constraint in a functional form; (ii) How to guarantee the constraints satisfaction? The (functional) constraints relationship should be maintained in the predicted output in both training and test process. To address these issues, we propose a two-stage approach that can discover the functional relations (i.e. constraints) from data and further integrate the constraints seamlessly when forecasting the multivariate time series. Specifically, we first train a neural network with a selected target node as its output and all the other nodes as dependent variables (i.e. the input of this neural network), and identify the most relevant dependent nodes based on this trained network. We then re-train it to learn the relationship among the target and the discovered relevant nodes. Next, we incorporate these functional constraints into the network backbones by imposing them to the predicted output during both training and test process. More precisely, the output of the network could be guaranteed to satisfy the constraints by utilizing the constraint-satisfied transformation and loss minimization. We compare the proposed approach with SVM, fully connected networks, fully connected LSTM, and five backbone models (i.e., STGCN (Yu et al., 2018) , AGCRN (Bai et al., 2020) , Autoformer (Wu et al., 2021 ), FEDformer (Zhou et al., 2022 ), SCINet (Liu et al., 2022) ). Experimental results show that our approach significantly improves the performance over the original network backbones and other baseline models.

RELATED WORK

Univariate time series forecasting. Recently, much research focuses on time series forecasting with deep learning models due to their powerful representational capability and prediction performance, including feed-forward neural network, RNN (Rumelhart, 1986) and its variants LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014) . The transformer architecture and its variants (Vaswani et al., 2017; Simm et al., 2020; Zhou et al., 2021; Child et al., 2019; Lim et al., 2020; Li et al., 2019; Wu et al., 2021; Zhou et al., 2022) Multivariate time series forecasting. Spatio-temporal graph neural networks (Yu et al., 2018; Chen et al., 2019; Pan et al., 2021; Li et al., 2020) have been proposed to model the spatial correlation and temporal dependency in multivariate time-series. Apart from capturing the temporal dependence, these methods further model the spatial dependence among all time series via graph neural networks, leveraging the information from the neighboring time series to help forecasting the target one. It is well known that an informative graph structure is important to the graph time series forecasting. Therefore, many algorithms (Bai et al., 2020; Seo et al., 2016; Shang et al., 2021) were proposed to discovery the underlying graph structure. AGCRN (Bai et al., 2020) assumed the graph structure



Figure 1: Comparison between traditional graphbased modeling and our approach. (a) Graph structure learning with time-series similarity; (b) Functional relation field, modeling inter-node functional relationship in a linear form, where D is the target node and {A, B, C} represent dependent nodes.

also made much progress on univariate time-series forecasting on learning long-range dependence. In order to model the trend and seasonality of time series in an interpretable way, N-beats(Oreshkin et al., 2020)  network that stacked very deep fullconnection network based on backward and forward residual links has improved the multi-horizon prediction accuracy significantly. Moreover, DeepAR(Salinas et al., 2020)  and Deep State-Space Model (DSSM)(Rangapuram et al., 2018)  stack multi-layer LSTM network to generate parameters of one-step-ahead Gaussian predictive distributions for multi-horizon prediction.

