DYNAMIC GRAPH REPRESENTATION LEARNING WITH FOURIER TEMPORAL STATE EMBEDDING

Abstract

Static graph representation learning has been applied in many tasks over the years thanks to the invention of unsupervised graph embedding methods and more recently, graph neural networks (GNNs). However, in many cases, we are to handle dynamic graphs where the structures of graphs and labels of the nodes are evolving steadily with time. This has posed a great challenge to existing methods in time and memory efficiency. In this work, we present a new method named Fourier Temporal State Embedding (FTSE) to address the temporal information in dynamic graph representation learning. FTSE offered time and memory-efficient solution through applying signal processing techniques to the temporal graph signals. We paired the Fourier Transform with an efficient edge network and provided a new prototype of modeling dynamic graph evolution with high precision. FTSE can also prevent the 'history explosion' that exists in sequential models. The empirical study shows that our proposed approach achieves significantly better performance than previous approaches on public datasets across multiple tasks.

1. INTRODUCTION

Graph Representation Learning learns the graphs with low-dimensional vectors at nodes and graphs level. (Perozzi et al., 2014; Tang et al., 2015; Wang et al., 2016; Cao et al., 2015; Ou et al., 2016) In recent years, deep neural networks (DNNs) are extended to process graphical data and have been utilized in a plethora of real-time cases. (Estrach et al., 2014; Duvenaud et al., 2015; Defferrard et al., 2016; Li et al., 2015; Gilmer et al., 2017; Kipf & Welling, 2016; Hamilton et al., 2017; Jin et al., 2017; Chen et al., 2018; Veličković et al., 2018; Gao & Ji, 2019) . One of the most popular networks is Graph Convolution Neural Networks (GCNs) which originated from Spectral Graph Theory but developed into spatial-based varieties. GCNs are natural extensions of Convolution Neural Networks (CNNs) which has been widely studied and used in many applications in different fields of research. Traditional GCNs have achieved commendable performance on static graphs. However, many applications involved dynamic graphs where nodes and edges evolved steadily over time. For example, a social network is updated on a day-to-day basis as people developed new friends. The dynamic graph represented users' evolving social relationships. In financial networks, transactions between nodes naturally adopt time-stamps as temporal features. Transactions of different nature may perform differently in a financial network where our main focus is to find malicious parties. Learning the evolving nature of graphs is an important task where we predict future graphical features and classify nodes based on their past behaviors. Learning evolving graphs poses great challenges on traditional GCN as temporal features can not be easily incorporated into learning algorithms. The simple way of concatenating GCNs with RNNs is straight forward in handling dynamic graphs, but it suffered from many drawbacks. We can summarize them as three folds: Firstly, the embedding vector of each node is not static and will be evolving with time. Models need to be capable of capturing the evolving nature. Secondly, the memory and computation cost for batch training is huge to keep multiple graphs from different timesteps in the memory at the same time. Finally, the large number of timesteps within a single batch brings difficulties to high precision modeling. There is also a focus on using Deep Neural Networks to generate the graph embedding recently (Trivedi et al., 2018; Pareja et al., 2020; Xu et al., 2020) as another direction compared with traditional unsupervised dynamic graph embedding approaches (Nguyen et al., 2018; Li et al., 2018; Goyal et al., 2018; 2020) . Existing methods normally utilize Sequential Models (e.g.Recurrent Neural Networks(RNNs)) to learn temporal features. However, as the graph is a non-linear data structure, sequential model-based approaches are memory-costly to train and evaluate with the information from the whole graph as input. Meanwhile, pure GCN approaches built for static graphs are inefficient in capturing evolving features. Some approaches combining GCNs with RNNs (Trivedi et al., 2018; Pareja et al., 2020) are costly to evaluate due to the high time complexity induced by repeated Graph Convolutions as well as high space complexity caused by a large number of network parameters. Meanwhile, the RNN-based method could only see through a fixed amount of history timesteps in training, which makes the prediction imprecise. We introduced Fourier temporal state embedding (FTSE) to address the above problem. Instead of using sequential models to model the evolving nature of edges, we formalize the existence of edges as a signal, transforming the original embedding problem into signal processing. We also designed a simple and efficient Edge-Convolution Network structure for FTSE and compared the complexity of it with RNN based approaches. Our main technique is Discrete-Time Fourier Transform (DTFT for short), which transforms the discrete temporal signal into its continuous frequency domain. Therefore, we can embed history timesteps into a fixed-length vector, enlarging the receptive field in a single batch. Our empirical study shows that FTSE is an efficient method in modeling temporal graphs and a good approach to model the signal with high precision. We summarize the contribution of this work as follows: 1. We proposed Fourier Temporal State Embedding (FTSE) to learn dynamic graph representation via transforming time-series signal into the frequency domain. FTSE directly modeled the harmonic component rather than timesteps. We also designed a simple but potent edge convolution network to model continuous-time dynamic graphs. FTSE is also the first GCN based approach capable of modeling continuous time dynamic graphs (CTDGs). 2. We studied the drawbacks of sequential-based methods in time and space complexity and justified that FTSE has much lower complexity and smaller parameter scale, making it a more efficient alternative to sequential-based approaches. This has also been proven with experiments. 3. Extensive empirical study showed that FTSE significantly outperforms previous methods in convergence speed as well as model performance in Link Prediction, Node/Edge Classification, achieving more than 10% improvement on some datasets.

2. RELATED WORK

Many static network embedding methods are proposed to map the nodes to low-dimensional vector representations while preserving network structure as well as node information. Both supervised and unsupervised techniques have been designed. Dynamic graph embedding methods are often extensions of their static counterparts. DANE (Roweis & Saul, 2000; Belkin & Niyogi, 2002) et al., 2018) , which proposes to walk on the temporal order. Deep learning approaches are also popular in this area thanks to the flourishing new models. Dyn-GEM (Kamra et al., 2017) used an autoencoding approach which minimizes the reconstruction loss as well as the distance of connected nodes in the embedding space. The point process-based approach is also popular in dynamic knowledge graph modeling. KnowEvolve (Trivedi et al., 2018) and DyRep (Trivedi et al., 2018) model the occurrence of edges as a point-process and model the intensity function with neural networks. DynamicTriad (Zhou et al., 2018) focuses on the basic 'triad closure process' where a group of three vertices is developed from unconnected vertices. They proved that this process is fundamental in graph evolving thereby making their model able to predict network dynamics. HTNE (Zhou et al., 2018) used the Hawkes process with the attention mechanism to determine the influence of historical neighbors. Point process based approaches are especially good at event time prediction.



used a matrix factorization-based approach to generate static node embeddings from eigenvectors of graph Laplacian matrix. This work was extended by(Li et al., 2017)  by updating eigenvectors from previous ones to generate dynamic node embeddings. Random-walk based approaches(Perozzi et al.,  2014; Grover & Leskovec, 2016)  used normalized inner products of node embedding to model the transition probabilities in random-walk. These two approaches are extended by CTDANE (Nguyen

