TELEPORT GRAPH CONVOLUTIONAL NETWORKS

Abstract

We consider the limitations in message-passing graph neural networks. In message-passing operations, each node aggregates information from its neighboring nodes. To enlarge the receptive field, graph neural networks need to stack multiple message-passing graph convolution layers, which leads to the over-fitting issue and over-smoothing issue. To address these limitations, we propose a teleport graph convolution layer (TeleGCL) that uses teleport functions to enable each node to aggregate information from a much larger neighborhood. For each node, teleport functions select relevant nodes beyond the local neighborhood, thereby resulting in a larger receptive field. To apply our structure-aware teleport function, we propose a novel method to construct structural features for nodes in the graph. Based on our TeleGCL, we build a family of teleport graph convolutional networks. The empirical results on graph and node classification tasks demonstrate the effectiveness of our proposed methods.

1. INTRODUCTION

Graph neural networks (GNNs) have shown great capability in solving challenging tasks on graph data such as node classification (Grover & Leskovec, 2016; Kipf & Welling, 2017; Veličković et al., 2017; Gao et al., 2018) , graph classification (Xu et al., 2018; Gao & Ji, 2019; You et al., 2019) , and link prediction (Zhang & Chen, 2018; Chen et al., 2019; Zhou et al., 2019) . Most graph convolutional networks are based on message-passing operations, in which each node aggregates information from its neighboring nodes. To enable a larger receptive field (Chen et al., 2016) , GNNs need to stack multiple layers, which is straightforward but can result in several issues. Firstly, stacking multiple layers involves massive trainable parameters, which consequently increases the risk of over-fitting. Secondly, message-passing operations mostly use averaging to combine the aggregated features, which significantly reduces the distinguishability of network embeddings. From this point, GNNs that are based on message-passing operations can not use deep network architecture due to these limitations. Some works such as Geom- GCN (Pei et al., 2020) try to solve these issues by involving more nodes in the feature aggregation process. However, Geom-GCN doesn't consider the original graph topology information when generating the additional set of nodes for aggregation, which can neglect some relevant nodes from a structural perspective. To address the above limitations and increase the receptive field effectively, we propose a teleport graph convolution layer (TeleGCL) that uses teleport functions to select highly-relevant nodes at the global scope. A teleport function computes relevances between the center node and other nodes beyond the local neighborhood. The nodes with particular relevances are teleported for the center node. Here, the selection of teleported nodes is not restricted by the graph topology. This enables the center node to gather information from a larger neighborhood without going deep, which helps to avoid over-fitting and over-smoothing issues. In particular, we propose two teleport functions; those are structure-aware and feature-aware teleport functions. They compute the nodes' relevances from graph structural perspective and node features perspective, respectively. Based on our TeleGCL, we build a family of teleport graph convolutional networks. The empirical results on graph and node classification tasks demonstrate the effectiveness of our proposed methods. et al., 2020) have achieved state-of-the-art performances on various challenging tasks in the field of network embedding. The mainstream of graph deep learning operations follows a message-passing schema. In a message-passing operation, each node sends its features, known as message, to its neighboring nodes in the graph. Then each node aggregates messages from its neighborhood and uses them to update its features. When combing the aggregated features, different strategies can be applied. In the graph convolution layer (GCN) (Kipf & Welling, 2017), features from neighboring nodes are given equal weights in the aggregation process. To assign different weights to different neighboring nodes, the graph attention network (Veličković et al., 2017 ) employs an attention mechanism to compute aggregation weights. Based on these message-passing operations, graph neural networks stack multiple layers, which enables a larger receptive field. Recently, some research works try to perform message passing beyond the local neighborhood. Pei et al. (2020) proposed to construct a continuous latent space that enables graph neural networks to perform feature learning in the latent space. To be specific, it first projects nodes' features to a 2-dimensional latent and continuous space. Based on the latent space, a structural neighborhood is constructed based on the Euclidean distance of each pair of nodes in the 2-dimensional space. In this process, the construction of structural features does not consider the graph connectivity information in the graph. Thus, the structural neighborhood in (Pei et al., 2020) is still built on node features without considering the graph topology. In this work, we propose a method to generate structure-aware features for each node. In particular, we use the graph connectivity and similarity information with the neighboring nodes and construct a feature vector for each node. By considering graph connectivity, our constructed structural features can reflect graph topology information.

3. TELEPORT GRAPH CONVOLUTIONAL NETWORKS

In this work, we propose the teleport graph convolution layer (TeleGCL) that enables a center node to aggregate information beyond regular neighborhood structure by using some teleport functions. To enable effective node teleportation, we propose two teleport functions from structure-aware and feature-aware perspectives. Specifically, we propose a novel method to construct structural features for nodes, which can be used by structure-aware functions to select relevant nodes. Based on our TeleGCL, we propose the teleport graph convolutional networks for network embedding learning.

3.1. LIMITATIONS OF MESSAGE-PASSING OPERATIONS

Currently, most graph convolution networks are based on message-passing operations. In a messagepassing operation, each node aggregates information from its neighboring nodes that usually are the one-hop neighborhood. Intuitively, it is beneficial to use information from a large neighborhood for network embedding learning. To enlarge the receptive field, a straight way is to stack multiple message-passing layers. A graph convolutional network with k layers enables nodes to receive information from a k-hop neighborhood. However, this method results in two issues. Firstly, it increases the risk of over-fitting by involving much more trainable parameters. The number of trainable parameters in the network increases when stacking multiple layers. Unlike regular convolutional neural networks, there is no effective graph pooling layer that can enlarge the receptive field without involving trainable parameters. Stacking many graph convolution layers will inevitably increase the risk of over-fitting. Secondly, stacking multiple layers will reduce the distinguishability of network embeddings, which is often referred to as the over-smoothing issue (Pei et al., 2020) . Due to the invariant property in graph structures, message-passing operations cannot learn trainable weights in the aggregation process (Kipf & Welling, 2017; Gao et al., 2018) . Averaging operation is usually used for information aggregation from the neighborhood. Consequently, information from relevant distant nodes will be diluted and each node carries similar information. In this work, we propose a teleport graph convolution layer to address this issue. This layer enables each node to aggregate information from a set of relevant nodes that are not directly connected to the center node in the original graph structure. Teleport functions are used to determine the relevant nodes from different perspectives.

3.2. TELEPORT GRAPH CONVOLUTION LAYER

To address the limitations in message-passing operations, we propose the teleport graph convolution layer (TeleGCL), which enables nodes to aggregate information beyond their local neighborhoods.

