NEURAL POOLING FOR GRAPH NEURAL NETWORKS

Abstract

Tasks such as graph classification, require graph pooling to learn graph-level representations from constituent node representations. In this work, we propose two novel methods using fully connected neural network layers for graph pooling, namely Neural Pooling Method 1 and 2. Our proposed methods have the ability to handle variable number of nodes in different graphs, and are also invariant to the isomorphic structures of graphs. In addition, compared to existing graph pooling methods, our proposed methods are able to capture information from all nodes, collect second-order statistics, and leverage the ability of neural networks to learn relationships among node representations, making them more powerful. We perform experiments on graph classification tasks in the bio-informatics and social network domains to determine the effectiveness of our proposed methods. Experimental results show that our methods lead to an absolute increase of upto 1.2% in classification accuracy over previous works and a general decrease in standard deviation across multiple runs indicating greater reliability. Experimental results also indicate that this improvement in performance is consistent across several datasets.

1. INTRODUCTION

Over the past several years, there is a growing number of applications where data is generated from non-Euclidean domains and is represented as graphs with complex relationships and interdependency between entities. Deep learning generalised from grid-like data to the graph domain has led to the development of the remarkably successful Graph Neural Networks (GNNs) (Fan et al., 2019; Gao et al., 2019; Ma et al., 2019a; Wang et al., 2019b) and its numerous variants such the Graph Convolutional Network (GCN) (Kipf & Welling, 2017) , GraphSAGE (Hamilton et al., 2017) , graph attention network (GAT) (Veličković et al., 2018) , jumping knowledge network (JK) (Xu et al., 2018) , and graph isomorphism networks (GINs) (Xu et al., 2019) , etc. Pooling is a common operation in deep learning on grid-like data, such as images. Pooling layers provide an approach to down sampling feature maps by summarizing the presence of features in patches of the feature map. It reduces dimensionality and also provides local translational invariance. In the case of graph data, pooling is used to obtain a representation of a graph using its constituent node representations. However, it is challenging to develop graph pooling methods due to the some special properties of graph data such as the variable number of nodes in different graphs and the isomorphic structures of graphs. Firstly, the number of nodes varies in different graphs, while the graph representations are usually required to have the same fixed size to fit into other downstream machine learning models where they are used for tasks such as classification. Therefore, graph pooling should be capable of handling the variable number of node representations as inputs and producing fixed-sized graph representations. Secondly, unlike images and texts where we can order pixels and words according to the spatial structural information, there is no inherent ordering relationship among nodes in graphs. Therefore, isomorphic graphs should have the same graph representation, and hence, graph pooling should give the same output by taking node representations in any order as inputs. Our main contributions in this work are two novel graph pooling methods, Neural Pooling Method 1 and 2. These new pooling methods allow us to do the following,i) produce the same dimensional graph representation for graphs with variable number of nodes, ii) remain invariant to the isomorphic structures of graphs, iii) collect second-order statistics, iv) leverage trainable parameters in the form of fully connected neural networks to learn relationships between underlying node representations to generate high quality graph representations which are then used for graph classification tasks. Experiments are performed on four benchmark bio-informatics datasets and five popular social network datasets to demonstrate the effectiveness and superiority of our proposed graph pooling methods. Experimental results show that our methods lead to an improvement in classification accuracy over existing methods and are also more reliable as compared to previous works.

2.1. GRAPH NEURAL NETWORKS

A graph can be represented by its adjacency matrix and node features. Formally, for a graph G consisting of n nodes, its topology information can be represented by an adjacency matrix A ∈ {0, 1} n × n and the node features can be represented as X ∈ R n × d , assuming each node has a d-dimensional feature vector. GNNs learn feature representations for different nodes using these matrices (Gilmer et al., 2017) . Several approaches are proposed to investigate deep GNNs, and they generally follow a neighborhood information aggregation scheme (Gilmer et al., 2017; Xu et al., 2019; Kipf & Welling, 2017; Hamilton et al., 2017; Veličković et al., 2018) . In each step, the representation of a node is updated by aggregating the representations of its neighbors. Graph Convolutional Networks (GCNs) are popular variants of GNNs and inspired by the first order graph Laplacian methods (Kipf & Welling, 2017) . Graph pooling is used to connect embedded graphs outputted by GNN layers with classifiers for graph classification. Given a graph, GNN layers produce node representations, where each node is embedded as a vector. Graph pooling is applied after GNN layers to process node representations into a single feature vector as the graph representation. A classifier takes the graph representation and performs graph classification.

2.2. GRAPH POOLING

Early studies employ simple methods such as averaging and summation as graph pooling (Xu et al., 2019; Duvenaud et al., 2015; Defferrard et al., 2016) . However, averaging and summation do not capture the feature correlation information, curtailing the overall model performance (Zhang et al., 2018) . Other studies have proposed advanced graph pooling methods, including DIFFPOOL (Ying et al., 2018) , SORT-POOL (Zhang et al., 2018 ), TOPKPOOL (Gao & Ji, 2019) , SAGPOOL (Lee et al., 2019), and EIGENPOOL (Ma et al., 2019b) , and achieve great performance on multiple benchmark datasets. EIGENPOOL involves the computation of eigenvectors, which is slow and expensive. DIFFPOOL (Ying et al., 2018) treats the graph pooling as a node clustering problem. A cluster of nodes from the original graph are merged to form a new node in the new graph. DIFF-POOL (Ying et al., 2018) proposes to perform the graph convolution operation on node features to obtain node clustering assignment matrix. Intuitively, the class assignment of a given node should depend on the class assignments of other neighbouring nodes. However, DIFFPOOL does not explicitly consider high-order structural relationships, which we that are important for graph pooling. SORTPOOL (Zhang et al., 2018) , TOPKPOOL (Gao & Ji, 2019), and SAGPOOL (Lee et al., 2019) learn to select important nodes from the original graph and use these nodes to build a new graph. They share the similar idea to learn a sorting vector based on node representations, which indicates the importance of different nodes. Then only the top k important nodes are selected to form a new graph while the other nodes are ignored. However, the ignored nodes may contain important features and this information is lost during pooling. It is worth noting that all the graph pooling methods mentioned till now only collect first-order statistics (Boureau et al., 2010) . A recent study has proposed second order graph pooling methods SOP ool bimap and SOP ool attention (Wang & Ji, 2020) . In this work, we propose two novel methods using fully connected neural network layers for graph pooling, namely Neural Pooling Method 1 and 2. Compared to existing graph pooling methods, our proposed methods are able to capture information from all nodes, collect second-order statistics, and leverage the ability of neural networks to learn relationships among node representations, making them more powerful.

