ANALYZING THE EXPRESSIVE POWER OF GRAPH NEURAL NETWORKS IN A SPECTRAL PERSPECTIVE

Abstract

In the recent literature of Graph Neural Networks (GNN), the expressive power of models has been studied through their capability to distinguish if two given graphs are isomorphic or not. Since the graph isomorphism problem is NP-intermediate, and Weisfeiler-Lehman (WL) test can give sufficient but not enough evidence in polynomial time, the theoretical power of GNNs is usually evaluated by the equivalence of WL-test order, followed by an empirical analysis of the models on some reference inductive and transductive datasets. However, such analysis does not account the signal processing pipeline, whose capability is generally evaluated in the spectral domain. In this paper, we argue that a spectral analysis of GNNs behavior can provide a complementary point of view to go one step further in the understanding of GNNs. By bridging the gap between the spectral and spatial design of graph convolutions, we theoretically demonstrate some equivalence of the graph convolution process regardless it is designed in the spatial or the spectral domain. Using this connection, we managed to re-formulate most of the state-of-the-art graph neural networks into one common framework. This general framework allows to lead a spectral analysis of the most popular GNNs, explaining their performance and showing their limits according to spectral point of view. Our theoretical spectral analysis is confirmed by experiments on various graph databases. Furthermore, we demonstrate the necessity of high and/or band-pass filters on a graph dataset, while the majority of GNN is limited to only low-pass and inevitably it fails.

1. INTRODUCTION

Over the last five years, many Graph Neural Networks (GNNs) have been proposed in the literature of geometric deep learning (Veličković et al., 2018; Gilmer et al., 2017; Bronstein et al., 2017; Battaglia et al., 2018) , in order to generalize the very efficient deep learning paradigm into the world of graphs. This large number of contributions explains a new challenge recently tackled by the community, which consists in assessing the expressive power of GNNs. In this area of research, there is a consensus to evaluate the theoretic expressive power of GNNs according to equivalence of Weisfeiler-Lehman (WL) test order (Morris et al., 2019; Xu et al., 2019; Maron et al., 2019b; a) . Hence, GNNs models are frequently classified as "as powerful as 1-WL", "as powerful as 2-WL", . . . , "as powerful as k-WL". However, this perspective cannot make differences between two methods if they are as powerful as the same WL test order. Moreover, it does not always explain success or failure of any GNN on common benchmark datasets. In this paper, we claim that analyzing theoretically and experimentally GNNs with a spectral point of view can bring a new perspective on their expressive power. So far, GNNs have been generally studied separately as spectral based or as spatial based (Wu et al., 2019b; Chami et al., 2020) . To the best of our knowledge, Message Passing Neural Networks (MPNNs) (Gilmer et al., 2017) and GraphNets (Battaglia et al., 2018) are the only attempts to merge both approaches in the same framework. However, these models are not able to generalize custom designed spectral filters, as well as the effect of each convolution support in a multi convolution case. The spatial-spectral connection is also mentioned indirectly in several cornerstone studies by Defferrard et al. (2016) ; Kipf & Welling (2017) ; Levie et al. (2019) . Since the spectral-spatial interchangeability is missing, they did not propose to show spectral behavior of any graph convolution. Recent studies have also attempted to show, for a limited number of spatial GNNs, that they act as low-pass filters (NT & Maehara, 2019; Wu et al., 2019a) . NT & Maehara (2019) concluded that using adjacency induces low-pass effects, while Wu et al. (2019a) studied a single spatial GNN's spectral behavior by assuming adding self-connection changes the given topology of the graph. In this paper, we bridge the gap between spectral and spatial domains for GNNs. Our first contribution consists in demonstrating the equivalence of convolution processes regardless if they are defined as spatial or as spectral GNN. Using this connection, we propose a new general framework and taxonomy for GNNs as the second contribution. Taking advantage of this equivalence, our third contribution is to provide a spectral analysis of any GNN model. This spectral analysis is another perspective for the analysis of expressive power of GNNs. Our theoretical spectral analysis is confirmed by experiments on various well-known graph datasets. Furthermore, we show the necessity of high and/or band-pass filters in our experiments, while the majority of GNNs are limited to only low-pass filters and thus inevitably fail when dealing with these problems. The code used in this paper is available at https://github.com/balcilar/gnn-spectral-expressive-power. The remainder of this paper is organized as follows. Section 2 introduces convolutional GNNs and presents existing approaches. In Section 3 and Section 4, we describe the main contributions mentioned above. Section 5 presents a series of experiments and results which validate our propositions. Finally, Section 6 concludes this paper.

2. PROBLEM STATEMENT AND STATE OF THE ART

Let G be a graph with n nodes and an arbitrary number of edges. Connectivity is given by the adjacency matrix A ∈ {0, 1} n×n and features are defined on nodes by X ∈ R n×f0 , with f 0 the length of feature vectors. For any matrix X, we used X i , X :j and X i,j to refer its i-th column vector, j-th row vector and scalar value on (i, j) location, respectively. A graph Laplacian is L = D -A (or L = I -D -1/2 AD -1/2 ) where D ∈ R n×n is the diagonal degree matrix and I is the identity. Through eigendecomposition, L can be written by L = U diag(λ)U T where each column of U ∈ R n×n is an eigenvector of L, λ ∈ R n gathers the eigenvalues of L and diag(.) function creates a diagonal matrix whose diagonal elements are from a given vector. We use superscript to refer same kind variable as base. For instance, H (l) ∈ R n×f l refers node representation on layer l whose feature dimension is f l . A Graph Convolution layer takes the node representation of the previous layer H (l-1) as input and produces a new representation H (l) , with H (0) = X.

2.1. SPECTRAL APPROACHES

Spectral GNNs rely on the spectral graph theory (Chung, 1997) . In this framework, signals on graphs are filtered using the eigendecomposition of graph Laplacian (Shuman et al., 2013) . By transposing the convolution theorem to graphs, the spectral filtering in the frequency domain can be defined by x f lt = U diag(Φ(λ))U x, where Φ(.) is the desired filter function. As a consequence, a graph convolution layer in spectral domain can be written by a sum of filtered signals followed by an activation function as in (Bruna et al., 2013) , namely H (l+1) j = σ f l i=1 U diag(F (l,j) i )U H (l) i , for j ∈ {1, . . . , f l+1 }. Here, σ is the activation function, F (l,j) ∈ R n×f l is the corresponding weight vector to be tuned as used in (Henaff et al., 2015) for the single-graph problem known as non-parametric spectral GNN. A first drawback is the necessity of Fourier and inverse Fourier transform by matrix multiplication of U and U T . Another drawback occurs when generalizing the approach to multi-graph learning problems. Indeed, the k-th element of the vector F (l,j) i weights the contribution of the k-th eigenvector to the output. Those weights are not shareable between graphs of different sizes, which means a

