SIGN AND BASIS INVARIANT NETWORKS FOR SPECTRAL GRAPH REPRESENTATION LEARNING

Abstract

We introduce SignNet and BasisNet-new neural architectures that are invariant to two key symmetries displayed by eigenvectors: (i) sign flips, since if v is an eigenvector then so is -v; and (ii) more general basis symmetries, which occur in higher dimensional eigenspaces with infinitely many choices of basis eigenvectors. We prove that under certain conditions our networks are universal, i.e., they can approximate any continuous function of eigenvectors with the desired invariances. When used with Laplacian eigenvectors, our networks are provably more expressive than existing spectral methods on graphs; for instance, they subsume all spectral graph convolutions, certain spectral graph invariants, and previously proposed graph positional encodings as special cases. Experiments show that our networks significantly outperform existing baselines on molecular graph regression, learning expressive graph representations, and learning neural fields on triangle meshes. Our code is available at https://github.com/cptq/SignNet-BasisNet.

1. INTRODUCTION

Numerous machine learning models process eigenvectors, which arise in various settings including principal component analysis, matrix factorizations, and operators associated to graphs or manifolds. An important example is the use of Laplacian eigenvectors to encode information about the structure of a graph or manifold (Belkin & Niyogi, 2003; Von Luxburg, 2007; Lévy, 2006) . Positional encodings that involve Laplacian eigenvectors have recently been used to generalize Transformers to graphs (Kreuzer et al., 2021; Dwivedi & Bresson, 2021) , and to improve the expressive power and empirical performance of graph neural networks (GNNs) (Dwivedi et al., 2022) . Furthermore, these eigenvectors are crucial for defining spectral operations on graphs that are foundational to graph signal processing and spectral GNNs (Ortega et al., 2018; Bruna et al., 2014) . However, there are nontrivial symmetries that should be accounted for when processing eigenvectors, as has been noted in many fields (Eastment & Krzanowski, 1982; Rustamov et al., 2007; Bro et al., 2008; Ovsjanikov et al., 2008) . For instance, if v is an eigenvector, then so is -v, with the same eigenvalue. More generally, if an eigenvalue has higher multiplicity, then there are infinitely many unit-norm eigenvectors that can be chosen. Indeed, a full set of linearly independent eigenvectors is only defined up to a change of basis in each eigenspace. In the case of sign invariance, for any k eigenvectors there are 2 k possible choices of sign. Accordingly, prior works on graph positional encodings randomly flip eigenvector signs during training in order to approximately learn sign invariance (Kreuzer et al., 2021; Dwivedi et al., 2020; Kim et al., 2022) . However, learning all 2 k invariances is challenging and limits the effectiveness of Laplacian eigenvectors for encoding positional information. Sign invariance is a special case of basis invariance when all eigenvalues are distinct, but general basis invariance is even more difficult to deal with. In Appendix C.2, we show that higher dimensional eigenspaces are abundant in real datasets; for instance, 64% of molecule graphs in the ZINC dataset have a higher dimensional eigenspace. In this work, we address the sign and basis ambiguity problems by developing new neural networks-SignNet and BasisNet. Under certain conditions, our networks are universal and can approximate any continuous function of eigenvectors with the proper invariances. Moreover, our networks are theoretically powerful for graph representation learning-they can provably approximate and go beyond both spectral graph convolutions and powerful spectral invariants, which allows our networks to express graph properties like subgraph counts that message passing neural networks cannot. Laplacian eigenvectors with SignNet and BasisNet can provably approximate many previously proposed graph positional encodings, so our networks are general and remove the need for choosing one of the many positional encodings in the literature. Experiments on molecular graph regression tasks, learning expressive graph representations, and texture reconstruction on triangle meshes illustrate the empirical benefits of our models' approximation power and invariances. 

2. SIGN AND BASIS INVARIANT NETWORKS

(Q 1 , Q 2 , Q 3 )). For an n × n symmetric matrix, let λ 1 ≤ . . . ≤ λ n be the eigenvalues and v 1 , . . . , v n the corresponding eigenvectors, which we may assume to form an orthonormal basis. For instance, we could consider the normalized graph Laplacian L = I -D -1/2 AD -1/2 , where A ∈ R n×n is the adjacency matrix and D is the diagonal degree matrix of some underlying graph. For undirected graphs, L is symmetric. Nonsymmetric matrices can be handled very similarly, as we show in Appendix B.1. Motivation. Our goal is to parameterize a class of models f (v 1 , . . . , v k ) taking k eigenvectors as input in a manner that respects the eigenvector symmetries. This is because eigenvectors capture much information about data; for instance, Laplacian eigenvectors of a graph capture clusters, subgraph frequencies, connectivity, and many other useful properties (Von Luxburg, 2007; Cvetković et al., 1997) . A major motivation for processing eigenvector input is for graph positional encodings, which are additional features appended to each node in a graph that give information about the position of that node in the graph. These additional features are crucial for generalizing Transformers to graphs, and also have been found to improve performance of GNNs (Dwivedi et al., 2020; 2022) . Figure 2 illustrates a standard pipeline and the use of our SignNet within it: the input adjacency, node features, and eigenvectors of a graph are used to compute a prediction about the graph. Laplacian eigenvectors are processed before being fed into this prediction model. Laplacian eigenvectors have been widely used as positional encodings, and many works have noted that sign and/or basis invariance should be addressed in this case (Dwivedi & Bresson, 2021; Beaini et al., 2021; Dwivedi et al., 2020; Kreuzer et al., 2021; Mialon et al., 2021; Dwivedi et al., 2022; Kim et al., 2022) . Sign invariance. For any eigenvector v i , the sign flipped -v i is also an eigenvector, so a function f : R n×k → R dout (where d out is an arbitrary output dimension) should be sign invariant: f (v 1 , . . . , v k ) = f (s 1 v 1 , . . . , s k v k ) for all sign choices s i ∈ {-1, 1}. That is, we want f to be invariant to the product group {-1, 1} k . This captures all eigenvector symmetries if the eigenvalues λ i are distinct and the eigenvectors are unit-norm. Basis invariance. If the eigenvalues have higher multiplicity, then there are further symmetries. Let V 1 , . . . , V l be bases of eigenspaces-i.e., V i = v i1 . . . v i d i ∈ R n×di has orthonormal columns and spans the eigenspace associated with the shared eigenvalue µ i = λ i1 = . . . = λ i d i . Any other orthonormal basis that spans the eigenspace is of the form V i Q for some orthogonal Q ∈ O(d i ) ⊆ R di×di (see Appendix F.2). Thus, a function f : R n× l i=1 di → R dout that is invariant

