GRAPH CONTRASTIVE LEARNING UNDER HETEROPHILY: UTILIZING GRAPH FILTERS TO GENERATE GRAPH VIEWS Anonymous

Abstract

Graph Neural Networks have achieved tremendous success in (semi-)supervised tasks for which task-specific node labels are available. However, obtaining labels is expensive in many domains, specially as the graphs grow larger in size. Hence, there has been a growing interest in the application of self-supervised techniques, in particular contrastive learning (CL), to graph data. In general, CL methods work by maximizing the agreement between encoded augmentations of the same example, and minimizing agreement between encoded augmentations of different examples. However, we show that existing graph CL methods perform very poorly on graphs with heterophily, in which connected nodes tend to belong to different classes. First, we show that this is attributed to the ineffectiveness of existing graph augmentation methods. Then, we leverage graph filters to directly generate augmented graph views for graph CL under heterophily. In particular, instead of explicitly augmenting the graph topology and encoding the augmentations, we use a high-pass filter in the encoder to generate node representations only based on high-frequency graph signals. Then, we contrast the high-pass filtered representations with their low-pass counterparts produced by the same encoder, to generate representations. Our experimental results confirm that our proposed method, HLCL, outperforms state-of-the-art CL methods on benchmark graphs with heterophily, by up to 10%.

1. INTRODUCTION

Graph neural networks (GNNs) are powerful tools for learning graph-structured data in various domains, including social networks, biological compound structures, and citation networks (Kipf & Welling, 2016; Hamilton et al., 2017; Veličković et al., 2017) . In general, GNNs leverage the graph's adjacency matrix to update the node representations by aggregating information from their neighbors. This can be seen as a lowpass filter that smooths the graph signals and produces similar node representations (Nt & Maehara, 2019) . GNNs have achieved great success in supervised and semi-supervised learning, where task-specific labels are available. However, obtaining high-quality labels is very expensive in many domains, specially as graphs grow larger in size. This has motivated a recent body of work on self-supervised learning on graphs that learn the representations in an unsupervised manner (Velickovic et al., 2019; Peng et al., 2020; Qiu et al., 2020; Hassani & Khasahmadi, 2020; Zhu et al., 2020b) . Among self-supervised methods, Contrastive Learning (CL) has shown a great success by achieving comparable performance with its supervised counterparts (Chen et al., 2020) . Contrastive learning obtains representations by maximizing the mutual information between different augmented views of the same example, and minimizing agreement between differently augmented views of different examples. Despite being successful on graphs with homophily, where neighboring nodes tend to share the same label, existing graph CL methods cannot learn high-quality representations for graphs with heterophily, where connected nodes often belong to different classes Zhu et al. (2020b) . State-of-the-art graph CL methods work by contrasting the encoded node representations in two explicitly augmented graph views, generated by altering the graph topology or node features (Zhu et al., 2020c; 2021b;  Figure 1 : Our proposed HLCL framework leverages a high-pass graph filter in the encoder to generate a non-smooth graph view, and contrast it with its smooth counterparts generated by the same encoder. Velickovic et al., 2019; Thakoor et al., 2021) . Specifically, topology augmentation methods alter the graph structure by removing or adding nodes or edges. Feature augmentation methods alter the node features by masking particular columns, dropping features at random, or randomly shuffling the node features. The choice of the augmentation strategy has been shown to have a great influence on the performance of graph CL methods (Zhang & Ma, 2022; Zhu et al., 2021b) . In particular, topology augmentations such as edge dropping that produce sparser graphs are the most effective for graphs with homophily (Zhu et al., 2021b) . In this work, we first study the effect of existing graph augmentation methods on the performance of graph CL under heterophily. We show that topology and feature augmentation methods that can effectively boost the performance of graph CL under homophily do not provide any considerable advantage under heterophily. In particular, we experimentally confirm that for graphs with homophily, topology augmentation effectively decreases the homophily ratio, and feature augmentation changes the variance of features in a neighborhood. Hence, explicit graph augmentation techniques enable the low-pass GNN encoder to better distinguish the smooth node representations-that are similar in a neighborhood-by adding more diversity to the graph. However, such methods cannot effectively diversify and alter the homophily ratio or variance of features in graphs with heterophily. Indeed, for such graphs, incorporating the graph structure is crucial for generating high-quality representations that can distinguish dissimilar nodes from their neighborhood. Next, we address the above shortcoming by proposing an effective augmentation strategy for CL on graphs with heterophily, while ensuring high-quality representation learning on graphs with homophily. The key idea of our proposed method, HLCL, is to leverage a high-pass graph filter in a typical GNN encoder to directly generate non-smooth node representations, and contrast them with their smooth counterparts generated by the same encoder. More specifically, we use the normalized Lapcalian matrix as the high-pass filter in the GNN encoder to aggregate the node representations. The Laplacian matrix magnifies the differences in the node features in a neighborhood and makes the representations distinct. This is crucial for learning high-quality representation under heterophily. In contrast, a typical GNN encoder uses the normalized adjacency matrix as a low-pass filter to aggregate the node features with those of their neighbors, and enforce similar representations for the nodes in the same neighborhood. Maximizing the mutual information between the high-pass and low-pass filtered representations enables the encoder to learn rich representations on graphs with heterophily. Importantly, our method is complementary to explicit graph augmentation techniques, and such methods can be applied to further boost the performance, particularly on graphs with homophily. This allows our framework to achieve state-of-the-art under heterophily and comparable performance under homophily as we confirm by our experiments. We show the effectiveness of our HLCL framework through extensive experiments on graphs with heterophily and homophily for unsupervised representations learning under the linear evaluation protocol. Our

