CLEP: EXPLOITING EDGE PARTITIONING FOR GRAPH CONTRASTIVE LEARNING

Abstract

Generative and contrastive are two fundamental unsupervised approaches to model graph information. The graph generative models extract intra-graph information whereas the graph contrastive learning methods focus on inter-graph information. Combining these complementary sources of information can potentially enhance the expressiveness of graph representations, which, nevertheless, is underinvestigated by existing methods. In this work, we introduce a probabilistic framework called contrastive learning with edge partitioning (CLEP) that integrates generative modeling and graph contrastive learning. CLEP models edge generation by aggregating latent node interactions over multiple overlapping hidden communities. Inspired by the assembling behavior of communities in graph generation, CLEP learns community-specific graph embeddings, which are assembled together to represent the entire graph and further used to predict the graph's identity via a contrastive objective. To relate each embedding to one hidden community, we define a set of community-specific weighted edges for node feature aggregation by partitioning the observed edges according to the latent node interactions associated with the corresponding hidden community. With these unique designs, CLEP is able to model the statistical dependency among hidden communities, graph structures, as well as the identity of each graph; it can also be trained end-to-end via variational inference. We evaluate CLEP on real-world benchmarks under self-supervised and semi-supervised settings and achieve promising results, which demonstrate the effectiveness of our method. Various exploratory studies are also conducted to highlight the characteristics of the inferred hidden communities and the potential benefits they bring to representation learning.

1. INTRODUCTION

Generative modeling and contrastive learning are both commonly employed to learn graph representations without label supervision. Both types of methods learn the embedding space by leveraging some ground-truth information from the observed graphs, but the aspects of data that each type chooses to fit are different. Graph generative models (Kipf & Welling, 2016; Mehta et al., 2019; Wang et al., 2020) prioritize intra-graph information, i.e., the information in each individual graph. The representations provided by graph generative models are usually related to the formation of its own edges. Contrariwise, graph contrastive learning methods (You et al., 2020; 2021; 2022; Xie et al., 2022) focus on capturing inter-graph information, they put graphs under comparison to highlight the inherent similarity and differences among a group of graphs. The difference in the focused graph information leads to complementary strengths and weaknesses of graph generative and contratsive learning methods. The advantage of graph generative models is their ability to recover the structural information of some latent factors, which is lost during graph generation. These latent factors, relevant to each graph in the sense of its own formation, usually preserve valuable information for various graph-analytic tasks. However, the quality of the embeddings provided by graph generative models is questionable because the encoded information is limited to the"expression levels" of these latent factors, which may be insufficient to downstream tasks other than graph generation. Unlike generative models, graph contrastive learning methods cannot automatically find meaningful latent factors in the graph, but they are well recognized for producing high-quality feature representations once the raw structural information is given. An integration of graph generative modeling and graph contrastive learning potentially combines the complementary strengths of these two classes of methods, which would further benefit representation learning. However, such a direction is insufficiently explored. In this work, we introduce Contrastive Learning with Edge Partitioning (CLEP), a probabilistic framework that provides a concrete solution to the integration of graph generative modeling with graph contrastive learning. To better leverage the strengths of both methods, we assign different duties to the graph generative model and the contrastive learner, where the former takes the responsibility of extracting the hidden structures of the latent factors which explain graph generation, while the latter treats the inferred hidden structures as new raw inputs, learns their corresponding graph embeddings in a contrastive paradigm, and then aggregates all the embeddings in a way that is intuitively compatible with how the hidden structures interact during graph generation. More specifically, CLEP is developed upon the graph generative models that explain the formation of edges by cumulative latent node interactions associated with some hidden communities (Yang & Leskovec, 2012; 2013; 2014; Zhou, 2015) . Membership and intra-community node interactions are assumed to be independent among different hidden communities. According to the generative model, we embody the structures of these communities by explicitly modeling the average node interactions in each community and partitioning the observed edges accordingly. As indicated by Jenatton et al. ( 2012), node interactions that happen in different communities may follow multiple relations. It is highly likely that, when nodes interact under different relations, the information exchange that comes along also focuses on different aspects. To better capture the potentially heterogeneous communityspecific information, we define a set of encoders to process the information that comes from different communities. The training of these encoders are based on aggregating their corresponding contrastive learning tasks with a set of weights that measure the "importance" of each community. Finally, we gather the community-specific graph embeddings to represent the overall information on a graph, as an analogy with the assembling behavior of communities in graph generation. We summarize the major contributions of this work as follows: • We propose CLEP as an integration of graph generative model and graph contrastive learning, which can effectively capture both intraand inter-graph information. • We formulate the statistical problem of training CLEP as the maximum likelihood estimation of a latent variable model, which supports end-to-end optimization via variational inference. • We show through exploratory studies that the strength of factor-wise representation learning is to capture nonrepetitive graph information from different hidden factors, which offers more flexible embedding selection & combination when facing various downstream tasks. • We train CLEP under self-supervised and semi-supervised settings and conduct an extensive empirical evaluation of the obtained graph representations, finding that CLEP consistently outperforms existing arts on various real-world benchmarks.

2. PRELIMINARIES

Information encoding on graphs. A graph encoder maps the information on each graph to a vector representation. For a graph G with N nodes, its given information usually includes a node feature matrix X ∈ R N ×F and an adjacency matrix A. The most effective way to aggregate these two forms of information is based on graph neural networks (GNNs) (Kipf & Welling, 2017; Hamilton et al., 2017; Veličković et al., 2018; Xu et al., 2019) . For a GNN with T layers, denoting H 0 = X, its propagation rule can be summarized as H t = AGG t (f t (H t-1 ), A), t ∈ [1, T ], where AGG denotes neighborhood aggregation and f (•) denotes nonlinear transformation. Appending a readout layer to a regular GNN converts it from a node-level encoder to a graph-level encoder, where the readout operation is defined as h = READOUT({H t } t=1,T ). In the sequel, we use h V (A, X) to denote node-level encoders, and use h G (A, X) to denote graph-level encoders. Graph contrastive learning (GCL). The ground-truth information that GCL methods use to train the graph encoders is the uniqueness of each graph. The distribution of the graph representations in the embedding space is expected to capture the inherent similarity and differences among the graphs. To this end, positive pairs are created, with each one consisting of two nonidentical views of the same graph. Some contrastive methods (Xie et al., 2022) only pull together the representations of positive

